| ID |
Date |
Author |
Subject |
|
606
|
Thu Apr 25 15:02:50 2024 |
Marc | 16:00 - 0:00 Thursday 25 April |
16:20:
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
16.45 Checks:
DSSSD bias & leakage current - Grafana - attachments 2
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 6
ucesb - attachment 7
XX.XX Checks
DSSSD bias & leakage current - Grafana - attachments X
FEE64 temperatures OK - attachment X
ADC data item stats - attachment X
per FEE64 Rate spectra - attachment X
Merger etc - attachment X
ucesb - attachment X |
|
613
|
Sat Apr 27 15:13:06 2024 |
Norah | 16:00 - 00:00 Saturday 27 April | 14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
|
610
|
Fri Apr 26 15:34:00 2024 |
RDP | 16:00 - 00:00 Friday 26 April | This appeared in the system log:
Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
These messages seem to appear sporadically.
18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?
18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.
aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters
Thanks Nic!
22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)
22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.
|
|
645
|
Sun Jun 9 16:38:30 2024 |
Norah , Muneerah, JB | 16:0-00:00 9 June 2024 | AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.
17:00
DSSSD bias & leakage current - attachment 2
FEE64 temperatures OK - attachment 3
Statistics attachment 4
17:39
Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.
23:19 Flange removed. Starting to take beam.
|
|
616
|
Sun Apr 28 15:09:34 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 | 16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
|
620
|
Sun Apr 28 20:35:57 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 | 21:30 screenshots
and
23:24 screenshots |
|
588
|
Mon Apr 22 15:17:07 2024 |
DSJ | 16.00-0.00 22/04/24 | First checks of shift. All looks ok.
Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.
Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)
checks at 16.00 - merger has crashed - aida01 dropped out - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09
Reset 01, restarted DAQ, and restarted as R16.
16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.
Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms
RUN17 STARTED 16.51|
HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.
20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift
21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps
22.48 - things apear ok. (Attachments 23-26) |
|
231
|
Fri Apr 16 15:52:08 2021 |
DSJ | 16 April 16.00 shift | Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 10 3 1 1 2 1 3 3 3 6 : 36156
aida02 : 25 10 4 3 1 2 2 3 3 3 6 : 36500
aida03 : 27 8 5 2 3 4 3 3 2 3 6 : 36092
aida04 : 30 4 4 5 0 2 4 2 3 3 6 : 36472
aida05 : 19 5 7 1 3 2 2 2 2 4 6 : 37060
aida06 : 18 13 1 2 1 3 2 3 3 3 6 : 36544
aida07 : 24 7 3 2 2 3 1 3 2 4 6 : 37384
aida08 : 15 11 2 5 2 4 1 4 3 3 6 : 37076
aida09 : 7 3 4 1 5 6 5 4 3 2 6 : 36308
aida10 : 8 3 13 4 3 3 1 4 2 3 6 : 36040
aida11 : 18 11 17 7 4 4 4 3 2 3 6 : 36752
aida12 : 12 7 7 7 4 4 2 3 2 3 6 : 36024
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
17:27. Checked system stats, temps, leakage etc. looks ok
18:49. Tape server writing to disk /NULL/R21_9+
/TapeData points to /media/SecondDrive/ - 3.6Tb free
19.15. checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 9 16 2 2 2 1 3 3 3 6 : 36424
aida02 : 1 3 5 4 1 3 3 2 3 3 6 : 36268
aida03 : 24 13 11 6 3 3 2 4 2 3 6 : 36472
aida04 : 5 4 4 4 3 3 3 2 3 3 6 : 36404
aida05 : 17 8 4 2 2 2 2 2 2 4 6 : 36996
aida06 : 26 11 3 2 0 3 2 3 3 3 6 : 36528
aida07 : 24 7 2 1 1 1 2 3 2 4 6 : 37272
aida08 : 13 12 6 3 3 4 1 4 3 3 6 : 37140
aida09 : 22 15 19 2 6 5 4 4 3 2 6 : 36416
aida10 : 16 11 6 4 3 3 1 4 2 3 6 : 36024
aida11 : 12 4 5 4 2 3 3 3 2 3 6 : 35872
aida12 : 18 12 9 4 4 4 2 3 2 3 6 : 36024
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
19.53. Checked system stats, temps, leakage etc. looks ok
20:30. Checked system stats, temps, leakage etc. looks ok
20:59. Checked system stats, temps, leakage etc. looks ok
21:31 System checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb40 : 6
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 21 9 17 3 2 2 1 3 3 3 6 : 36492
aida02 : 5 1 1 6 0 2 2 3 3 3 6 : 36332
aida03 : 2 10 11 6 4 4 3 3 2 3 6 : 36296
aida04 : 27 12 3 4 0 2 3 2 3 3 6 : 36220
aida05 : 19 9 7 2 1 2 2 2 2 4 6 : 36996
aida06 : 28 5 0 4 0 3 1 3 3 3 6 : 36248
aida07 : 19 13 10 6 2 2 1 3 2 4 6 : 37524
aida08 : 18 9 6 0 1 4 1 3 3 3 6 : 36400
aida09 : 17 19 12 3 6 5 4 4 3 2 6 : 36348
aida10 : 23 9 7 4 2 3 1 4 2 3 6 : 35988
aida11 : 19 10 3 3 2 3 3 3 2 3 6 : 35884
aida12 : 12 6 0 3 2 4 2 3 2 3 6 : 35648
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
22.06. Checked system stats, temps, leakage etc. looks ok
22.18CEST BNC PB-5 amplitude changed from 1V to 2V
fioler NULL/R21_67
22.23CEST All histograms zero'd
22.43. Checked system stats, temps, leakage etc. looks ok
23.21. Checked system stats, temps, leakage etc. looks ok
23.46 system checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb40 : 6
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 17 12 7 0 3 3 2 4 2 3 6 : 36180
aida02 : 35 3 5 2 0 2 1 3 3 3 6 : 36148
aida03 : 22 6 10 2 1 3 2 4 2 3 6 : 36136
aida04 : 27 9 3 5 2 4 3 3 2 3 6 : 36100
aida05 : 20 9 3 3 2 2 2 2 2 4 6 : 37032
aida06 : 25 11 1 2 0 3 1 3 3 3 6 : 36236
aida07 : 20 6 6 2 3 3 2 4 1 4 6 : 37216
aida08 : 25 10 0 2 1 3 1 3 3 3 6 : 36276
aida09 : 10 4 3 4 4 5 4 4 3 2 6 : 35960
aida10 : 26 9 3 0 1 3 1 4 2 3 6 : 35744
aida11 : 2 3 4 2 2 3 3 3 2 3 6 : 35744
aida12 : 4 9 7 2 2 4 2 3 2 3 6 : 35720
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021 |
|
197
|
Sat Mar 13 23:08:08 2021 |
CA | 14th March 00:00-08:00 | 00:08 DAQ continues ok - writing to file R46_221
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x82a0 : 0x82a4 : 4
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x2 : 0x3 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 11 6 2 5 1 3 2 2 3 4 10 : 54492
aida02 : 12 7 3 4 3 3 3 2 3 3 6 : 36440
aida03 : 4 4 2 4 2 3 2 2 3 3 6 : 36048
aida04 : 18 9 4 2 1 3 2 4 1 4 15 : 73936
aida05 : 31 11 8 7 6 4 2 18 7 4 7 : 55220
aida06 : 36 8 18 4 4 2 1 4 3 2 5 : 31088
aida07 : 26 7 4 3 1 2 1 3 3 3 6 : 36224
aida08 : 25 12 5 4 4 3 2 3 3 3 6 : 36884
aida09 : 2 9 4 2 2 2 2 3 3 3 6 : 36432
aida10 : 11 3 4 6 4 3 1 3 3 3 6 : 36548
aida11 : 21 8 5 0 1 3 3 3 2 3 6 : 35748
aida12 : 10 6 6 2 4 4 1 3 3 3 6 : 36600
00:14 FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
Detector bias/leakage currents ok - attachment 3
00:17 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
01:30 beam off
01:31 beam back
02:19 system wide checks
same as entry at 00:09, *except*
Base Current Difference
aida07 fault 0x82a0 : 0x82a5 : 5
White Rabbit error counter test result: Passed 11, Failed 1
02:22 FEE64 Temperatures ok - attachment 4
Good event statistics ok - attachment 5
Detector bias/leakage currents ok - attachment 6
02:30 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
no further bad timestamp errors in NewMerger terminal
02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7
02:50 ... and sure enough lose connection to aida07 after checking stats
DAQ stop, Merger/TapeServer closed
telnet aida07 -> sent reboot command
mounted ok, reset DAQ, TapeService and Merge
restarted DATA forwarding to MBS
all looks to be ok now -> writing to R56
03:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x2 : 0x5 : 3
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 13 5 3 7 2 2 2 2 3 4 10 : 54508
aida02 : 2 7 3 2 2 3 1 4 3 3 6 : 36784
aida03 : 8 8 1 3 2 3 1 3 3 3 6 : 36304
aida04 : 20 9 2 3 0 3 2 4 1 4 15 : 73880
aida05 : 51 74 70 54 53 49 22 18 7 4 7 : 72188
aida06 : 15 15 3 9 1 2 2 4 3 2 5 : 31044
aida07 : 24 6 2 3 3 4 2 3 2 3 7 : 39888
aida08 : 19 15 1 2 1 3 1 4 3 3 6 : 36820
aida09 : 18 4 2 7 3 4 1 3 3 3 6 : 36648
aida10 : 20 7 1 1 1 3 3 2 3 3 6 : 36216
aida11 : 25 7 5 3 4 2 1 4 2 3 6 : 35916
aida12 : 3 3 2 2 3 2 3 2 3 3 6 : 36164
03:18 FEE64 Temperatures ok - attachment 8
Good event statistics ok - attachment 9
Detector bias/leakage currents ok - attachment 10
Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
03:29 more bad timestamp error messages - attachment 11
03:52 analyser output on R56_35 - attachment 12
no timewarps
03:53 DAQ continues ok, writing to file R56_41
05:18 system wide checks:
same as entry at 03:08, *but* with additional WR decoder status error
Base Current Difference
aida07 fault 0x8c74 : 0x8c77 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
05:19 FEE64 Temperatures ok - attachment 14
Good event statistics ok - attachment 13
Detector bias/leakage currents ok - attachment 15
Merger ok - 4.4M items/s
TapeServer ok - 15 MB/s
05:25 rate spectra - attachment 16
05:33 AIDA implant XY hit patterns - attachments 17/18/19
05:56 BuTIS interface control set to 0x3 at 04.52 UTC
07:06 system wide checks - same error messages as entry at 05:18
07:08 FEE64 Temperatures ok - attachment 20
good event statistics ok - attachment 21
detector bias/leakage currents ok - attachment 22
Merger ok - 4.3M items/s
TapeServer ok - 15 MB/s
07:22 no bad timestamp errors since 02:30 UTC |
|
597
|
Tue Apr 23 13:15:43 2024 |
PP | 14:00 checks | All looks good.
Screenshots attached |
|
96
|
Wed Nov 13 09:10:37 2019 |
CA, NH, OH | 13th November 2019 | 10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration
lowered slow comparator threshold to 0xa
DAQ start
pulser on
zero'd histograms
pulser peak widths:
FEE width (ch)
aida01 - 130
aida02 - 168
aida03 - 83
aida04 - 83
aida05 - 200
aida06 - 148
aida07 - 250
aida08 - 81
aida09 - 134
aida10 - 190
aida11 - 333
aida12 - 149
For comparison, peak widths from yesterday's measurement:
pulser peak widths - FEE width(ch)
1 128
2 145
3 84
4 74
5 207
6 125
7 172
8 80
9 193
10 183
11 238
12 138
Attachments 1 & 2 : pulser peak spectra
Attachment 3 - good event statistics
Attachment 4 - detector bias/ leakage currents
|
|
443
|
Fri May 13 15:00:23 2022 |
CB | 13 March - 16:00-24:00 shift | 16:00 Took over remotely from OH, TD who remain in GSI
Stats OK - attach 1
Stats & Leakage OK - attach 2
Temps OK
Grafana OK - attach 3
System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
17:29 All good, but no beam.
17:30 Beam is back
17:58 No beam again
18:10 Beam is back. Stats OK.
Ucesb OK - attach 4
Stats & Leakage currents ok - attach 5
Temps OK - attach 6
Grafana OK - attach 7
System-wide checks as before.
19:36 No beam again.
20:37 Beam is back
Stats OK - attach 8
Leakage OK
Temps OK
Grafana OK - attach 9
Ucesb OK - attach 10
System-wide checks as before
21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.
22:51 Stats & Leakage current OK - attach 11
Temps OK
Grafana OK - attach 12
Ucesb OK - attach 13
System-wide checks as before |
|
573
|
Sat Apr 13 14:04:44 2024 |
JB | 13 April Noise checks | 15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).
TEMP OK.
Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.
9/16 <20kHz, max 176k
15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.
No change in noise observed.
9/16 <20kHz, max 187k
15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.
10/16 <20kHz, max 158k.
Current bPlast grounding scheme in attachments 8-9.
|
|
594
|
Tue Apr 23 11:09:00 2024 |
PP | 12:00 checks | Everything is OK now.
Screenshots attached. |
|
572
|
Thu Apr 11 22:04:50 2024 |
JB | 12.04.2024 AIDA-bPlast noise optimisation | 15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.
Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.
Histograms okay, some FEE64s now have hot channels. Attachment 2.
9/16 aidas < 20kHz - max rate 162k
18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.
TO-DO (kicking the can down the road):
- Some work to do on noise, but we might have to accept the situation as it currently is.
- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.
- Try different grounding configuration:
- Grounding drain wires to frame. Observe AIDA conditions.
- Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.
|
|
571
|
Thu Apr 11 08:27:21 2024 |
JB, TD | 11 April noise tests |
TO-DO for 11.04.2024
- Try bringing bPlast drain wire ground back to the PSU ground for PN 300 and R&SRMP4040.
- Recheck the downstream detector bias and ground scheme.
-
9:27 Restarted AIDA and hit go at 9:54. Situation the same as it was in the evening of 10.04.2024. 10 out of 16 FEE64s in good condition. Waveforms of aida16,
TEMP OK - HV OK. See attachments 1-6.
10:20 We grounded the output of the R&SRMP4040 to to the drain wire ground of the bPlast ribbon cables and the output of the PN 300 PSU that is powering the booster boards. Results given by attachments 7 -11. We only saw a marginal improvement in the overall noise condition.
10:32 We connected the ground of the frame to the drain wire ground, that is also mutually connected to the ground of the R&SRMP4040 output ground and PN 300 output ground. Results given by attachments ???. We did not see any improvement in the overal noise condition.
11.00 DSSSD HV OFF
FEE64 power OFF
Check seating of all adaptor PCBs and drain/ground wires secured - generally OK
Restart
1.8.W spectra - 20us FSR - attachments 15-16
per FEE64 Rate spectra - attachment 17
10/16 < 20k, max 110k
ADC data item stats - attachment 18
Incremental improvement.
11.45 DSSSD HV OFF
FEE64 power OFF
test - daisy chain removed
Restart
1.8.W spectra - 20us FSR - attachments 20-21
per FEE64 Rate spectra - attachment 19
10/16 < 20k, max 210k
ADC data item stats - attachment 22
overall somewhat worse
12.20 DSSSD HV OFF
FEE64 power OFF
Re-install test - daisy chain, Tighten aida04 DSSSD ribbon cable drain wire.
Restart
1.8.W spectra - 20us FSR - attachments 23-24
per FEE64 Rate spectra - attachment 25
ADC data item stats - attachment 26
10/16 < 20k, max 110k
Status quo ante
aida04 & aida08 1*W & 2*W spectra - 200us FSR - attachments 27-31
large transients observed for aida04 1*W - cable or ASIC fault?
aida16 **W spectra - 200us FSR - attachments 32-35
all channels apopear to be working but mix of high/low noise channels for asics #3-4, asics #1-2 all appear high noise
14.00 DSSSD HV OFF
FEE64 power OFF
Disconnect ribbon cables from aida04 adaptor PCB
Restart
n+n FEE64 1.8.W spectra - 20us FSR - attachments 37
per FEE64 Rate spectra - attachment 36
ADC data item stats - attachment 38
Implies origin of large transients observed in aida04 asic#1 is downstream of the FEE64 adaptor PCB, i.e. ribbon cable or DSSSD.
14.40 DSSSD HV OFF
FEE64 power OFF
re-connect ribbon cables from aida04 adaptor PCB
1x pin J2 slightly bent - straightended with screwdriver - FFSD connector insertion OK
Restart
n+n FEE64 1.8.W spectra - 20us FSR - attachments 40-41
per FEE64 Rate spectra - attachment 42-43
ADC data item stats - attachment 39
10x < 20k, max 120k
aida11 asic#4 1.8.W spectra - 200us FSR - attachment
no large transients observed
DSSSD #1 & #2 bias from -120V to -100V
ADC data item stats - attachment 40
9x < 20k, max 220k (aida08)
per FEE64 Rate spectra - attachment 41
17.15 bPlas ON -
Current bPlast ground configuration:
bPlas current ground configuration - drainwires of all ribbon cables excluding short side (cont. with snout) are grounded back to the bPlast R&SRMP4040 PSU, on the 29 V output. 4 V PSU to booster boards are floating, output of PN 300 is grounded to the PSU. The mesytec PSU, that also powers bPlast SiPMs at
29V is not grounded to anything but the frame. Snout is currently light tight, I suspect internal radiation from FATIMA.
It is noted that bPlast current draw fluctuated significantly (+/- 300 mV) as a result of thresholds set to the detector. I tried to set the thresholds HIGH to stabilise the bPlast detector as the power draw is fluctuating greatly.
AIDA noise very good
ADC data item stats - attachment 42
12x < 20k, max 83k (aida04)
per FEE64 Rate spectra - attachment 43
To Do list
- separate 29V (low current) and 4V (high current) return paths/ground refs
- The mesytec PSU that powers the bPlast detector is running to frame, it might be an idea to ground ribbon c
|
|
591
|
Tue Apr 23 09:21:01 2024 |
PP | 10:20 checks - aida02 is down | Aida02 is down. Contacted Nick.
The rest looks stable, see screenshots.
10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14
Go > All 16 writing to merger, looks OK
aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again |
|
621
|
Sun Apr 28 22:52:06 2024 |
Marc | 0:00-08:00 Monday 29 April | Just another manic magic Monday, ....
Taking over Betool's AIDA shift. All is good.
This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.
01:15: AIDA04 is down again. DAQ shift crew informed.
01:27: AIDA04 is back and shift crew has been informed.
01:30 Checks:
Bias and leakage current - Attachment 1-2
Temperatures - Attachment 3
ADC Data item - Attachment 4
Rate Spectra - Attachment 5
UCESB - Attachment 6
Timestamp checks - Attachement 7
06:00 It has run very smoothly.
|
|
136
|
Mon Mar 9 13:50:08 2020 |
LS, CA, DK | 09/03/2020 system checks | 14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?
15:06 Systems checks fine.
Master clock failed (no master clock)
all checks passed
memory checks all around 38k as normal
Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees
Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k
Attachment 4: leakage currents seem normal at expected values
15:16 Merger working for all FEEs
tape server is on but no storage
data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)
ASIC controls checked and are all OK
15.48 low energy pulsar peak FWHM
fee FHWM
1 89.08
2 132.06
3 70.73
4 81.84
5 62.86
6 79.92
7 122.13
8 73.31
9 198.39
10 115.53
11 100.55
12 144.04
ALL FEEs in downstream SSD are high
16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)
16:25 included rates, note not to common scale in y (attachment 9)
ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
Did not seem completely reproducible?
We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad
Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)
19:15 Update
There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems
One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space
This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)
Now we have brought back up the AIDA systems and should run through the checks
Most system checks look good (except some calibration errors)
Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool
Bias and leak currents attached as #10
Temps as #11
Stats as #12
19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.
21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET
21:51 AIDA running to no storage again |
|
601
|
Wed Apr 24 09:04:55 2024 |
TD | 08:00-16:00 Wednesday 24 April |
09.56 DAQ continues data file S100_alpha/R18_122
FRS setting 162Eu
DSSSD bias & leakage current OK - attachments 1-2
FEE64 temperatures OK - attachment 3
aida02 ASIC temp - known fault
All system wide checks OK *except* as shown - attachments 4-6
WR timestamps OK - attachment 7
ASIC settings aida01 & aida02 - attachments 8-9
ADC, DISC, PAUSE, RESUME, WR & Correlation Scaler data time stats - attachments 10-15
per FEE64 Rate spectra - attachments 16-18
unusually can observe HEC Rates esp. in DSSSD #1
per p+n FEE64 1.8.L spectra - attachments 19-21
aida09 pulser peak width 61 ch FWHM
common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths
per n+n FEE64 1.8.L spectra - attachment 22
per FEE64 1.8.H spectra - attachments 23-24
per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?
Merger, TapeServer etc - attachments 27-28
ucesb - attachments 29-30
spill cycle reported as 3.5s, extraction 1.5s
10.36 All histograms & stats zero'd
BNC PB-5 settings - attachment 31
DSSSD bias & leakage current - Grafana Alerting - attachment 32
can observe bias & leakage current in greater detail
DSSSD #1 bias stable, c. 10s nA micro structure observed in leakage current
DSSSD #2 bias c. 10mV variations, micro structure observed in leakage current but smaller variations cf. DSSSD#1
10.46 analysis of data file R18_131
max dead time 12% aida04, 6% aida02, all others 0.1% or less
no timewarps
HEC data item rates 1.6kHz - correcting for spill cycle => 2.3kHz in spill (online analysis shows no HEC events observed outside spill)
assume typical HEC event multiplicity = 3 => implant rate ~ 760Hz in spill or 2700 implants per spill
SC42 L/R currently report c. 2kHz
|
|