 Tue Jun 11 15:01:07 2024, PP TD, 16:00-00:00 shift Tuesday 11/06/24 13x
|
All seems OK
Screenshots attached.
17.33 per FEE64 1.8.W spectra - attachments 8-9
per p+n FEE64 1.8.L spectra - attachment 10
aida09 pulser peak width 66 ch FWHM
cf
c. 11:00 today 57 ch FWHM https://elog.ph.ed.ac.uk/DESPEC/650
cf
c. 00:10 today 53 ch FWHM https://elog.ph.ed.ac.uk/DESPEC/649
17.48 ADC data item stats - attachment 11
All histograms zero'd
17.51 Analysis data file S181/R6_96 - attachment 12
max deadtime 5% (aida08)
HEC Data items 3.7kHz, LEC data items 1.8MHz
cf.
c. 12.29 today 1.1MHz https://elog.ph.ed.ac.uk/DESPEC/650
c. 00.09 today 1.1MHz https://elog.ph.ed.ac.uk/DESPEC/649 |
|
Mon Jun 10 15:07:24 2024, Marc, 16:00-00:00 shift Monday 10/06/24 31x
|
Evening shift
5pm full checks:
DSSSD bias & leakage current ok - attachment # 1-2
FEE64 temperatures ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7
7pm full checks:
DSSSD bias & leakage current ok - attachment # 8-9
FEE64 temperatures ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15
9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23
9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31
|
|
Sun Apr 28 16:26:47 2024, Betool Alayed, 16:00-00:00 Sunday 28 April 10x
|
19:24 pm screenshots
and
17:18 pm screenshots |
|
Wed Jun 12 15:41:07 2024, Betool Alayed, 16:00-00:00 Wed 12 Jun 24 27x
|
5pm full checks:
DSSSD bias & leakage current ok - attachment # 1-2
FEE64 temperatures ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6
7pm full checks:
DSSSD bias & leakage current ok - attachment # 7-8
FEE64 temperatures ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13
9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20
11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27 |
|
Thu Jun 13 21:06:44 2024, Robert Page, 16:00 to 00:00 shift on 13/6/2024 26x
|
Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.
However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).
Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.
Another set of sceenshots taken at around 23:00 - attachments 19 - 26.
|
|
Thu Apr 25 15:02:50 2024, Marc, 16:00 - 0:00 Thursday 25 April 8x
|
16:20:
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
16.45 Checks:
DSSSD bias & leakage current - Grafana - attachments 2
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 6
ucesb - attachment 7
XX.XX Checks
DSSSD bias & leakage current - Grafana - attachments X
FEE64 temperatures OK - attachment X
ADC data item stats - attachment X
per FEE64 Rate spectra - attachment X
Merger etc - attachment X
ucesb - attachment X |
|
Sat Apr 27 15:13:06 2024, Norah, 16:00 - 00:00 Saturday 27 April 15x
|
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
|
Fri Apr 26 15:34:00 2024, RDP, 16:00 - 00:00 Friday 26 April 6x
|
This appeared in the system log:
Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
These messages seem to appear sporadically.
18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?
18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.
aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters
Thanks Nic!
22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)
22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.
|
|
Sun Jun 9 16:38:30 2024, Norah , Muneerah, JB, 16:0-00:00 9 June 2024
|
AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.
17:00
DSSSD bias & leakage current - attachment 2
FEE64 temperatures OK - attachment 3
Statistics attachment 4
17:39
Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.
23:19 Flange removed. Starting to take beam.
|
|
Sun Apr 28 15:09:34 2024, Betool Alayed, 16:0-00:00 28 April 2024
|
16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
|
Sun Apr 28 20:35:57 2024, Betool Alayed, 16:0-00:00 28 April 2024 10x
|
21:30 screenshots
and
23:24 screenshots |
|
Mon Apr 22 15:17:07 2024, DSJ, 16.00-0.00 22/04/24 26x
|
First checks of shift. All looks ok.
Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.
Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)
checks at 16.00 - merger has crashed - aida01 dropped out - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09
Reset 01, restarted DAQ, and restarted as R16.
16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.
Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms
RUN17 STARTED 16.51|
HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.
20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift
21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps
22.48 - things apear ok. (Attachments 23-26) |
|
Fri Apr 16 15:52:08 2021, DSJ , 16 April 16.00 shift 12x
|
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 10 3 1 1 2 1 3 3 3 6 : 36156
aida02 : 25 10 4 3 1 2 2 3 3 3 6 : 36500
aida03 : 27 8 5 2 3 4 3 3 2 3 6 : 36092
aida04 : 30 4 4 5 0 2 4 2 3 3 6 : 36472
aida05 : 19 5 7 1 3 2 2 2 2 4 6 : 37060
aida06 : 18 13 1 2 1 3 2 3 3 3 6 : 36544
aida07 : 24 7 3 2 2 3 1 3 2 4 6 : 37384
aida08 : 15 11 2 5 2 4 1 4 3 3 6 : 37076
aida09 : 7 3 4 1 5 6 5 4 3 2 6 : 36308
aida10 : 8 3 13 4 3 3 1 4 2 3 6 : 36040
aida11 : 18 11 17 7 4 4 4 3 2 3 6 : 36752
aida12 : 12 7 7 7 4 4 2 3 2 3 6 : 36024
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
17:27. Checked system stats, temps, leakage etc. looks ok
18:49. Tape server writing to disk /NULL/R21_9+
/TapeData points to /media/SecondDrive/ - 3.6Tb free
19.15. checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 9 16 2 2 2 1 3 3 3 6 : 36424
aida02 : 1 3 5 4 1 3 3 2 3 3 6 : 36268
aida03 : 24 13 11 6 3 3 2 4 2 3 6 : 36472
aida04 : 5 4 4 4 3 3 3 2 3 3 6 : 36404
aida05 : 17 8 4 2 2 2 2 2 2 4 6 : 36996
aida06 : 26 11 3 2 0 3 2 3 3 3 6 : 36528
aida07 : 24 7 2 1 1 1 2 3 2 4 6 : 37272
aida08 : 13 12 6 3 3 4 1 4 3 3 6 : 37140
aida09 : 22 15 19 2 6 5 4 4 3 2 6 : 36416
aida10 : 16 11 6 4 3 3 1 4 2 3 6 : 36024
aida11 : 12 4 5 4 2 3 3 3 2 3 6 : 35872
aida12 : 18 12 9 4 4 4 2 3 2 3 6 : 36024
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
19.53. Checked system stats, temps, leakage etc. looks ok
20:30. Checked system stats, temps, leakage etc. looks ok
20:59. Checked system stats, temps, leakage etc. looks ok
21:31 System checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb40 : 6
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 21 9 17 3 2 2 1 3 3 3 6 : 36492
aida02 : 5 1 1 6 0 2 2 3 3 3 6 : 36332
aida03 : 2 10 11 6 4 4 3 3 2 3 6 : 36296
aida04 : 27 12 3 4 0 2 3 2 3 3 6 : 36220
aida05 : 19 9 7 2 1 2 2 2 2 4 6 : 36996
aida06 : 28 5 0 4 0 3 1 3 3 3 6 : 36248
aida07 : 19 13 10 6 2 2 1 3 2 4 6 : 37524
aida08 : 18 9 6 0 1 4 1 3 3 3 6 : 36400
aida09 : 17 19 12 3 6 5 4 4 3 2 6 : 36348
aida10 : 23 9 7 4 2 3 1 4 2 3 6 : 35988
aida11 : 19 10 3 3 2 3 3 3 2 3 6 : 35884
aida12 : 12 6 0 3 2 4 2 3 2 3 6 : 35648
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
22.06. Checked system stats, temps, leakage etc. looks ok
22.18CEST BNC PB-5 amplitude changed from 1V to 2V
fioler NULL/R21_67
22.23CEST All histograms zero'd
22.43. Checked system stats, temps, leakage etc. looks ok
23.21. Checked system stats, temps, leakage etc. looks ok
23.46 system checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb40 : 6
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 17 12 7 0 3 3 2 4 2 3 6 : 36180
aida02 : 35 3 5 2 0 2 1 3 3 3 6 : 36148
aida03 : 22 6 10 2 1 3 2 4 2 3 6 : 36136
aida04 : 27 9 3 5 2 4 3 3 2 3 6 : 36100
aida05 : 20 9 3 3 2 2 2 2 2 4 6 : 37032
aida06 : 25 11 1 2 0 3 1 3 3 3 6 : 36236
aida07 : 20 6 6 2 3 3 2 4 1 4 6 : 37216
aida08 : 25 10 0 2 1 3 1 3 3 3 6 : 36276
aida09 : 10 4 3 4 4 5 4 4 3 2 6 : 35960
aida10 : 26 9 3 0 1 3 1 4 2 3 6 : 35744
aida11 : 2 3 4 2 2 3 3 3 2 3 6 : 35744
aida12 : 4 9 7 2 2 4 2 3 2 3 6 : 35720
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021 |
|
Sat Mar 13 23:08:08 2021, CA, 14th March 00:00-08:00 22x
|
00:08 DAQ continues ok - writing to file R46_221
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x82a0 : 0x82a4 : 4
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x2 : 0x3 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 11 6 2 5 1 3 2 2 3 4 10 : 54492
aida02 : 12 7 3 4 3 3 3 2 3 3 6 : 36440
aida03 : 4 4 2 4 2 3 2 2 3 3 6 : 36048
aida04 : 18 9 4 2 1 3 2 4 1 4 15 : 73936
aida05 : 31 11 8 7 6 4 2 18 7 4 7 : 55220
aida06 : 36 8 18 4 4 2 1 4 3 2 5 : 31088
aida07 : 26 7 4 3 1 2 1 3 3 3 6 : 36224
aida08 : 25 12 5 4 4 3 2 3 3 3 6 : 36884
aida09 : 2 9 4 2 2 2 2 3 3 3 6 : 36432
aida10 : 11 3 4 6 4 3 1 3 3 3 6 : 36548
aida11 : 21 8 5 0 1 3 3 3 2 3 6 : 35748
aida12 : 10 6 6 2 4 4 1 3 3 3 6 : 36600
00:14 FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
Detector bias/leakage currents ok - attachment 3
00:17 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
01:30 beam off
01:31 beam back
02:19 system wide checks
same as entry at 00:09, *except*
Base Current Difference
aida07 fault 0x82a0 : 0x82a5 : 5
White Rabbit error counter test result: Passed 11, Failed 1
02:22 FEE64 Temperatures ok - attachment 4
Good event statistics ok - attachment 5
Detector bias/leakage currents ok - attachment 6
02:30 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
no further bad timestamp errors in NewMerger terminal
02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7
02:50 ... and sure enough lose connection to aida07 after checking stats
DAQ stop, Merger/TapeServer closed
telnet aida07 -> sent reboot command
mounted ok, reset DAQ, TapeService and Merge
restarted DATA forwarding to MBS
all looks to be ok now -> writing to R56
03:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x2 : 0x5 : 3
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 13 5 3 7 2 2 2 2 3 4 10 : 54508
aida02 : 2 7 3 2 2 3 1 4 3 3 6 : 36784
aida03 : 8 8 1 3 2 3 1 3 3 3 6 : 36304
aida04 : 20 9 2 3 0 3 2 4 1 4 15 : 73880
aida05 : 51 74 70 54 53 49 22 18 7 4 7 : 72188
aida06 : 15 15 3 9 1 2 2 4 3 2 5 : 31044
aida07 : 24 6 2 3 3 4 2 3 2 3 7 : 39888
aida08 : 19 15 1 2 1 3 1 4 3 3 6 : 36820
aida09 : 18 4 2 7 3 4 1 3 3 3 6 : 36648
aida10 : 20 7 1 1 1 3 3 2 3 3 6 : 36216
aida11 : 25 7 5 3 4 2 1 4 2 3 6 : 35916
aida12 : 3 3 2 2 3 2 3 2 3 3 6 : 36164
03:18 FEE64 Temperatures ok - attachment 8
Good event statistics ok - attachment 9
Detector bias/leakage currents ok - attachment 10
Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
03:29 more bad timestamp error messages - attachment 11
03:52 analyser output on R56_35 - attachment 12
no timewarps
03:53 DAQ continues ok, writing to file R56_41
05:18 system wide checks:
same as entry at 03:08, *but* with additional WR decoder status error
Base Current Difference
aida07 fault 0x8c74 : 0x8c77 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
05:19 FEE64 Temperatures ok - attachment 14
Good event statistics ok - attachment 13
Detector bias/leakage currents ok - attachment 15
Merger ok - 4.4M items/s
TapeServer ok - 15 MB/s
05:25 rate spectra - attachment 16
05:33 AIDA implant XY hit patterns - attachments 17/18/19
05:56 BuTIS interface control set to 0x3 at 04.52 UTC
07:06 system wide checks - same error messages as entry at 05:18
07:08 FEE64 Temperatures ok - attachment 20
good event statistics ok - attachment 21
detector bias/leakage currents ok - attachment 22
Merger ok - 4.3M items/s
TapeServer ok - 15 MB/s
07:22 no bad timestamp errors since 02:30 UTC |
|
Tue Apr 23 13:15:43 2024, PP, 14:00 checks 8x
|
All looks good.
Screenshots attached |
|
Wed Nov 13 09:10:37 2019, CA, NH, OH, 13th November 2019
|
10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration
lowered slow comparator threshold to 0xa
DAQ start
pulser on
zero'd histograms
pulser peak widths:
FEE width (ch)
aida01 - 130
aida02 - 168
aida03 - 83
aida04 - 83
aida05 - 200
aida06 - 148
aida07 - 250
aida08 - 81
aida09 - 134
aida10 - 190
aida11 - 333
aida12 - 149
For comparison, peak widths from yesterday's measurement:
pulser peak widths - FEE width(ch)
1 128
2 145
3 84
4 74
5 207
6 125
7 172
8 80
9 193
10 183
11 238
12 138
Attachments 1 & 2 : pulser peak spectra
Attachment 3 - good event statistics
Attachment 4 - detector bias/ leakage currents
|
|
Fri May 13 15:00:23 2022, CB, 13 March - 16:00-24:00 shift 10x
|
16:00 Took over remotely from OH, TD who remain in GSI
Stats OK - attach 1
Stats & Leakage OK - attach 2
Temps OK
Grafana OK - attach 3
System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
17:29 All good, but no beam.
17:30 Beam is back
17:58 No beam again
18:10 Beam is back. Stats OK.
Ucesb OK - attach 4
Stats & Leakage currents ok - attach 5
Temps OK - attach 6
Grafana OK - attach 7
System-wide checks as before.
19:36 No beam again.
20:37 Beam is back
Stats OK - attach 8
Leakage OK
Temps OK
Grafana OK - attach 9
Ucesb OK - attach 10
System-wide checks as before
21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.
22:51 Stats & Leakage current OK - attach 11
Temps OK
Grafana OK - attach 12
Ucesb OK - attach 13
System-wide checks as before |
|
Sat Apr 13 14:04:44 2024, JB, 13 April Noise checks 9x
|
15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).
TEMP OK.
Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.
9/16 <20kHz, max 176k
15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.
No change in noise observed.
9/16 <20kHz, max 187k
15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.
10/16 <20kHz, max 158k.
Current bPlast grounding scheme in attachments 8-9.
|
|
Tue Apr 23 11:09:00 2024, PP, 12:00 checks 8x
|
Everything is OK now.
Screenshots attached. |
|
Thu Apr 11 22:04:50 2024, JB, 12.04.2024 AIDA-bPlast noise optimisation
|
15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.
Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.
Histograms okay, some FEE64s now have hot channels. Attachment 2.
9/16 aidas < 20kHz - max rate 162k
18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.
TO-DO (kicking the can down the road):
- Some work to do on noise, but we might have to accept the situation as it currently is.
- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.
- Try different grounding configuration:
- Grounding drain wires to frame. Observe AIDA conditions.
- Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.
|
|