 Fri May 6 09:31:41 2022, TD NH, Friday 6 May 33x
|
10:30 DAQ appears to have stopped
Attachment 2 - DSSSD bias and leakage current OK
Attachments 1, 3-5 - zero rate adc data items all FEE64s - note aida08 adc data item counter zero
RESET/SETUP/GO aida08 *only*
DAQ STOP *all* FEE64s
Attachment 6 - FEE64 temperatures OK
Attachments 7-23 - system wide checks - all OK
10:48 DAQ GO *all* FEE64s
all spectra zero'd
attachment 29 - ADC data items
n+n FEE64s aida02, 4, 6 & 8 c. 250k
p+n FEE64s aida07, 9 & 12 c. 100-150k, all other FEE64s < 100k
attachments 26 & 27 1.8.L spectra
aida01 peak width 112ch FWHM
aida02 peak width 485 ch FWHM
attachments 24 & 25 1.8.W spectra 20us FSR
attachment 30 - aida05 1*H spectra - aida05 asic#1 is producing HEC data
attachment 31 - aida05 also producing disc data
attachments 32 & 33 merger and tape server OK - 14MB/s to disk - no data to MBS currently?
18.36 Disconnect ribbon cabling from adaptor PCB - continue to observe HEC events from aida05 asic#1
Disconnect test inputs from adaptor PCB - continue to observe HEC events from aida05 asic#1
Conclude aida05 asic#1 has probably failed in operation and should be replaced. Other FEE64s are currently in CRYRING hall and measurement is in progress - ML says we should be able to access Monday. |
|
Fri Jun 4 08:40:52 2021, TD, Friday 4 June 8x
|
09.20 DAQ found stopped
files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight
aida15 rebooted at 00.02
regained control of DAQ and issued DAQ STOP by restarting NewMerger *twice*
11.00 DAQ reset and setup
All system wide checks OK *except*
Base Current Difference
aida05 fault 0x4d3c : 0x4d3c : 0
aida05 : WR status 0x10
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 2 4 2 8 5 4 1 2 2 4 7 : 41352
aida02 : 0 3 3 6 6 4 1 2 2 4 7 : 41352
aida03 : 2 4 2 12 1 3 2 2 2 4 7 : 41352
aida04 : 2 2 1 7 4 3 2 2 2 4 7 : 41352
aida05 : 2 3 1 5 3 2 3 2 2 4 7 : 41360
aida06 : 2 4 2 6 4 3 2 2 2 4 7 : 41352
aida07 : 1 5 2 9 4 2 4 3 1 4 7 : 41324
aida08 : 0 3 1 7 4 3 4 3 1 4 7 : 41352
aida09 : 3 3 0 8 2 2 3 2 2 4 7 : 41380
aida10 : 1 2 2 11 3 2 3 3 3 3 7 : 41044
aida11 : 2 3 1 6 6 4 2 3 3 3 7 : 41072
aida12 : 4 2 1 8 3 3 3 3 1 4 7 : 41072
aida13 : 2 4 1 8 4 4 2 3 3 3 7 : 41016
aida14 : 1 4 1 9 4 4 2 3 3 3 7 : 41044
aida15 : 0 5 1 2 2 1 2 4 3 4 7 : 42872
aida16 : 1 3 2 9 4 4 4 4 2 3 7 : 41052
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
11.03 DAQ resumes file S496/R32
alpha background
slow comparator 0x64
BNC PB-5 pulser OFF
11.09 1.8.W spectra - 20us FSR - attachments 1 & 2
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
ADC data items - attachment 4
FEE64 temperatures OK - attachment 5
DSSSD bias & leakage currents OK - attachment 6
Merger/Tape Server/Merger statistics - attachment 7
no merger errors reported since previous restart
13.07 analysis of file S496/R32_16 - attachment 8
max deadtime 0.18% aida04 |
|
Fri Jan 31 17:26:47 2025, CC, TD, MP, Friday 31 January contd. 16x
|
18.18 bPlas and BB7 installs complete
Restart AIDA DAQ, Merger, Tape Server and re-test
per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 69 ch FWHM = 42keV FWHM
per FEE64 Rate Spectra - attachment 2
BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK
per FEE64 1.8.W spectra - 20us FSR - attachments 3-5
ADC data item stats - attachment 6
aida01 aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
BB7 aida10 19k
WR timestamps OK - attachment 7
Merger, TapeSever etc - attachments 8-9
disk directpry /TapeData/FEB25
working OK
DSSSD bias & leakage current OK - attachment 10
System wide checks OK - attachment 11-15
note global clock status 6 errors reported earlier today have now gone
FEE64 temperatures OK - attachment 16
18.38 Transition to safe state
DAQ STOP
disable data transfer 1
detector bias OFF
FEE64 power OFF
Can restart as follows
1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps
If all OK can re-connect to Merger/TapeServer as follows
1) DAQ STOP
2) enable data transfer 1
3) DAQ GO
Can disconnect from Merger/TapeServer as follows
1) DAQ STOP
2) disable data transfer 1
|
|
Fri Jan 31 09:37:47 2025, CC, TD, MP, Friday 31 January 26x
|
Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000
N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.
n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted
BB7 aida10 (asics #1-2 p+n, #3-4 n+n)
BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream)
10.30 FEE64 power ON
DSSSD bias ON (BB18 only) - attachment 1
leakage current OK
ASIC settings 2024Dec13-17.02.45 restored
p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf
Attachments 2-7 WR timestamp aida05 & aida12, system wide check
aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
WR decoder status aida02
per FEE64 Rate spectra - attachment 8
aida10 high rate - BB7 not biased
rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments
ADC data item stats - attachment 9
rates generally OK
aida10 high - BB7 not biased
aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!
BNC PB-5 settings - attachment 10
WR timestamps OK - attachment 11
FEE64 temperatures OK - attachment 12
12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet
ADC data item stats - attachment 13
all rates higher cf. attachment 9
per FEE64 Rate spectra - attachment 14
per p+n FEE64 1.8.L spectra - attachment 15
aida09 pulser peak width 69 ch FWHM = 49keV FWHM
aida14 pulser peak width 50 ch FWHM = 35keV FWHM
per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17
per n+n FEE64 1.8W spectra - 20us FSR - attachment 18
DSSSD bias & leakage current OK - attachment 19
ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%
FEE64 temperatures OK - attachment 20
DSSSD bias volatge & leakage current OK - attachment 21
CAEN N1419ET ch#0 BB18, ch#1 BB7
Install of bPlas drivers, cabling, grounds complete
ADC data item stats - attachment 22
per FEE64 Rate spectra - attachment 23
all BB18 p+n FEE64s *except* aida05 show very low rates of noise
per p+n FEE64 1.8.L spectra - attachment 24
aida09 pulser peak width 62 ch FWHM = 42keV FWHM
BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD
per FEE64 1.8.W spectra - 20us FSR - attachments 25-26 |
|
Fri Apr 30 08:38:12 2021, TD, Friday 30 April 6x
|
09.40 All system wide checks OK *except*
FEE64 module aida07 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Thu Apr 29 17:50:06 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Thu Apr 29 14:44:05 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
09.44 DSSSD bias & leakage currents Ok - attachment 1
Merger time seq errors - attachment 2
no merger errors since restart yesterday
FEE64 temps OK - attachment 3
good event stats - attachment 4
Merger/Tape Server server consoles - attachment 5
Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 6
DAQ continues OK - no data storage |
|
Thu May 2 23:08:43 2024, TD, Friday 3 May
|
00.05 FEE64 temperatures OK
ADC data item stats OK
09.31 FEE64 temperatures OK
ADC data item stats OK
13.15 FEE64 temperatures OK
ADC data item stats OK
9/16 <20k max aida16 164k
20.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida03 no data - DAQ reset
9/16 <20k max aida16 158k
23.15 FEE64 temperatures OK
ADC data item stats
aida03 no data - DAQ reset
10/16 <20k max aida16 153k |
|
Mon Apr 1 10:14:09 2019, CA, NH, TD, Friday 29th March 2019
|
16:30 good event statistics ok, with 9,10,11,12 running faster (detector FEE64s) (attachment 1)
FEE temperatures ok (attachment 2)
detector bias/leakage currents ok (attachment 3)
system wide checks ok *except* FEE64 Linux memory information check causes script error,
WR decoder status returns errors
DAQ & Merger running ok
slow comparator threshold raised to 0x64
|
|
Fri May 28 10:29:07 2021, TD, Friday 28 May 9x
|
11.30 DAQ continues - file S496/R18_472
alpha background
All system wide checks OK *except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b2 : 2
aida02 fault 0xfe72 : 0xfe74 : 2
aida03 fault 0x97f4 : 0x97f6 : 2
aida04 fault 0x950 : 0x952 : 2
aida05 fault 0x8e93 : 0x8e95 : 2
aida06 fault 0x545a : 0x545c : 2
aida07 fault 0xc04d : 0xc04f : 2
aida08 fault 0xa0f7 : 0xa0f9 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 4 7 1 3 2 2 2 2 4 6 : 36980
aida02 : 3 1 5 1 2 4 3 3 2 4 6 : 37892
aida03 : 8 8 6 6 3 3 2 3 3 3 6 : 36800
aida04 : 37 12 5 6 3 4 2 3 3 3 6 : 37060
aida05 : 11 5 6 4 2 2 2 2 2 4 6 : 37044
aida06 : 4 3 3 3 2 2 2 3 3 4 6 : 38456
aida07 : 1 12 3 2 0 3 3 2 2 4 6 : 37204
aida08 : 13 8 1 2 3 4 3 3 2 4 6 : 38020
aida09 : 7 7 5 1 1 3 2 2 2 4 6 : 36996
aida10 : 12 7 9 2 1 3 3 3 1 4 6 : 36856
aida11 : 2 9 3 1 2 3 3 3 1 4 6 : 36768
aida12 : 11 4 3 3 2 4 2 3 2 4 6 : 37724
aida13 : 10 8 6 2 2 3 2 2 2 4 6 : 37128
aida14 : 16 7 7 1 0 4 2 2 2 4 6 : 37128
aida15 : 12 6 2 2 1 3 3 4 2 4 6 : 38272
aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.30 analysis of file S496/R18_471 - attachment 9
max deadtime 0.18% aida04 |
|
Fri Jun 24 15:04:06 2022, Marc, Friday 24th June - evening shift 11x
|
16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.
Next wide check will be in about an hour.
17:00
Stats -ok - attachment 1
Temperatures ok - attachement 2
Leakage current ok but - attachment 3
ucesb screen-shot - attachment 4
Wide check completed. Nothing different.
WR status decoder status:
Base Current Difference
aida07 fault 0xc53d : 0xc547 : 10
aida08 fault 0xf1be : 0xf1e9 : 43
White Rabbit error counter test result: Passed 6, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp::
Base Current Difference
aida07 fault 0x2a : 0x2c : 2
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.
19:20 -
Stats ok - attachment 6
Temp ok - attachment 7
Leakage current ok - attachement 8
Wide check completed and same output as above.
22:00
Stats ok - attachment 9
Temp ok - attachment 10
Leakage current ok - attachement 11
Wide check completed and same output as above.
|
|
Fri Jun 24 00:34:55 2022, AM, Friday 24 June 00:00-08:00 20x
|
01:30 Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good
03:30 Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good
05:30 Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good
07:30 Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good |
|
Fri Jun 24 06:56:06 2022, OH, Friday 24 June 008:00-16:00 16x
|
07:56 Took over from AM. No issues reported overnight
Stats ok - attachment 1
Temperature ok - attachment 2
Bias and leakage current ok - attachment 3
System wide checks:
Clocks ok
WR
Base Current Difference
aida07 fault 0xc4fe : 0xc53b : 61
aida08 fault 0xf0e9 : 0xf1b7 : 206
White Rabbit error counter test result: Passed 6, Failed 2
FPGA Errors
Base Current Difference
aida07 fault 0x11 : 0x29 : 24
aida08 fault 0x1a : 0x2c : 18
FPGA Timestamp error counter test result: Passed 6, Failed 2
Analysis of R3_359 - attachment 4
Dead time still around 15.9%
Merger item rate around 2E6-4E6
Tape server rate at 7750 kB/s
Current HDD free space 1.7 TB
Time left in HDD 2.59 days
Will run out of space at some point on Sunday
Have started compression of uncompressed raw data on the HDD. Using nice +10
08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
Am regularly checking the statistics
Was able to stop the DAQ by relaunching the merger.
Reset the FEEs and recovered without a powercylce
Started R4 following this stop
08:58 They have taken the beam to change the ion source for maybe 2 hours
Will stop writing data but continue forwarding to MBS
No storage ticked. Following break will be on R5
11:10 Beam is starting to come back R5 started
Stats ok - attachment 6
Temps ok - attachment 7
Bias and leakage currents ok - attachment 8
13:05 Statistics ok - attachment 9
Temps ok - attachment 10
Bias and leakage curents ok - attachment 11
System wide checks:
WR
Base Current Difference
aida07 fault 0xc53d : 0xc540 : 3
aida08 fault 0xf1be : 0xf1d0 : 18
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_33 - attachment 12
15:28 Statistics ok - attachment 13
Temperature ok - attachment 14
Bias and leakage current ok - attachment 15
System wide checks:
Base Current Difference
aida07 fault 0xc53d : 0xc546 : 9
aida08 fault 0xf1be : 0xf1e0 : 34
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_63 - attachment 16 |
|
Fri Apr 23 07:09:57 2021, LJW, Friday 23rd April 08:00-12:00 9x
|
08:30
System Checks
Clock Status error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 1
Temperatures - See attachment 2
Bias & Leakage Currents - See attachment 3
10:02
System Checks:
Clock Status Error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration Error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status Error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 4
Temperatures - See attachment 5
Bias & Leakage Currents - See attachment 6
10:45
System Checks
Clock Status Error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration check Error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status Error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 7
Temperatures - See attachment 8
Bias & Leakage Currents - See attachment 9
|
|
Thu Apr 22 23:17:46 2021, DJ , Friday 23rd April 00:00-08:00 27x
|
Took over from TD ... All is well.
00:45 System Wide Checks
Clock: 12 passed, 0 failed.
ADC Calibration: Passed 12, 0 failed.
White Rabbit Decoder: Passed 4,failed 8. - See screenshot.
FPGA Passed 11, failed 1. - See screenshot.
FEE64 Linux: See screenshot.
-----
02:49 run closed. Checking beam.
03:00 File 55 opened.
03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.
File 56 opened at 4:24.
---------
04:30 System Wide Checks (See screenshots)
White Rabbit Decoder: Passed 12, failed 0.
FPGA: page not loading properly.
----
05:37 System wide checks. See screenshots
Clock: Status: 11 passed. 1 fail.
ADC calibration: Passed 9, failed 3.
White Rabbit decoder: Passed 12, failed 0.
-----
07:32 System wide checks (See screenshots).
|
|
Fri Apr 23 20:19:52 2021, TD, Friday 23 April 20.00-00.00 7x
|
21.19 system wide checks
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 7 5 7 3 3 2 4 2 3 6 : 36356
aida02 : 67 29 27 4 0 4 2 3 2 3 4 : 28196
aida03 : 14 5 2 4 3 5 2 2 3 3 6 : 36416
aida04 : 5 5 0 7 2 3 3 3 3 3 6 : 36892
aida05 : 26 5 4 3 3 3 1 3 3 3 6 : 36464
aida06 : 20 7 5 2 3 4 2 2 3 3 6 : 36312
aida07 : 24 7 1 2 3 4 2 3 3 3 6 : 36776
aida08 : 22 7 6 2 3 3 1 3 3 3 6 : 36464
aida09 : 20 7 1 2 1 3 3 2 3 3 6 : 36248
aida10 : 9 1 6 1 1 2 2 4 2 3 6 : 35820
aida11 : 15 4 2 3 3 4 3 3 2 3 6 : 35996
aida12 : 22 11 6 2 2 3 2 2 3 3 6 : 36176
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Fri Apr 23 03:51:08 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
21.20 DSSSD biases & leakage currents OK - attachment 1
FEE64 tempos OK - attachment 2
good event stats - attachment 3
Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4
23.35 System wide checks - no changes to above
DSSSD biases & leakage currents OK - attachment 5
FEE64 tempos OK - attachment 6
good event stats - attachment 7
23.46 DAQ continues file S460/R51_378 |
|
Fri Apr 23 13:13:46 2021, MA, TD, Friday 23 April 12.00-16.00 6x
|
14.15 System wide checks
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp errors - reports 'server internal error'
14.16 DSSSD bias & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events stats - attachment 3
14.22 DAQ continues file S460/R51_205
beam intensity 2.2e+09/s
15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6
Clock check
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC check
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA
Loge page error!
Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 6 2 1 2 3 2 4 2 3 6 : 36044
aida02 : 15 3 2 2 0 3 2 3 2 3 4 : 27188
aida03 : 29 4 0 3 2 4 2 2 3 3 6 : 36212
aida04 : 4 6 4 4 2 3 3 3 3 3 6 : 36864
aida05 : 19 5 4 2 3 3 1 3 3 3 6 : 36404
aida06 : 24 9 5 2 3 3 3 2 3 3 6 : 36472
aida07 : 20 4 9 0 2 3 2 2 3 3 6 : 36096
aida08 : 22 9 4 3 1 3 1 3 3 3 6 : 36352
aida09 : 20 7 3 2 2 3 3 2 3 3 6 : 36344
aida10 : 25 9 9 5 2 3 2 3 3 3 6 : 36828
aida11 : 24 11 2 0 1 3 1 3 3 3 6 : 36248
aida12 : 26 5 4 2 1 3 2 2 3 3 6 : 36048 |
|
Fri Mar 22 08:31:39 2024, TD, Friday 22 March 39x
|
09.30 Systems check
CAEN N1419ET only channel #0 ON, all other channels off
DSSSD bias -120V leakage current -5.500uA - attachment 1
Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )
FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine
All system wide checks OK *except*
aida04 WR decoder status 0x10 - attachment 3
WR timestamps OK - attachment 4
09.42 NH reports "195au implanting in aida, Ca 100 per spill"
ASIC settings
LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity
09.46 all histograms and stats zero'd
ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9
per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
aida08 noise significantly lower than all other FEE64s
per FEE64 1.8.H spectra - attachments 12-13
data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss
per FEE64 1.8.L spectra - attachments 14-15
per FEE64 Rate Stat spectra - attachments 16-19
Merger, TapeServer - attachments 20-21
Merger idle !?
Tape Server no storage mode but forwarding data at c. 1Mb/s
data file R31
13.30 NH reports "beam over"
DSSSD bias -120V leakage current -6.500uA - attachment 22
FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine
All system wide checks OK *except*
aida05 & aida07 FPGA timestamp errors - attachment 24
aida04 WR decoder status 0x10 - attachment 25
WR timestamps OK - attachment 26
ADC data items stats - attachments 27
per FEE64 Rate spectra - attachments 28-29
per FEE64 1.8.L spectra - attachments 30-31
per FEE64 1.8.H spectra - attachments 32-33
per FEE64 1.8.W spectra - 20us FSR - attachments 34-35
Merger, TapeServer - attachments 36-37
Merger idle !?
Tape Server no storage mode but forwarding data at c. 1Mb/s
data file R31
20:30 AIDA Powered down
DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)
Merger still showing idle... "no data to storage" makes xfer Links disappear?
Tape server stopped
11.20 Saturday 23 March
Grafana DSSSD bias/leakage current monitor screenshot - attachment 38
https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now
DSSSD#1 leakage current recovered to c. pre-beam values
08.05 Monday 25 March
Grafana DSSSD bias/leakage current monitor screenshot - attachment 39 |
|
Fri May 21 15:03:48 2021, CA, Friday 21st May 16:00 - 00:00 shift 17x
|
16:00 CA takes over
DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4
PULSER SETTINGS
---------------------------
Pulse is ON
Positive Tail Pulse
Trigger Source is Internal Clock
Trigger Threshold is 3.5
Amplitude : 2.0 Volts
Rep Rate : 2.0 hZ
Delay : 250.0 ns
Fall Time : 1 ms
Attenuation : 1
Display is : Volts
Equivalent keV is : 200.0
Ramp Start at 0.01 Volts
Ramp Stop at 9.99 Volts
Ramp Start at 1.0 keV
Ramp Stop at 999.0 keV
Ramp Time is 60 seconds
# Ramp Cycles is 1
16:14 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x570 : 112
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
16:17 temperatures ok - attachment 1
statistics ok - attachment 2
detector bias / leakage currents ok - attachment 3
analysis of R14_1625 - FEE7 deadtime at 4.5%
18:06 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x571 : 113
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
18:07 temperatures ok - attachment 4
statistics ok - attachment 5
detector bias / leakage currents ok - attachment 6
analysis of R14_1662 - FEE7 deadtime at 4.3%
19:07 DAQ continues ok - writing to file R14_1682
20:07 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x572 : 114
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
20:10 temperatures ok - attachment 7
statistics ok - attachment 8
detector bias / leakage currents ok - attachment 9
analysis of R14_1703 - max deadtime at 4.0% - attachment 10
20:35 rate spectra - attachment 11
HEC spectra - attachment 12 & 13
20:36 all histograms zeroed
20:37 DAQ continues ok - writing to file R14_1712
20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.
~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files
21:12 no beam - writing to file R14_1723
21:17 beam is back
22:07 system wide checks all ok - WR decoder fault same as entry at 20:07
22:08 temperatures ok - attachment 14
statistics ok - attachment 15
detector bias / leakage currents ok - attachment 16
analysis of R14_1740 - max deadtime at 4.2% - attachment 17
|
|
Fri May 21 07:03:50 2021, OH, TD, Friday 21 May 08:00-16:00 9x
|
08:00 OH and TD take over
Current settings p+n sides at 0xc
DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4
PULSER SETTINGS
---------------------------
Pulse is ON
Positive Tail Pulse
Trigger Source is Internal Clock
Trigger Threshold is 3.5
Amplitude : 2.0 Volts
Rep Rate : 2.0 hZ
Delay : 250.0 ns
Fall Time : 1 ms
Attenuation : 1
Display is : Volts
Equivalent keV is : 200.0
Ramp Start at 0.01 Volts
Ramp Stop at 9.99 Volts
Ramp Start at 1.0 keV
Ramp Stop at 999.0 keV
Ramp Time is 60 seconds
# Ramp Cycles is 1
08:29 System wide checks ok - WR difference at 111
Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
Temperature ok - attachment 2
Bias and leakage current ok - attachment 3
09:58 System wide checks ok - WR difference at 111
Statistics ok - attachment 4
Temperature ok - attachment 5
Bias and leakage current ok - attachment 6
Max deadtime this morning has been around 5.5% on FEE8
12:32 System wide checks ok - WR difference at 111
Statistics ok - attachment 7
Temperature ok - attachment 9
Bias and leakage current ok - attachment 8 |
|
Fri Feb 21 09:39:38 2020, TD, NH, Friday 21 February 7x
|
Attachment 1 - layout 2 - stat spectra
Attachment 2 - detector biases & leakage currents - OK
Attachment 3 - FEE64 temperatures - OK
Attachments 4-7 - TapeServer, Merger, Merger Statistics, good event statistics - OK
10:40 ASIC check load x2
13.12 DAQ stopped c. 10.50
unable to connect messages for multiple FEE64s
After lunch powwer-cycled FEE64s and restarted DAQ
aida03 crashed (see https://elog.ph.ed.ac.uk/DESPEC/131 ) and rebooted by issuing reboot command as root
Data file 20FEB20/R1
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa -> 0x64
BNC PB-5 pulser OFF
continue alpha background run |
|
Fri May 20 16:22:01 2022, TD, Friday 20 May
|
17.20 DAQ stopped
Attachment 1 - DSSSD bias & leakage current
Attachment 2 - grafana DSSSD bias, leakage current & temp - OK
Attachment 3 - FEE64 temps - OK |
|