 Thu May 20 07:03:02 2021, CA, Thursday May 20th 08:00 - 16:00 shift 40x
|
08:00 CA takes over
08:04 DAQ continues ok - writing to file R14_989
09:30 all system wide checks ok *except* WR decoder status;
Base Current Difference
aida05 fault 0x500 : 0x56a : 106
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
09:34 temperatures ok - attachment 1
stats ok - attachment 2
bias/leakage currents ok - attachment 3
R14_1016 analysis - max deadtime ~4.3% - attachment 4
11:30 all system wide checks ok *except* WR decoder status;
Base Current Difference
aida05 fault 0x500 : 0x56b : 107
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
11:34 temperatures ok - attachment 5
stats ok - attachment 6
bias/leakage currents ok - attachment 7
R14_1052 analysis - max deadtime ~3.6% - attachment 8
13:05 beam optimisation currently taking place
DAQ continues ok and writing to file R14_1085
13:17 no beam
13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30.
13:31 temperatures ok - attachment 9
stats ok - attachment 10
bias/leakage currents ok - attachment 11
R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12
13:37 beam is back - rate spectra - attachment 13
DAQ ok - writing to file R14_1088
13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14
14:21 beam off - issue with UNILAC
DESPEC also report issues with their DAQ
AIDA continues to forward data to MBS ok - lower rates while beam off
14.30 1.8.H spectra - attachments 16-18
1.8.L spectra - attachments 19-22
aida01 pulser peak width 99 ch FWHM
Rate spectra - attachment 23
Grafana - DSSSD bias & leakage currents past 7 days - attachment 24
DSSSD bias & leakage currents - attachment 25
Merger/Tape Server/Merger stats - attachment 26
14.41 All histograms & stats zero'd
14.55 Lost Activity Monitors - attachments 27, 28 & 35
Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler
15:00 beam is back
15:05 AIDA writing to file R14_1108
ADC data items - attachment 36
15:27 system wide checks ok *except* WR decoder status
Base Current Difference
aida05 fault 0x500 : 0x56c : 108
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
15:30 temperatures ok - attachment 37
stats ok - attachment 38
bias/leakage currents ok - attachment 39
R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40 |
|
Thu Mar 7 11:09:05 2024, HA, JB, CC, TD, MG, Thursday 7 March
|
Snout assembly
DSSSDs
Upstream 3208-10/3208-18/3208-20
Downstream 3131-5/3131-10/3131-12
p+n junction side bias continuity checked OK for *all* wafers
all c. 22 Ohm as expected
n+n Ohmic side bias continuity checked OK for both DSSSDs
upstream 73 Ohm, downstream 92 Ohm
n+n Ohmic side bias wafer #2
positioned at top of snout assembly ( lower stage snout side marked 'T' )
p+n junction side faces downstream
Distances
Using top edge of lower stage snout as reference
ref - middle of upstream bPlas 8.0cm
ref - upstream DSSSD 11.0cm
ref - downstream DSSSD 12.0cm
ref - middle of downstream bPlas 14.0cm
Measurements consistent to +/-1mm between LHS and RHS of snout assembly as viewed from top side
13.00 At this position the Kapton PCBs bulged in/out and there was some concern that when the upper stage
snout was installed it would the Kapton PCBs into the 'active' area. It was decided to move the 2x bPlas
and 2x DSSSDs c. 5mm downstream.
Jeroen and Phillip cut sections from 4x PEEK 6mm spacers so that they would clip onto 3mm dia support rods.
We raised the assembly fro the inter stage and inserted the modified spacers between the inter stage and
the other spacers above. The Kapton PCBs now run fairly straight to the DSSSD connectors.
Distances re-measured as
Using top edge of lower stage snout as reference
ref - middle of upstream bPlas 8.6cm
ref - upstream DSSSD 11.5cm
ref - downstream DSSSD 12.4cm
ref - middle of downstream bPlas 14.5cm |
|
Thu Apr 7 12:42:14 2022, OH, NH, TD, Thursday 7 April 8x
|
Re-assembled triple snout re-installed on the AIDA stand - DSSSD ribbon cables and grounds re-connected.
FEE64 adaptor PCB LKs per https://elog.ph.ed.ac.uk/DESPEC/409
Attachments 1 & 2 1.8.L spectra
p+n FEE64 aida10 pulser peak width 97 ch FWHM
n+n FEE64 aida04 pulser peak width 550 ch FWHM
Attachment 3 rate spectra
missing channels aida01, aida07 & aida11 - misaligned ERNI and/or Samtec connectors?
Attachments 4 & 5 1.8.W spectra
Attachment 6 ADC data item rates
aida10 OK, all others c. 300-400k
Attachment 7 FEE64 temperatures OK
Attachment 8 DSSSD bias & leakage currents OK |
|
Thu Jun 6 11:13:29 2024, TD, Thursday 6 June 9x
|
12.09 DSSSD bias & leakage current - attachment 1
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
8x < 20k, max 247k aida08
per FEE64 Rate spectra - attachment 4
12.16 all histograms & stats zero'd
21.08 DSSSD bias & leakage current - attachments 5-6
FEE64 temperatures OK - attachment 7
ADC data item stats - attachment 8
8x < 20k, max 230k aida08
per FEE64 Rate spectra - attachment 9
|
|
Fri Mar 5 22:53:05 2021, OH, Thursday 5th of March
|
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.
The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.
Beam plug is in front of AIDA. This is causing a huge spike in the Ge. |
|
Thu Oct 31 09:00:58 2019, CA, TD, NH, Thursday 31 October 14x
|
10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%
10.15 DSSSD stack as follows
DSSSD serial thickness depletion
# (um) voltage (V)
1 3208-7 1019 125 upstream
2 3208-13 1020 120
3 3208-14 1020 130 downstream
11.30 AIDA power relays sequenced on
DAQ start
system wide checks ok
11.33 attachment 2 - fee temperatures - ok
attachment 3 - good event statistics with slow comparator threshold = 100
attachment 4 - good event statistics with slow comparator threshold = 50
attachment 5 - good event statistics with slow comparator threshold = 20
attachment 6 - good event statistics with slow comparator threshold = 10
13.51 attachment 7 - Rates
- aida09 1 ASIC not working (no data)
attachment 8 - good event statistics
attachments 9/10 - 1.8.L spectra
- no peak observed in aida09
attachments 11/12 - 1.8.W spectra
- waveform only observed in naida 2,4,7,8,12
pulser peak widths - FEE width(ch)
1 71
2 96
3 66
4 79
5 74
6 96
7 89
8 83
9 no peak
10 82
11 153
12 120
17.26 FATIMA moved upstream around AIDA
attachment 13 - good event statistics
ASIC check-load performed
attachment 14 - good event statistics following checkload |
|
Thu Jun 3 10:16:39 2021, TD, Thursday 3 June 8x
|
11.19 Hot HEC channels multiple FEE64s
ASIC check fixes
System wide checks OK * except*
Base Current Difference
aida01 fault 0x8940 : 0x8941 : 1
aida02 fault 0x9b32 : 0x9b33 : 1
aida03 fault 0xf266 : 0xf267 : 1
aida04 fault 0x579b : 0x579c : 1
White Rabbit error counter test result: Passed 12, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 11 4 2 2 2 2 2 3 2 4 7 : 41516
aida02 : 10 7 2 0 1 2 2 3 2 4 7 : 41408
aida03 : 10 8 2 2 2 4 1 4 2 3 7 : 40008
aida04 : 5 4 4 2 2 3 1 4 2 3 7 : 39860
aida05 : 12 3 2 2 1 2 2 3 2 4 7 : 41448
aida06 : 15 9 2 2 3 2 1 3 2 4 7 : 41380
aida07 : 12 8 1 2 2 3 3 2 2 4 7 : 41408
aida08 : 9 6 3 1 2 3 3 2 2 4 7 : 41380
aida09 : 15 6 1 1 2 2 2 3 2 4 7 : 41500
aida10 : 15 6 3 2 1 3 3 2 1 4 7 : 40348
aida11 : 8 2 1 1 1 3 1 3 2 4 7 : 41248
aida12 : 18 3 2 1 3 3 2 2 2 4 7 : 41184
aida13 : 7 4 1 4 3 4 3 3 1 4 7 : 41100
aida14 : 18 6 1 4 3 4 2 2 1 4 7 : 40392
aida15 : 7 7 1 3 1 2 1 3 2 4 7 : 41220
aida16 : 4 4 2 3 3 3 2 4 1 4 7 : 41200
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Wed Jun 02 16:27:30 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
1.8.W spectra - 20us FSR - attachments 2 & 3
ADC data items - attachment 4
FEE64 temperatures OK - attachment 5
DSSSD bias & leakage currents OK - attachment 6
Merger/Tape Server/Merger statistics - attachment 7
no merger errors reported since previous restart
11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36
13.09 analysis of file S496/R31_16 - attachment 8
max deadtime 0.31% aida04 |
|
Thu May 27 12:25:53 2021, TD, Thursday 27 May 9x
|
13.29 DAQ continues - file S496/R18_443
alpha background
All system wide checks OK *except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b2 : 2
aida02 fault 0xfe72 : 0xfe74 : 2
aida03 fault 0x97f4 : 0x97f6 : 2
aida04 fault 0x950 : 0x952 : 2
aida05 fault 0x8e93 : 0x8e95 : 2
aida06 fault 0x545a : 0x545c : 2
aida07 fault 0xc04d : 0xc04f : 2
aida08 fault 0xa0f7 : 0xa0f9 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 2 7 7 1 3 2 2 2 2 4 6 : 37008
aida02 : 3 1 5 1 3 4 3 3 2 4 6 : 37956
aida03 : 2 6 5 6 2 4 2 3 3 3 6 : 36808
aida04 : 24 12 5 5 4 3 2 3 3 3 6 : 36912
aida05 : 8 1 9 3 2 2 2 2 2 4 6 : 37016
aida06 : 3 6 2 2 2 2 2 3 3 4 6 : 38428
aida07 : 6 17 1 2 0 3 3 2 2 4 6 : 37232
aida08 : 4 2 1 2 3 4 3 3 2 4 6 : 37936
aida09 : 15 7 5 2 3 4 3 3 1 4 6 : 37060
aida10 : 7 6 9 2 0 3 3 3 1 4 6 : 36764
aida11 : 19 10 2 0 2 3 3 3 1 4 6 : 36796
aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724
aida13 : 17 9 2 0 0 4 2 2 2 4 6 : 37036
aida14 : 15 4 7 1 1 3 2 2 2 4 6 : 37036
aida15 : 9 6 1 2 1 3 3 4 2 4 6 : 38244
aida16 : 11 11 8 2 0 3 2 2 2 4 6 : 37060
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.30 analysis of file S496/R18_442 - attachment 9
max deadtime 0.28% aida04 |
|
Thu Jun 23 15:10:17 2022, Philippos Papadakis, Thursday 23 June 16:00-24:00 15x
|
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc525 : 39
aida08 fault 0xf0e9 : 0xf166 : 125
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x1c : 11
aida08 fault 0x1a : 0x23 : 9
Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)
18:05
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc527 : 41
aida08 fault 0xf0e9 : 0xf177 : 142
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x1c : 11
aida08 fault 0x1a : 0x24 : 10
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)
20:22
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc529 : 43
aida08 fault 0xf0e9 : 0xf17f : 150
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x20 : 15
aida08 fault 0x1a : 0x25 : 11
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)
21:56
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc529 : 43
aida08 fault 0xf0e9 : 0xf186 : 157
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x21 : 16
aida08 fault 0x1a : 0x26 : 12
Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)
23:50
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc52e : 48
aida08 fault 0xf0e9 : 0xf18c : 163
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x21 : 16
aida08 fault 0x1a : 0x27 : 13
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15) |
|
Thu Jun 23 06:56:02 2022, OH, Thursday 23 June 08:00-16:00 11x
|
07:56 Taken over from AM
Current White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc51a : 28
aida08 fault 0xf0e9 : 0xf132 : 73
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
FPGA Timestamp error counter test result: Passed 6, Failed 2
Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link
Statistics ok - attachment 1
FEE temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
Analysis of file R3_93 - attachment 4
Still deadtime in aida02 of ~15%
Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
Currently 2.2 TB of space on /media/SecondDrive
Current data rate of 6.23MB/s
At current data rate this should last 4 days
08:27 Currently no beam
Problem with Alvarez 3 now
Tomorrow will have a 1 hour break due to changing of the source
09:36 Beam back
09:59 System wide checks
Base Current Difference
aida07 fault 0xc4fe : 0xc51b : 29
aida08 fault 0xf0e9 : 0xf138 : 79
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
Stats ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
11:14 FRS group is changing from Hg setting back to Pt
11:36 Back on Pt R3_129
14:14 Stats ok - attachment 8
Temperature ok - attachment 9
Bias and leakage currents ok - attachment 10
System wide checks:
WR:
Base Current Difference
aida07 fault 0xc4fe : 0xc523 : 37
aida08 fault 0xf0e9 : 0xf156 : 109
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x11 : 0x1b : 10
aida08 fault 0x1a : 0x20 : 6
Analysis of R3_156 - attachment 11 |
|
Wed Jun 22 23:18:45 2022, AM, OH, TD, MA, Thursday 23 June 19x
|
00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
Attempted to recover restarting merger but this caused 01 to drop out.
00:10 DAQ recovered and AIDA01 recovered with a reboot
01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good
03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good
05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good
07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good |
|
Thu Apr 22 11:03:57 2021, JS, Thursday 22nd April: 12:00-16:00 12x
|
12:00 Taking over from Liam, all is well.
12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz
12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they
need to check and will need apprx. 15 min.")
12:30
ucesb ok (no beam)
DB No faults
Stats ok.
12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.
13:00
We are taking intermittent beam while they work on the FRS
DB No faults
ucesb ok elog:261/1
Current ok elog:261/2
Stats ok elog:261/3
Temps ok elog:261/4
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
Base Current Difference
aida12 fault 0x0 : 0x46 : 70
FPGA Timestamp error counter test result: Passed 11, Failed 1
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 6 5 3 2 2 2 3 2 3 7 : 39580
aida02 : 21 7 4 2 2 3 2 3 2 3 7 : 39692
aida03 : 20 10 3 3 0 3 2 3 2 3 7 : 39600
aida04 : 8 7 4 5 2 4 3 2 2 3 7 : 39608
aida05 : 17 14 1 2 3 3 4 2 2 3 7 : 39748
aida06 : 17 10 2 3 2 2 4 2 2 3 7 : 39572
aida07 : 27 10 8 7 2 3 3 2 2 3 7 : 39708
aida08 : 9 6 2 3 2 1 3 3 2 3 7 : 39636
aida09 : 1 2 1 4 0 3 3 2 2 3 7 : 39204
aida10 : 5 8 4 4 2 2 1 3 2 3 7 : 39316
aida11 : 6 4 5 3 2 3 2 2 2 3 7 : 39144
aida12 : 15 5 8 1 0 4 1 3 2 3 7 : 39428
13:32
ucesb ok
DB No faults
Stats ok
14:01
ucesb ok
DB No faults
Stats ok
14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
cat /var/log/messages | grep /MIDAS - attachments 6 & 7
Grafana DSSSD bias & leakage currents for last 7 days - attachment 8
14:55
DB No faults
ucesb ok
Current ok elog:261/9
Stats ok elog:261/10
Temps ok elog:261/11
FPGA Timestamp Check Error:
Base Current Difference
aida12 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 11, Failed 1
15:30
ucesb ok
DB No faults
Stats ok
15:59
ucesb ok elog:261/12
DB No faults
Stats ok |
|
Thu Apr 22 07:06:37 2021, LPG, Thursday 22nd April: 08:00-12:00 8x
|
08:01 - Taking over from Corrigan...
ucesb ok!
DB ok!
Stats ok!
08:30
ucesb ok!
DB ok!
Stats ok!
09:15
ucesb ok! elog:260/4
DB ok!
Stats ok!
Leakage currents ok! elog:260/1
Temperatures ok! elog:260/2
MIDAS Rates ok! elog:260/3
System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x40 : 64
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!
09:45
ucesb ok!
DB ok!
Stats ok!
10:30
ucesb ok!
DB ok!
Stats ok!
11:00
ucesb ok! elog:260/5
DB ok!
Stats ok!
Leakage currents ok! elog:260/6
Temperatures ok! elog:260/7
MIDAS Rates ok! elog:260/8
System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x42 : 66
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!
11:30
ucesb ok! elog:260/5
DB ok!
Stats ok!
11:55
ucesb ok!
DB ok!
Stats ok! |
|
Thu Apr 22 15:18:20 2021, TD, Thursday 22 April 16.00-00.00 36x
|
16.20 zero all histograms, statistics & merger statistics
most recent merger error messages
MERGE Data Link (29984): bad timestamp 6 3 0xc1ac7f1c 0x0dce7672 0x00002cda5dce7672 0x16782cda5dce7672 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp 6 3 0xc18b7e62 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp 6 3 0xc1957ebc 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp 6 3 0xc1b67ecb 0x0dcf0ae2 0x00002cda5dcf0ae2 0x16782cda5dcf0ae2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp 6 3 0xc1957ea3 0x0dcf12b2 0x00002cda5dcf12b2 0x16782cda5dcf12b2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp 6 3 0xc1867fba 0x0b2ee142 0x00002dc9eb2ee142 0x16782dc9eb2ee142 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp 6 3 0x86800889 0x0ad28158 0x00002dc9eb2ee142 0x16782dc9ead28158 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp 6 3 0xc1be7e6e 0x0b2ef0e2 0x00002dc9eb2ef0e2 0x16782dc9eb2ef0e2 0x16782dc9fad2c172
MERGE Data Link (29984): bad timestamp 6 3 0xc1b08006 0x0b2f5e42 0x00002dc9eb2f5e42 0x16782dc9eb2f5e42 0x16782dc9fad34502
MERGE Data Link (29984): bad timestamp 6 3 0xc1807de7 0x0ad3c342 0x00002dc9eb2f6de2 0x16782dc9ead3c342 0x16782dc9fad34502
MERGE Statistics counters now cleared
16.21 merger server terminal reset & clear
TapeData -> /media/ThirdDRive/TapeData
[npg@aidas-gsi bin64]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 13G 34G 28% /
tmpfs 7.9G 559M 7.3G 7% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 79G 302G 21% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 3.9T 2.9T 58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 6.3T 605G 92% /media/SecondDrive
/dev/sdc1 7.2T 3.6T 3.3T 52% /media/ThirdDrive
at current rate (10.5Mb/s) we have 3.6d remaining
17.40 *all* system wide checks OK *except* FPGA timestamp errors
Base Current Difference
aida12 fault 0x0 : 0x51 : 81
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Wed Apr 21 01:51:54 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
17.43 FEE64 temps OK - attachment 1
statistics - attachments 2-8
per FEE64 rate spectra - attachments 9 & 10
1.8.L spectra - attachments 11-14
1.8.H spectra - attachments 15-16
1.8.W spectra 20us FSR - attachments 17-18
1.8.W spectra 200us, 2ms & 20ms FSR - attachments 19-21, 30-32
merger, tape server & merger statistics 19-25
ucesb - attachment 26
Grafana DSSD biases & leakage currents - attachment 27
taperserver & tapeserver/merger server terminals - no timewarp errors since c. 15.25 - attachment 28
DSSSD biases & leakage currents - attachment 29
22.17 *all* system wide checks OK *except*
Base Current Difference
aida01 fault 0x3612 : 0x3613 : 1
aida02 fault 0x523c : 0x523d : 1
aida03 fault 0xc5cd : 0xc5ce : 1
aida04 fault 0x14f1 : 0x14f2 : 1
aida05 fault 0x9a4e : 0x9a4f : 1
aida06 fault 0xfdd2 : 0xfdd3 : 1
aida07 fault 0x4a0 : 0x4a1 : 1
aida08 fault 0xc796 : 0xc797 : 1
White Rabbit error counter test result: Passed 4, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x57 : 87
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Wed Apr 21 01:51:54 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
Most recent merger errors
MERGE Data Link (29979): bad timestamp 1 3 0xc05080ef 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc06f80d7 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc0778222 0x0c88b272 0x000047aa2c88b272 0x167847aa2c88b272 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc0598098 0x0c88e152 0x000047aa2c88e152 0x167847aa2c88e152 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc042815b 0x0c894eb2 0x000047aa2c894eb2 0x167847aa2c894eb2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc05980eb 0x0c899502 0x000047aa2c899502 0x167847aa2c899502 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc075813c 0x0c89e322 0x000047aa2c89e322 0x167847aa2c89e322 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp 1 3 0xc04280d5 0x0c89f2c2 0x000047aa2c89f2c2 0x167847aa2c89f2c2 0x167847aa2c8a34d2
MERGE Actor (29995): Working with 0 from 12 data sources
MERGE Actor (29995): Working with 0 from 12 data sources
22.20 DSSSD biases & leakage currents - attachment 33
FEE64 temps OK - attachment 34
statistics - attachments 35
merger statistics 36
|
|
Thu Sep 20 08:57:48 2018, OH, Thursday 20th September
|
09:57 Alpha run stoppped R2_0 -> R2_2
Thresholds dropped from 0x64 to 0xa
Pulser turned on again
System wide checks ok
FEE Temp ok - attachment 1
Bias ok, leakage at 1.485uA at 160V
Stats in similar place to yesterday - attachment 2
Low Energy pulser walkthrough
1.0V to 0.4V in 0.1V steps
2 minutes per step
Pulser at 50Hz
10:29 Run started R3_0
10:45 Run stopped R3_2
1.8.L spectra for low energy pulser walkthrough - attachment 3
11:00
High Energy pulser walkthrough
1.8V to 0.V in 0.2V
2 minutes per step
Pulser at 50Hz
11:06 Run started R4_0
10:18 Run stopped R4_2
1.8.L spectra for high energy pulser walkthrough - attachment 4 |
|
Thu May 20 14:56:33 2021, OH, Thursday 20th May 16:00 - 24:00 10x
|
16:00 p+n threshold 0xc (120keV)
n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4
n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4
Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure.
17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector
18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now.
The deadtimes in FEE6 and 8 have risen to around 6.25%.
Looking at this mornings and yesterday evenings fill times this behavior isn't observed
18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago
18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2
This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file)
18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8
18:15 System wide checks ok except WR as usual
Base Current Difference
aida05 fault 0x500 : 0x56d : 109
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics increased at 18:02 still working on - attachment 2
FEE temp - attachment 3
Bias and leakage current - attachment 4
20:51 System wide checks ok - WR same as before
Statistics still high - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
22:52 Statistics still high - attachment 8
Temp ok - attachment 9
Bias and leakage current ok - attachment 10
System wide checks ok - WR difference still 109 |
|
Thu May 2 00:16:22 2024, TD, NH, Thursday 2 May
|
01.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
6/16 < 20k
10.25 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 restarted
7/16 < 20k, max 220k aida16
all histograms zero'd
13.20 per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 59 ch FWHM
FEE64 temperatures OK
ADC data item stats
aida04 no data
5/16 < 20k, max 332k aida08
14.37
Moved Time Machine delayed scaler from aida04 to aida07
aida04 no data
Reset merger etc
aida05 no data
Reset merger etc
All FEEs with data again
S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C
17.26 FEE64 temperatures OK
ADC data item stats OK
19.28 FEE64 temperatures OK
ADC data item stats
aida02 no data - DAQ reset
|
|
Thu May 19 17:00:46 2022, TD, Thursday 19 May
|
18.01 DAQ stopped
Attachment 1 - grafana DSSSD bias, leakage current & temp - OK
Attachment 2 - DSSSD bias & leakage current
Attachment 3 - FEE64 temps - OK
Attachment 4 - ADC data item stats |
|
Thu Mar 18 11:53:10 2021, TD, Thursday 18 March 7x
|
12.40 The snout w/bPlast plus 3xDSSSDs was removed yesterday
AIDA pulser tests via adaptor PCB test inputs, i.e. zero capacitative load and zero leakage current
per FEE64 1.8.W spectra - attachments 1-4
per FEE64 Rate spectra - attachment 5
per FEE64 1.8.L spectra - attachments 6-7
From the per FEE64 1.8.L
FEE64 FWHM
1 -
2 45
3 25
4 -
5 -
6 45
7 25
8 37
9 30
10 38
11 19
12 35
For comparison we would expect to obtain ~ 15-20 ch FWHM (~10.5-14 keV FWHM) based on tests
conducted at T9, STFC DL. The peak widths observed above are significantly worse than expected.
Are the adaptor PCBs currently grounded to FEE64 Cu mezzanine cover? |
|
Thu Apr 18 08:57:48 2024, JB, TD, Thursday 18 April 36x
|
09.52 DAQ continues file S100_alpha/R5
note merger not functioning correctly due to aida03, aida04 & aida09 out of WR sequence
DSSSD bias & leakage current OK - attachment 1
FEE64 temps OK - attachment 2
*except* aida02 ASIC temp which is known to be u/s
WR timestamps - attachment 3
aida03, aida04 & aida09 out of sequence
All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 4-6
ADC data item stats - attachment 7
per FEE64 Stat spectra - attachment 8
per FEE64 1.8.L spectra - attachments 9-10
per FEE64 1.8.W spectra - 20us FSR - attachments 11-12
Merger, TapeServer & Link Data Rates - attachments 13-15
Grafana - attachment 16
DSSSD bias/leakage current trends OK
19.15 alpha background file S100_alpha/R10
DSSSD bias & leakage current OK - attachment 17
FEE64 temps OK - attachment 18
*except* aida02 ASIC temp which is known to be u/s
All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 19-21
WR timestamps OK - attachment 22
ADC data item stats - attachment 23
Merger, TapeServer & Meger stats - attachments 24-26
data rate c. 2.8Mb/s
per FEE64 1.8.W spectra - 20us FSR - attachments 27-28
correlation scaler stats & data link stats - attachments 29-30
data rate dominated by MACB scaler inputs aida03 & aida04 ( c. 200kHz each )
22.16 DAQ continues Ok file S100_alpha/R10_35
Merger, TapeServer & Meger stats - attachments 31-33
data rate c. 2.8Mb/s
ADC data item stats - attachment 34
FEE64 temps OK - attachment 35
*except* aida02 ASIC temp which is known to be u/s
DSSSD bias & leakage current OK - attachment 36 |
|