 Fri Dec 20 12:04:03 2019, NH, TD, Friday 20 December 2019 7x
|
Fig 1: DSSD Leakage currents ok
Fig 2: FEE Temps ok
Fig 3: aida08 has a few timestamp errors in merger
Fig 4, 5: GSI WR OK
Fig 6: All System wide checks passed
FIg 7: Stats good
13:11 - Alpha Run Stopped, Final File: R9_6
FEEs powered off, DSSD bias off. |
|
Fri Nov 1 10:46:13 2019, CA, TD, NH, Friday 1st November 2019 6x
|
11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
before removing aluminium foil upstream of AIDA
note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF
11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
after removing aluminium foil upstream of AIDA
- base of AIDA snout has Mylar shielding |
|
Fri Mar 19 17:16:45 2021, TD, Friday 19 March 8x
|
Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04. |
|
Fri Apr 19 13:52:34 2024, JB, CC, TD, Friday 19 April contd. 23x
|
12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3
During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )
AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process
DSSSD bias & leakage current OK - attachment 1
FEE64 temps OK - attachment 2
*except* aida02 ASIC temp - known fault
System wide checks OK *except* WR/FPGA errors - attachments 3-4
WR timestamps OK - attachment 5
ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
ADC data items 10/16 < 20k, max 143k
per FEE64 Rate spectra - attachment 11
per FEE64 1.8.W spectra - 20us FSR - attachments 12-13
15.10 Synchronise ASIC clocks
Re-calibrate ALL ADCs
ASIC settings 2024Mar27-11.25.32
Changes ( https://elog.ph.ed.ac.uk/CARME/499 )
IBias LF feedback from 0xf to 0x8
Diode link threshold from 0xbf to 0xca
New ASIC settings saved as 2024Apr19-15.22.49
All FEE64 slow comparators -> 0x14
Data file S100_alpha/R12
Pulser ealkthrough (test +)
BNC PB-5 settings
Amplitude 10.0-1.0V @ 1.0V step
Attenuation x10
Frequency 25Hz
Polarity +
Tail pulse
tau_d 1ms
16.35 DAQ ends OK file S100_alpha/R12_24
per p+n FEE64 1.8.L spectra - attachment 14
aida09 pulser peak width 54 ch FWHM
17.38 Data file S100_alpha/R13
Pulser walkthrough (test -)
BNC PB-5 settings
Amplitude 10.0-1.0V @ 1.0V step
Attenuation x10
Frequency 25Hz
Polarity -
Tail pulse
tau_d 1ms
18.43 DAQ ends OK file S100_alpha/R13_20
per p+n FEE64 1.8.L spectra - attachment 15
18.55 Current status
FEE64 power ON
DSSSD bias ON
All FEE64 slow comparator 0x14
DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )
To Do
- S4 currently closed/controlled access for FRS startup
If we have access to S4 tomorrow
- switch test - to test + ( not critical )
- test AIDA interlock
- further alpha background?
22.47 DSSSD bias & leakage current OK - attachments 16-17
FEE64 temps OK - attachment 18
*except* aida02 ASIC temp - known fault
ADC data item stats - attachment 19
per FEE64 Rate spectra - attachment 20
Data link, Tape Server & Merger - attachments 21-23
|
|
Fri Apr 19 02:35:14 2024, TD, Friday 19 April 38x
|
03.30
09.10
09.25 data file S100_alpha/R11
BNC PB-5 pulser data
amplitude 10.0V
attenuation x10
tau_d = 1ms
polarity - ( initially + )
frequency 25Hz
ASIC settings2024Mar27-11.25.32
slow comparator 0x64 all FEE64s
aida02 pulser peak width 129 ch FWHM
polsrity +
aida09 pulser peak width 57 ch FWHM
09.40 switch to no storage
p+n FEE64 slow comparators 0x64 -> 0xa
n+n FEE64 slow comparators 0x64 -> 0xf
radioactive source 152Eu 370kBq installed on the bottom side of the snout, centred tranversely, located beneath bplas-DSSSD-DSSSD-bplas position.
ADC data item stats - attachment 35
9/16 < 20k, max 108k
per FEE64 Rate spectra - attachment 36
10.44 152Eu source removed
ADC data item stats - attachment 37
9/16 < 20k, max 102k
per FEE64 Rate spectra - attachment 38 |
|
Sat Jun 18 09:46:57 2022, OH, NH, Friday 17th June 23x
|
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con
Initial jumper configuration:
FEE LK
1 2 4
2 1 2 4
3 2 3 4
4 2 4
5 2 4
6 1 2 4
7 2 3 4
8 1 2 4
13:22 LK2, 3 and 4 on all p+n - attachment 1
14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
Same configuration but waveforms disabled (p+n improvment) - attachment 3
Layout 1 - attachment 4
14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
Previously all FEEs on their own pair within a psu
New order 1,3 2,4 5,7 6,8
Waveforms on - all bad - attachment 5
Waveforms off - all p+n better - attachment 6
14:54 interlock removed from AC mains relay
Also noticed adaptor board on 2 not in fully -> Now in
p+n generally better - attachment 7
Without waveforms - p+n better again - attachment 8
16:04 Platform moved into position
Interlock power cable removed from extension lead
w/ waveform - p+n slightly worse
w/o waveform - p+n same as beffore
16:14 LK3 removed from 1 and 5
w/ waveform -> p+n slightly worse - attachment 9
w/o waveform -> p+n best yet - attachment 10
16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
Removed from 6 -> only one LK1 per DSSD
w/ waveforms - all bad - attachment 11
w/o waveforms - no change - attachment 12
16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
w/ waveform p+n better - attachment 13
w/0 waveform p+n better, n+n worse? - attachment 14
17:21 Bias filters added to n+n
w/ waveform - no change - attachment 15
w/o wwaveofrm - P=n worse - attachment 16
Stats 0x21 - attachment 17
Stats 0x14 - attachment 18
Stats 0x12 - attachment 19
17:42 Bias filter removed and bias floated (LK1 removed)
w/ waveform all worse - attachment 20
w/o waveform -> p+n worse, n+n similar
18:29 LK1 back on 4+8
w/ waveform -> poor
w/0 waveform - recovered previous - attachment 20
18:32 LK 2 and 4 removed from 2 +6
Also noticed ground lemo on aida02 not fully inserted which has been corrected
w/ waveform poor - attachment 21
Threshold 0xF - attachment 22
w/o waveform - attachment 23 |
|
Fri Apr 16 07:19:00 2021, OH, MA, Friday 16th April 08:00-16:00 14x
|
08:19 System wide checks all ok
Baselines reset on WR and FPGA checks for the start of the day
Statistics ok - attachment 1
Temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
No corruptions have been noted since
10:10 System check, clcock and ADC ok
Base Current Difference
aida07 fault 0xfb3a : 0xfb3d : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA and memory check ok
Statistics ok - attachment 4
Temperatures ok - attachment 5
Bias and leakage currents ok - attachment 6
10:39 Fission fragments were punching through AIDA
This was even with the maxmimum degraders in place
They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.
11:38 They have found the issue with the degrader and have corrected it.
We now see next to no implants in AIDA as expect.
Histograms reset at around 11:35
Layout 2 shows very few implants in 5 minutes - attachment 7
12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
This was a large amount of dose going into the detectors which we should try to avoid
13:25 system check, clock and ADC ok
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics ok - attachment 9
Temperatures ok - attachment 10
Bias and leakage currents ok - attachment 11
FPGA and memory check ok
16:10 system check, clock and ADC ok
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics ok - attachment 12
Temperatures ok - attachment 13
Bias and leakage currents ok - attachment 14
FPGA and memory check ok |
|
Fri May 14 19:01:09 2021, JS, Friday 14th May 20:00-00:00 27x
|
20:00 Taking over from CB
20:33 Doing full checks.
Temps ok elog:319/8
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit -
Base Current Difference
aida01 fault 0xf932 : 0xf934 : 2
aida02 fault 0x62ec : 0x62ee : 2
aida03 fault 0x8679 : 0x867b : 2
aida04 fault 0xf0e4 : 0xf0e6 : 2
aida05 fault 0x9db8 : 0x9dc5 : 13
aida06 fault 0x7f18 : 0x7f1a : 2
aida07 fault 0xdd2c : 0xdd2e : 2
aida08 fault 0x1557 : 0x1559 : 2
White Rabbit error counter test result: Passed 8, Failed 8
FPGA Timestamp errors -
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 35 34 30 8 1 4 1 2 2 3 6 : 35772
aida02 : 37 30 28 5 3 1 2 4 2 3 6 : 36644
aida03 : 41 29 30 16 4 3 1 5 1 3 6 : 36588
aida04 : 13 6 23 12 4 4 4 4 2 2 6 : 35412
aida05 : 20 32 23 10 2 2 2 2 1 4 6 : 36736
aida06 : 14 5 10 5 1 3 2 3 2 3 6 : 35680
aida07 : 44 23 22 9 4 3 1 3 2 3 6 : 36200
aida08 : 39 30 28 9 0 3 2 3 3 2 6 : 35308
aida09 : 41 31 28 10 2 5 1 3 1 3 6 : 35484
aida10 : 43 23 27 7 3 1 1 3 3 3 6 : 36916
aida11 : 39 35 17 10 3 3 2 2 2 3 6 : 35908
aida12 : 37 36 15 10 2 2 2 4 3 2 6 : 35684
aida13 : 47 32 17 7 4 2 3 4 3 2 6 : 36012
aida14 : 23 33 14 10 3 4 3 4 3 2 6 : 36164
aida15 : 44 28 24 12 3 2 4 3 3 2 6 : 35920
aida16 : 22 17 24 17 3 4 2 3 1 3 6 : 35648
Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27
Resume info #3 - elog:319/3 same as elog:317/26
Pause info #2 - elog:319/4 same as elog:317/32
AIDA disk info #6 - elog:319/5 same as elog:317/31
AIDA ADC data items - elog:319/6 same as elog:317/30
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see
clearly once uploaded the screenshots]
Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%
21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay
21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit -
Base Current Difference
aida01 fault 0xf932 : 0xf934 : 2
aida02 fault 0x62ec : 0x62ee : 2
aida03 fault 0x8679 : 0x867b : 2
aida04 fault 0xf0e4 : 0xf0e6 : 2
aida05 fault 0x9db8 : 0x9dc5 : 13
aida06 fault 0x7f18 : 0x7f1a : 2
aida07 fault 0xdd2c : 0xdd2e : 2
aida08 fault 0x1557 : 0x1559 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 39 34 29 8 1 4 1 2 2 3 6 : 35772
aida02 : 46 29 28 5 3 2 2 3 2 3 6 : 36288
aida03 : 39 25 29 16 4 2 2 5 1 3 6 : 36660
aida04 : 14 6 19 12 4 4 3 4 2 2 6 : 35096
aida05 : 20 30 24 10 2 2 2 2 1 4 6 : 36736
aida06 : 26 4 11 5 2 4 3 2 2 3 6 : 35672
aida07 : 46 22 22 9 4 3 1 3 2 3 6 : 36200
aida08 : 28 32 28 9 0 2 2 4 3 2 6 : 35664
aida09 : 32 30 29 10 3 5 2 3 1 3 6 : 35776
aida10 : 39 23 25 7 3 1 1 3 3 3 6 : 36868
aida11 : 40 31 21 10 3 3 1 4 1 3 6 : 35688
aida12 : 17 31 12 10 2 2 3 4 3 2 6 : 35772
aida13 : 35 28 22 7 4 2 3 4 3 2 6 : 36012
aida14 : 30 33 14 10 3 4 3 4 3 2 6 : 36192
aida15 : 34 26 24 12 3 2 4 3 3 2 6 : 35864
aida16 : 38 24 27 17 3 4 2 3 1 3 6 : 35816
Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14
Aida Correlation Info #8 - elog:319/15 - rates showed zero, checked again and normal elog:319/17
Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%
22:00
Temps ok
Stats ok
ucesb ok
22:35
Temps ok
Stats ok
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up
again elog:319/17
22:54 They are closing the file, we are on R5_155
23:00 |
|
Fri May 14 14:57:32 2021, CB + OH, TD, NH, Friday 14th May 16:00-20:00 33x
|
16:00 Shift taken over from CA
16:00
Temperatures OK. Attach 1.
Upon reloading aida11 first returns "No respose" for all values.
Upon refreshing, aida11 responds but aida 9 returns "No response".
Upon refreshing again, attach 1.
Bias OK. Attach 2.
Statistics OK. Attach 3-7
System-wide checks
Clock OK
ADC - all fail
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc3 : 11
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 32 28 25 7 1 4 1 3 2 3 6 : 36112
aida02 : 43 28 29 5 4 2 3 4 1 3 6 : 36092
aida03 : 39 26 28 5 4 3 2 5 1 3 6 : 36428
aida04 : 9 5 21 8 5 4 4 4 2 2 6 : 35292
aida05 : 46 32 26 7 2 2 2 2 2 3 6 : 35768
aida06 : 4 5 16 1 3 4 2 4 1 3 6 : 35352
aida07 : 47 21 25 5 4 3 1 4 1 3 6 : 35604
aida08 : 62 33 32 6 0 2 3 3 3 2 6 : 35520
aida09 : 33 25 28 7 3 5 2 4 1 3 6 : 36140
aida10 : 47 29 22 6 3 2 2 2 2 3 6 : 35716
aida11 : 39 30 29 6 2 3 1 3 2 3 6 : 36124
aida12 : 38 36 20 6 3 2 3 3 3 2 6 : 35448
aida13 : 42 29 24 3 4 2 3 3 3 2 6 : 35440
aida14 : 31 28 18 7 3 3 4 4 2 2 6 : 35228
aida15 : 48 25 30 5 3 2 4 4 2 2 6 : 35272
aida16 : 37 16 28 6 3 3 2 2 2 3 6 : 35796
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time.
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20
17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17
System-wide checks
Clock - all pass
ADC - all pass
WR
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc3 : 11
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 56 35 27 8 1 4 1 2 2 3 6 : 35816
aida02 : 43 31 27 5 3 1 3 3 2 3 6 : 36404
aida03 : 45 28 28 16 4 2 2 4 1 3 6 : 36180
aida04 : 2 2 18 12 4 3 3 4 2 2 6 : 34872
aida05 : 28 28 24 10 2 2 2 2 1 4 6 : 36752
aida06 : 28 11 9 5 2 3 3 3 1 3 6 : 35064
aida07 : 52 24 23 9 4 3 1 2 2 3 6 : 35744
aida08 : 21 26 27 8 0 2 3 3 3 2 6 : 35284
aida09 : 40 32 26 10 3 5 1 3 1 3 6 : 35520
aida10 : 46 23 27 7 3 1 1 3 3 3 6 : 36928
aida11 : 49 31 20 10 2 2 1 2 2 3 6 : 35516
aida12 : 39 35 17 9 2 2 3 4 3 2 6 : 35940
aida13 : 40 32 20 7 4 2 3 4 3 2 6 : 36032
aida14 : 33 33 16 10 3 4 3 4 3 2 6 : 36236
aida15 : 53 27 25 12 3 2 4 4 2 2 6 : 35452
aida16 : 35 22 32 15 3 4 3 3 1 3 6 : 36060
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24
System-wide checks
Clock - all pass
ADC - all pass
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc3 : 11
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 41 31 31 8 1 4 1 2 2 3 6 : 35788
aida02 : 42 31 29 5 3 2 4 4 1 3 6 : 36304
aida03 : 29 30 29 16 4 3 2 4 1 3 6 : 36276
aida04 : 28 4 20 12 4 3 4 4 2 2 6 : 35280
aida05 : 25 28 23 10 2 2 2 2 1 4 6 : 36724
aida06 : 5 3 10 5 2 4 2 2 2 3 6 : 35308
aida07 : 38 26 22 9 4 3 1 2 2 3 6 : 35688
aida08 : 52 30 28 8 0 2 2 3 3 2 6 : 35200
aida09 : 22 26 21 10 3 4 1 4 1 3 6 : 35704
aida10 : 37 28 25 7 3 1 1 3 3 3 6 : 36900
aida11 : 53 30 20 10 2 3 2 2 2 3 6 : 35908
aida12 : 37 34 17 10 2 2 3 4 3 2 6 : 35956
aida13 : 47 34 16 7 4 2 3 4 3 2 6 : 36012
aida14 : 37 29 17 10 3 4 3 4 3 2 6 : 36236
aida15 : 44 28 25 12 3 2 4 4 2 2 6 : 35424
aida16 : 40 21 33 15 3 4 1 3 1 3 6 : 35576
18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%
19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32
System-wide checks
Clock, ADC - all pass
WR
Base Current Difference
aida01 fault 0xf932 : 0xf934 : 2
aida02 fault 0x62ec : 0x62ee : 2
aida03 fault 0x8679 : 0x867b : 2
aida04 fault 0xf0e4 : 0xf0e6 : 2
aida05 fault 0x9db8 : 0x9dc4 : 12
aida06 fault 0x7f18 : 0x7f1a : 2
aida07 fault 0xdd2c : 0xdd2e : 2
aida08 fault 0x1557 : 0x1559 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 41 33 30 8 1 4 1 2 2 3 6 : 35788
aida02 : 51 30 30 5 3 2 3 4 1 3 6 : 36092
aida03 : 52 31 28 16 4 2 2 5 1 3 6 : 36744
aida04 : 6 8 20 12 4 4 3 4 2 2 6 : 35096
aida05 : 30 31 22 10 2 2 2 2 1 4 6 : 36752
aida06 : 12 6 10 5 2 3 3 2 2 3 6 : 35488
aida07 : 53 24 21 9 4 3 1 3 2 3 6 : 36228
aida08 : 36 28 28 9 0 2 3 3 3 2 6 : 35408
aida09 : 27 28 25 10 3 4 2 3 1 3 6 : 35548
aida10 : 50 23 26 7 3 1 1 3 3 3 6 : 36928
aida11 : 42 28 22 10 3 3 2 2 2 3 6 : 35944
aida12 : 39 35 15 10 2 2 2 4 3 2 6 : 35684
aida13 : 43 34 17 7 4 2 3 4 3 2 6 : 36012
aida14 : 42 32 16 10 3 4 3 4 3 2 6 : 36264
aida15 : 36 30 25 12 3 2 4 3 3 2 6 : 35920
aida16 : 40 21 25 15 3 4 2 3 1 3 6 : 35704
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
19:40 Analysed R5_86. Attached. Dead time as before.
19:57 Shift handed over to James S |
|
Fri May 14 07:05:08 2021, CA, Friday 14th May 08:00 - 16:00 shift 24x
|
08:00 CA takes over
08:28 Problem at UNILAC - no beam
09:30 all system wide checks ok *except*
all FEE64 fail ADC calibration (no waveforms)
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc2 : 10
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp errors:
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
09:36 FEE64 Temperatures ok - attachment 1
Detector bias / leakage currents ok - attachment 2
Statistics - attachments 3-8
09:50 beam is back - writing to file R4_265
09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA
10:10 FRS increase degrader thickness
10.28 all histograms, stats and merger stats zero'd
10.50 all system wide checks ok *except*
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc2 : 10
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 3 2 2 2 2 2 2 4 3 3 6 : 36860
aida02 : 7 3 5 1 2 5 2 4 3 3 6 : 37284
aida03 : 7 5 3 1 4 4 2 3 3 3 6 : 36756
aida04 : 3 1 20 4 2 3 3 5 1 3 6 : 36052
aida05 : 19 25 26 6 2 2 3 2 2 3 6 : 35828
aida06 : 1 3 16 1 3 4 1 5 1 3 6 : 35580
aida07 : 6 4 1 2 3 3 2 3 3 3 6 : 36552
aida08 : 8 5 2 3 2 3 2 2 2 4 6 : 37064
aida09 : 9 7 1 2 2 4 2 3 3 3 6 : 36652
aida10 : 2 5 3 1 1 4 2 3 3 3 6 : 36544
aida11 : 16 4 4 2 1 2 2 3 3 3 6 : 36384
aida12 : 2 1 1 3 1 2 2 3 3 3 6 : 36288
aida13 : 0 1 1 2 0 4 3 2 3 3 6 : 36184
aida14 : 6 3 2 0 2 3 2 3 3 3 6 : 36432
aida15 : 25 8 2 1 2 3 1 2 2 4 6 : 36836
aida16 : 3 5 1 2 2 2 3 2 3 3 6 : 36100
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
10.50 DAQ continues S496/R4_304
11:18 FEE64 Temperatures ok - attachment 9
Detector bias & leakage currents ok - attachment 10
statistics - attachments 11-16
15:00 FEE64 avg. CPU usage (%)
1 52
2 55
3 55
4 60
5 93
6 50
7 53
8 55
9 55
10 51
11 50
12 67
13 56
14 55
15 55
16 58
15:08 all system wide checks ok *except*
all FEE64 fail ADC calibration (no waveform)
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc3 : 11
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA errors:
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
15:10 Temperatures ok - attachment 17
Detector bias / leakage current ok - attachment 18
Statistics - attachments 18 -24 |
|
Thu May 13 23:58:05 2021, OH, Friday 14th May 00:00 - 08:00 21x
|
00:00 OH Takes over.
PULSER SETTINGS
---------------------------
Pulse is ON
Positive Tail Pulse
Trigger Source is Internal Clock
Trigger Threshold is 3.5
Amplitude : 2.0 Volts
Rep Rate : 2.0 hZ
Delay : 250.0 ns
Fall Time : 1 ms
Attenuation : 1
Display is : Volts
Equivalent keV is : 200.0
Ramp Start at 0.01 Volts
Ramp Stop at 9.99 Volts
Ramp Start at 1.0 keV
Ramp Stop at 999.0 keV
Ramp Time is 60 seconds
# Ramp Cycles is 1
OK
0x210
p+n Thresholds 0xa (100keV)
n+n Threshold 0x20 (320keV)
01:00 System wide checks
Clock status ok
ADC calibration fail as expected (Waveforms off)
WR Errors
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dbf : 7
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Errors
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics ok - attachment 1
Temp ok - attachment 2
Bias and leakage current ok - attachment 3
01:42 Comparing the CS of AIDA3 and 4
Stats have been running for a while (Believe since 5am)
Looking at the counter ratio - attachment 4
We still have a deadtime of around 14% in FEE4 (Assuming FEE3 has no dead time)
01:56 Noticed /media/ThirdDrive was at 97% full
Let DESPEC locals know I would like to use the next run stop to change the directory.
They said they could do it now so they stopped their run.
Changed the symbolic link to /media/1e2....
Now writing to R4_
03:26 MBS DAQ Stopped as beam stopped
03:33 Beam back now and DAQ running again
04:44 System wide checks as before
Statistics ok - attachment 5
Temperatures ok - attachment - 7
Bias and leakage currents ok - attachment 6
05:53 MBS stops run as beam drops
06:00 MBS started a new run as beam is back
07:37 System wide checks clock ok and FPGA as before
WR additional difference on FEE5
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc0 : 8
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics ok - attachment 8
Temp ok - attachment 9
Bias and leakage ok - attachment 10
|
|
Fri Mar 13 08:01:53 2020, DK, Friday 13th 8:00 to 16:00 shift 8x
|
8:00 - See previous shift's elog entry. All good.
9:00
All system wide checks passed
Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
Tape Server okay - 27 MB/s
Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
Stats as usual - attachment #2
Temps normal - attachment #3
Biases and leaks fine - attachment #4
Disk usage projection
Space remaining: ~2 TB
Fill rate: ~2 GB / 1 minute
-> 1000 minutes =~ 17 hours
Still gzipping from R6_823
10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
Kbytes Written: -1986382400 Kbytes/sec: 26816
Rate is fixed at that value on update.
However, actual data files are being updated so data has been coming.
DK and NH consulted and believe the situation is fine.
We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.
10:25 Now we are on R10 after start and stop the TapeServer. Problem is resolved.
11:30
Implants: attachment #5
Stats: attachment #6
Biases / leaks: attachment #7
Temps: attachment #8
System wide checks performed, situation normal. |
|
Thu Mar 12 23:04:24 2020, LS, DK, Friday 13th 00:00 to 08:00 shift 20x
|
Shift file starting R9_639
system wide checks okay
FEE temperatures okay (attachment 1)
leakage currents normal and recorded to spreadsheet (attachment 2)
good event stats are normal (attachment 3)
merger running at 3.5 M items/sec
tape running at 26 MB/sec
rates histogram (attachment 4)
low energy histograms (attachment 5 and 6)
high energy histograms (attachment 7 and 8)
02.02
system wide checks okay
FEE temperatures okay (attachment 9)
leakage currents normal and recorded to spreadsheet (attachment 10)
good event stats are normal (attachment 11)
merger running at 3.5 M items/sec
tape running at 27 MB/sec
04.09
system wide checks okay
FEE temperatures okay (attachment 12)
leakage currents normal and recorded to spreadsheet (attachment 13)
good event stats are normal (attachment 14)
merger running at 3 M items/sec
tape running at 27 MB/sec
writing to file R9_840
06.07
system wide checks okay
FEE temperatures okay (attachment 15)
leakage currents normal and recorded to spreadsheet (attachment 16)
good event stats are normal (attachment 17)
merger running at 3 M items/sec
tape running at 27 MB/sec
07.59
system wide checks okay
FEE temperatures okay (attachment 18)
leakage currents normal and recorded to spreadsheet (attachment 19)
good event stats are normal (attachment 20)
merger running at 3 M items/sec
tape running at 26 MB/sec
as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space |
|
Fri May 13 07:36:18 2022, OH, TD, Friday 13 May 35x
|
08:36 DAQ Still running smoothly
Currently on run R1_328
At current data rates we have just over 4 days of space left on the current disk
Analysis of R1_327 - attachment 1
14.15 all histograms zero'd
baseline system wide checks counters
DAQ continues file S450/R4_33
netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
netint flushenable 60 - not changed
ASIC settings
slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc
BNC PB-5 settings
amplitude 1.000V
attenuation x1
frequency 2Hz
tau_d 1ms (tail pulse)
+ polarity
Attachments 2 & 3 - ucesb
Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.
Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK
Attachments 6-8 - NewMerger stats blocks 0, 1, & 6
Attachment 9 - NewMerger link stats
Attachments 10-11 - TapeServer/NewMerger
Attachment 12 - iptraf aida-gsi network interfaces
Attachments 13-14 - per FEE64 1.8.L spectra
pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM
Attachments 15-18 - per FEE64 rates and stat spectra
Attachments 19-22 - system wide checks
aida09 clock status 6, WR decoder status errors
Attachments 23-26 - adc, pause, resume correlation sclaer data item stats
Attachment 27 - FEE64 temps OK
Attachment 28 - DSSSSD bias and leakage currents OK
14.30 check ASIC load
14.47 attachment 29 - analysis S450/R4_38
15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate) |
|
Fri Jun 13 12:42:39 2025, CC, TD, Friday 13 June 2025 10x
|
13.42 ASIC settings 2025Jun12-13.53.04
*all* FEE64s slow comparator 0x64
all fast disc outputs disabled
p+n FEE64s waveform threshold 7000, n+n FEE64s 9000
CC grounding ribbon cable heavy duty braid to adaptor PCB and AIDA support assembly ground
'top' p+n FEE64s and snout to be completed
16.00 DSSSD bias & leakage current - attachment 1
FEE64 temps OK - attachment 2
All system wide checks OK *except* WR decoder status aida02 and aida06 - attachment 3
WR timestamps OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
per p+n FEE64 1.8.L spectra - attachment 7
aida09 pulser peak width 117 ch FWHM - no change since yesterday
per FEE64 1.8.W spectra - 20us FSR - attachments 8-10
|
|
Fri Mar 12 10:09:19 2021, LS, CA, Friday 12th March 11.00- 17x
|
11.00(Germany) System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment1)
Spectra rates (attachment2)
FEE temps (attachment3)
Leakage currents, written to google sheet (attachment4)
Merger~ 4.9M items/s
Tapeserver ~17MB/s
In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
AIDA has been taken out of the timesorter due to the high data rate, buffers were full
AIDA cannot be seen in ucesb or Go4.
13.00 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment5)
Spectra rates (attachment6)
FEE temps (attachment7)
Leakage currents, written to google sheet (attachment8)
Merger~ 5.1M items/s
Tapeserver ~18MB/s
No timestamp related errors this shift
13.37 AIDA MBS control restarted R33_388
System wide checks same as previous time
13.54 beam stopped for access, file R33_396
13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
Analysed R33_395, 396, 397, 398 no timewarps
14.29 beam back R33_415
14.54 see more bad timestamps in new merger terminal (attachment10)
14.55 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment11)
Spectra rates (attachment12)
FEE temps (attachment13)
Leakage currents, written to google sheet (attachment14)
Merger~ 5.0M items/s
Tapeserver ~17MB/s
16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463
16.15 CA takes over until 18:00
17:11 System wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Base Current Difference
aida07 fault 0xcdd6 : 0xcdd7 : 1
White Rabbit error counter test result: Passed 11, Failed 1
17:13 FEE64 temps ok - attachment 15
Statistics ok - attachment 16
bias and leakage currents ok - attachment 17 |
|
Fri Mar 12 08:32:46 2021, TD, Friday 12 March 08.00-
|
08.31 DAQ continues OK - file R33_263
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
08.32 No merger server error/warning messages since most recent restart c. 00.00 today
08.44 System wide checks
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x7a56 : 0x7a58 : 2
aida06 fault 0x65bb : 0x65bd : 2
aida07 fault 0xcdd1 : 0xcdd4 : 3
aida08 fault 0x2ab3 : 0x2ab5 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 14 6 3 2 1 2 2 3 3 4 11 : 58904
aida02 : 6 4 1 1 1 2 2 4 2 3 7 : 39848
aida03 : 5 6 4 2 1 3 1 3 2 3 7 : 39300
aida04 : 4 1 2 2 3 3 2 4 1 4 15 : 73912
aida05 : 1 5 2 2 2 4 2 3 2 3 7 : 39692
aida06 : 7 2 2 3 1 3 3 3 2 4 7 : 41836
aida07 : 6 3 1 2 3 2 2 3 3 3 7 : 40512
aida08 : 8 5 2 2 1 3 1 3 2 3 7 : 39272
aida09 : 1 2 3 1 1 3 2 3 1 4 7 : 40484
aida10 : 2 6 3 2 1 2 2 2 3 3 7 : 39912
aida11 : 2 3 1 0 2 4 1 3 2 3 7 : 39344
aida12 : 2 3 4 2 0 3 3 3 2 3 7 : 39712
FEE64 Temperatures OK - attachment 1
Good event statistics OK - attachment 2
Detector bias & leakage currents OK - attachment 3
Merger OK - 5.1M data items/s
TapeServer OK - 16Mb/s
08.54 p+n junction strip HEC spectra - attachment 1
Rate spectra - attachment 2
09.15 Burst of Merger server warning messages of type
MERGE Data Link (3547): bad timestamp 1 3 0x8128d6c8 0x05056328 0x00008d6c85056328 0x166b8d6c85056328 0x166b8d6d86da01ee
also multiple
Warning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one
MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in
relayWarning: At least one MIDAS block missed in relayWarning: A
The latter may reflect downstream problems writing MBS data |
|
Fri Mar 12 20:12:10 2021, TD, Friday 12 March
|
21.12 DAQ continues OK - file R33_617
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
21.15 System wide checks
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x7a58 : 0x7a5a : 2
aida06 fault 0x65bd : 0x65bf : 2
aida07 fault 0xcdd6 : 0xcdda : 4
aida08 fault 0x2ab5 : 0x2ab7 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 5 4 2 0 1 2 3 3 2 3 11 : 55956
aida02 : 21 4 1 2 2 4 2 2 1 4 7 : 40260
aida03 : 10 4 4 1 6 4 1 3 3 3 6 : 36648
aida04 : 7 2 4 0 2 3 2 4 1 4 15 : 73836
aida05 : 1 3 1 5 4 3 1 2 3 4 6 : 37964
aida06 : 12 3 5 2 1 3 3 3 2 4 7 : 41880
aida07 : 2 4 3 3 4 2 2 4 3 3 6 : 37048
aida08 : 17 13 2 3 2 3 2 3 2 3 7 : 39724
aida09 : 3 7 2 0 1 2 2 3 2 2 7 : 37284
aida10 : 2 5 3 2 2 3 1 3 2 3 7 : 39328
aida11 : 23 18 2 3 2 3 3 2 3 3 6 : 36460
aida12 : 18 7 3 1 1 4 2 2 2 3 7 : 39184
FEE64 Temperatures OK - attachment 1
Good event statistics OK - attachment 2
Detector bias & leakage currents OK - attachment 3
Merger OK - 4.9M data items/s
TapeServer OK - 16Mb/s
All histograms zero'd
21.50 At some point between R33_610 (20.59) and R33_625 (21.30) aida05 stopped producing data (zero good events - see attachment 2)
able to telnet to aida05 - no warnings/error messages in /var/log/messages
DAQ STOP (all except aida05 stopped OK, aida05 remained GOING)
issued aida05 reboot command via telnet command line
DAQ RESET/SETUP/GO (all FEE64s GOING OK except aida05 - zero good events)
21.15 All system wide checks OK *except*
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
21.50 aida04 & aida05 statistics for comparison - see attachments 4 & 5 |
|
Fri Feb 11 12:09:24 2022, NH, Fri 11 Feb AIDA Work
|
Use beam break between FRS experiments to finalise getting AIDA testable for PJCS etc.
USB hubs reconnected - all 16 ttyUSBs seem to be visible now
New inverter installed (Ortec 533 Sum Invert, brand new) - Pulser T connected to input B1 and inverted out B sent to -ve FEEs
VME crate for VETAR2 is powered on so White Rabbit should be available
All grounds reconnected on FEEs
All FEEs connected to pulser chain again
-
Moved interlock to new Al shelf on AIDA rack so it's not kicked
Powered on OK, all 4 lights on in box but AIDA Relay had to water light
Investigation shows the cable between relay and interlock was broken, one wire had come loose from connector
Resoldered and now it works as usual
System powered up OK
All 16 FEEs respond to serial
All 16 FEEs have WR
All 16 FEEs show pulser ADC entries
Seems good! |
|
Fri Jun 10 10:32:46 2022, NH, Fri 10 Jun 8x
|
Probe PSU output with 'scope again and 10X probe
Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)
Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled
Lots of fluctuation and jitter
Figs 4-6: DC 1MOhm output from all 3 power rails
Fig 7-8: Small "loop" using probe and observed scope spectra.
After lunch ideas:
- Try again with 1X probe attenuation, maybe record output to USB
- Remove unconnected MACBs from master
- |
|