 Mon Mar 8 17:02:38 2021, CA, Monday 8th March 18:00 - 00:00 12x
|
18:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
18:24 Rate spectra - attachment 4
18:30 Merger ok - 4.4M data items/s
TapeServer ok - 43MB/s
Data forwarding to MBS ok
19:28 beam back after a brief outage - AIDA on file R9_736
FRS having DAQ issues
19:57 FRS DAQ/Scalars back
20:12 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 5
FEE Temperatures - attachment 6
Bias and leakage currents ok - attachment 7
Merger ok - 4.5M data items/s
TapeServer ok - 46MB/s
Data forwarding to MBS ok
20:33 No beam - AIDA on file R9_821
21:08 rates spectra - attachment 8
22:35 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 9
FEE Temperatures - attachment 10
Bias and leakage currents ok - attachment 11
Merger ok - 4.4M data items/s
TapeServer ok - 43MB/s
Data forwarding to MBS ok
temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload.
23:07 media/ThirdDrive at 65% capacity
writing to file R9_1021
file compression currently up to R1_686
23:36 beam stable ~1e9 pps
rate spectra - attachment 12
|
|
Mon Mar 8 07:02:17 2021, CA, LS, Monday 8th March 08:00 - 17:00 24x
|
08:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude 1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
08:20 Merger ~4.5M data items/s
TapeServer ~ 45MB/s
Writing to file R6_1148
09:00 Beam off for optimisation - writing to file R6_1212
Expected back at 09:30
09:30 Beam will be off for a bit longer - an hour or two
09:34 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 4
FEE Temperatures - attachment 5
Bias and leakage currents ok - attachment 6
Merger ~4.5M data items/s
TapeServer ~ 45MB/s
10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds
to finish
Something confusing, will look at data and report to VP
Reverted to original merger and back... 45 MB/s
R9
12.00 (LS)
System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
Statistics (attachment7)
FEE temps (attatchment8)
Rate spectra (attachment9), beam still down
Leakage Currents written to sheets (attachment10)
Merger ~45M items/s
Tapeserver ~43MB/s
14.00 Still no beam but is expected to return soon
System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
Statistics (attachment11)
FEE temps (attatchment12)
Leakage Currents written to sheets (attachment13)
Merger ~45M items/s
Tapeserver ~44MB/s
14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)
15.00 Beam is back (attachment15)
Still writing to R9 (roughly around R9_386)
15.28 No beam (attachment16)
15.41 Beam back (attachment17)
Roughly R9 (roughly around R9_439)
16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
Statistics (attachment19)
Spectra Rate (attachment20)
FEE temps (attatchment21)
Leakage Currents written to sheets (attachment22),
still rising but looking on Grafana 7 day period should start to lower around 17.00
Merger ~45M items/s
Tapeserver ~43MB/s
16.55 Beam down (attachment23), back a couple minutes later
15.51 beam down (attachment24)
|
|
Sun Mar 7 23:13:50 2021, OH, Monday 8th March 13x
|
00:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:14 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4
02:15 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 5
FEE Temperatures - attachment 6
Bias and leakage currents ok - attachment 7
03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped
04:00 Noticed more bad merge events
System wide checks has also noted an FPGA error.
All other system wide checks same results as previous
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A3F29 BA0DF4D2 , WR/10=> 23DD31DC5CE3215, Readout Time => 23DD31DC6F34000
aida02 : White Rabbit=> 166A3F29 CC49ECD7 , WR/10=> 23DD31DC7A0FE15, Readout Time => 23DD31DC8428000
aida03 : White Rabbit=> 166A3F29 DF55B7E8 , WR/10=> 23DD31DC9889264, Readout Time => 23DD31DCA85C000
aida04 : White Rabbit=> 166A3F29 EECF82B7 , WR/10=> 23DD31DCB14C045, Readout Time => 23DD31DCC2B8000
aida05 : White Rabbit=> 166A3F29 FED2EFD2 , WR/10=> 23DD31DCCAEB195, Readout Time => 23DD31DCDC14000
aida06 : White Rabbit=> 166A3F2A 0EDC023D , WR/10=> 23DD31DCE49336C, Readout Time => 23DD31DCF204000
aida07 : White Rabbit=> 166A3F2A 1E8170D3 , WR/10=> 23DD31DCFD9BE7B, Readout Time => 23DD31DD10A0000
aida08 : White Rabbit=> 166A3F2A 30448F76 , WR/10=> 23DD31DD1A074BF, Readout Time => 23DD31DD2A34000
aida09 : White Rabbit=> 166A3F2A 3FE36E61 , WR/10=> 23DD31DD33057D6, Readout Time => 23DD31DD3B94000
aida10 : White Rabbit=> 166A3F2A 4E83A1F7 , WR/10=> 23DD31DD4A6C365, Readout Time => 23DD31DD46A8000
aida11 : White Rabbit=> 166A3F2A 5FD19086 , WR/10=> 23DD31DD661C1A7, Readout Time => 23DD31DD7664000
aida12 : White Rabbit=> 166A3F2A 6FE06F3D , WR/10=> 23DD31DD7FCD7EC, Readout Time => 23DD31DD9684000
04:07 Beam is back AIDA on file R6_815
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
ASIC check ok
Multiple merge data error events observed
06:04 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 11
FEE Temperatures - attachment 12
Bias and leakage currents ok - attachment 13
07:22 Beam noticed dropped but back quickly |
|
Mon Apr 8 16:39:00 2024, JB, CC, TD, Monday 8 April 11x
|
17.32 Power and detector bias cycle
DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
*except* aida02 ASIC temp u/s
ASIC settings 2024Mar27-11.25.32
slow comparator p+n FEE64s 0xa (100keV), n+n FEE64s 0xf (150keV)
BNC PB-5 pulser
amplitude 10.0V
attenuation x10
frequency 25Hz
polarity -
tau_d 1ms
tail pulse
test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser
test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser
All system wide checks OK *except* aida02 & aida03 WR decoder status errors - attachment 10
WR timestamps OK - attachment 11
ADC data item stats - attachment 3
12 of 16 < 20k, all < 100k
per FEE64 Rate spectra - attachment 4
per n+n FEE64 1.8.L spectra - attachment 5
1.8.W spectra - 20us FSR - attachments 6-9
preamplifier output noise generally very good
*Current ground configuration*
CAEN N1419ET LK fitted (non floating outputs)
ground cable from aida04-aida12-aida02-aida09-aida01-aida05
HV#0 aida12-aida03-aida15
ground cable from aida08-aida16-aida06-aida10-aida14-aida13
HV#1 aida16-aida07-aida11
LK3 fitted aida03, aida07
LK1 fitted aida02, aida04, aid06, aida08
test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser
test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser
all DSSSD ribbon cable drain wires grounded to their resepctive AIDA adaptor PCBs
JB observed open circuit between AIDA Al snout and AIDA support frame ( as expected )
AIDA PSU cabling as reported https://elog.ph.ed.ac.uk/DESPEC/560
Snout configuration:
- all bPlast cables are disconnected and floating.
- CC taped up all the cables and exit points in snout.
- CC, JB covered the exit of the snout with two additional layers of aluminium foil which were also taped shut.
To Do:
- repower detector, see if similar conditions persist.
- disconnect test - pulser, again see if conditions persist.
- aida04 asic#1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?
|
|
Mon Jun 7 10:33:48 2021, TD, Monday 7 June 9x
|
11.30 DAQ continues OK - file S496/R32_144
alpha background
All system wide checks OK *except*
Base Current Difference
aida01 fault 0x8944 : 0x8946 : 2
aida02 fault 0x9b36 : 0x9b38 : 2
aida03 fault 0xf26b : 0xf26d : 2
aida04 fault 0x57a0 : 0x57a2 : 2
aida05 fault 0x4d3c : 0x4d3d : 1
aida05 : WR status 0x10
aida06 fault 0x640c : 0x640d : 1
aida07 fault 0x255e : 0x255f : 1
aida08 fault 0xffbd : 0xffbe : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 9 5 11 6 3 4 2 2 1 4 7 : 40572
aida02 : 1 3 6 7 2 3 3 2 1 4 7 : 40540
aida03 : 21 7 7 8 2 4 1 3 2 3 7 : 39804
aida04 : 13 7 5 6 1 3 2 3 2 3 7 : 39740
aida05 : 2 8 6 5 1 2 2 3 1 4 7 : 40584
aida06 : 10 4 5 5 4 3 1 3 1 4 7 : 40632
aida07 : 9 3 4 7 3 3 3 2 1 4 7 : 40604
aida08 : 11 2 6 4 4 3 3 2 1 4 7 : 40604
aida09 : 10 2 5 4 2 2 2 3 1 4 7 : 40584
aida10 : 14 5 8 6 2 1 1 3 1 4 7 : 40352
aida11 : 8 6 9 5 4 4 3 3 2 3 7 : 40320
aida12 : 10 5 3 7 3 3 2 2 1 4 7 : 40352
aida13 : 7 5 4 6 5 4 3 3 2 3 7 : 40324
aida14 : 7 5 4 8 4 4 3 3 2 3 7 : 40324
aida15 : 8 4 7 4 2 2 2 4 1 4 7 : 41136
aida16 : 13 5 7 6 3 4 3 3 2 3 7 : 40268
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
11.35 analysis of file S496/R32_143 - attachment 9
max deadtime 0.29% aida03 |
|
Mon May 6 07:10:26 2024, TD, Monday 6 May
|
08.11 DSSSD#1 bias -120V leakage current -17.0uA
DSSSD#2 bias -120V leakage current -10.5uA
S4 temperature 25 deg C
cf. 04.00 30.4.24 S4 temperature 25 deg C
DSSSD#1 -21.9uA
DSSSD#2 -13.0uA
Leakage currents currently c. 80% of 30.4.24 values at same temperature |
|
Mon Jul 5 14:36:15 2021, NH, Monday 5 July
|
AIDA Unbiased and powered down
All FEEs were running before stoppage. None had rebooted.
Aida05 did not respond until merger was restarted, all other DAQs responded fine.
Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup
AIDA will remain in Snout in lab for time being.
FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now |
|
Mon Mar 4 11:35:30 2024, TD, Monday 4 March 11x
|
12.30 FEE64 41:d7:cd ASIC mezzanine u/s, no data, ASIC temperature c. 20 deg C ( ambient ) low
Replaced ASIC mezzanine with new ASIC mezzanine
Installed as aida08
Water temperature & pressure as measured outside S4 area - OK
Manually power FEE64s
BNC PB-5 local control/ON
Amplitude 1.0V
Attenuation x10
Polarity +
tau_d 1ms
Frequency 22Hz
FEE64 temps - attachment 1
aida02 with new ASIC mezzanine continues to ramp to c. 512 deg C - initially reads 0, next refresh c. 70 and third refresh c. 512 deg C
aida02 Virtex and PSU temps OK, aida02 cooling plate ambient to touch
aida08 ASIC temp c. 45 deg C as expected, Virtex and PSU temps OK
All system wide checks OK
WR timestamp OK - attachment 2
aida04 ASIC settings - attachment 3
ADC data item stats OK - attachment 4
aida04 Rate spectrum - attachment 5
aida04 *.*.L spectra - attachments 6-9
1.8.L pulser peak width 13 ch FWHM
aida04 1.8.W spectra - 20us FSR - attachment 10-11 |
|
Mon Apr 4 09:19:49 2022, TD, Monday 4 April 12x
|
AIDA adaptor PCB config - all LKs removed *except*
aida04 & aida08 LK1
aida02 & aida07 LK2, LK3 & LK4
aida09, aida01, aida10, aida11, aida15, aida12 & aida16 LK2 & LK4
10.05 check all ASICs all FEE64s
zero histograms
Attachment 1 per FEE64 rate spectra
Attachment 2 per FEE64 p+n 1.8.L spectra - aida10 peak width 79 ch FWHM
Attachment 3 per FEE64 p+n 1.8.W spectra 20us FSR
Attachment 4 per FEE64 n+n 1.8.W spectra 20us FSR
Attachments 5-8 - system wide checks - all OK *except*
FPGA timestamp errors no longer works
aida05 & aida09 report WR decoder error 0x10
Attachment 9 ADC data item stats
Attachment 10 FEE64 temps OK
Attachment 11 DSSSD bias & leakage current OK
Attachment 12 per FEE64 n+n 1.8.L spectra - aida08 peak width 369 ch FWHM
14.25 DAQ STOP
DSSSD bias OFF
AIDA triple snout removed from AIDA stand and moved to table by aida-gsi (Messeheute) |
|
Mon Jun 3 07:35:56 2024, TD, Monday 3 June 8x
|
08.28 DSSSD bias & leakage current - attachments 1-2
FEE64 temps OK - attachment 3
ADC data item stats - attachment 4
per FEE64 Stat spectra - attachment 5
per FEE64 1.8.L spectra - attachment 6
per FEE64 1.8.W spectra - 0us FSR - attachments 7-8
08.39 DAQ stops file S181/R1_36
FEE64 power OFF
DSSSD bias ON
11.12 DSSSD bias OFF |
|
Mon Apr 29 08:11:16 2024, TD, Monday 29 April 41x
|
08.52 Beam off
09.11 New data file R22
Monitor accumulated implanted activity in bPlas and DSSSDs
ASIC settings 2024Apr26-01.29.53
DSSSD bias & leakage current OK - attachments 1-3
FEE64 temperatures OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
Merger etc - attachment 7
analysis data file R22_19 - attachment 8
all deadtimes 0.3%, or less
no timewarps
HEC data item rate 390Hz
10.00 all histograms & stats zero'd
13.24 DSSSD bias & leakage current OK - attachments 9-10
FEE64 temperatures OK - attachment 11
ADC data item stats - attachment 12
per FEE64 Rate spectra - attachment 13
per p+n FEE64 1.8.L spectra - attachments 14-16
common x/y axes - peak height proxy for width
aida09 pulser peak width 58 ch FWHM (~41keV FWHM) cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
electronic noise today very similar to electronic noise Monday 22 April
per FEE64 1.8.W spectra - 20us FSR - attachments 17-19
Merger etc - attachment 20
analysis data file R22_60 - attachment 21
max deadtime 1.7% (aida11), all others <1%
no timewarps
HEC data item rate 710Hz
14.13 ucesb - attachment 22
14.15 aida04 not producing data
14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
15.27 *all* FEE64s stop producing data - reason unknown
16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
16.29 DSSSD bias & leakage current OK - attachments 23-24
FEE64 temperatures OK - attachment 25
ADC data item stats - attachment 26
per FEE64 Rate spectra - attachment 27
Merger etc - attachment 28
20.45 DSSSD bias & leakage current OK - attachments 29-30
FEE64 temperatures OK - attachment 32
ADC data item stats - attachment 31
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
20.54 analysis data file R22_145 - attachment 35
max deadtime 1.5% (aida11), all others <1%
no timewarps
HEC data item rate 775Hz
03.45 DSSSD bias & leakage current OK - attachments 36-37
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41 |
|
Mon Jun 27 17:03:09 2022, MA, Monday 27th June 16:00-00:00 6x
|
16:00 Took over the shift from OH no beam yet.
18:00 Still no beam yet.
Statistics, Temperature, Current are checked and attached 1-3
system wide checks same as last updated in the previoues shift.
22:00 The beam is back but not taking data yet! FRS team doing some checkings
Statistics, Temperature, Current are checked and attached 4-6
system wide checks same as last updated in the previoues shift.
23:30 beam is back and taking data |
|
Mon Jun 27 06:45:22 2022, OH, Monday 27th June 08:00-16:00
|
07:45 Spoke to David and the beam has been gone since about 05:30
Reason for the loss of beam is a vacuum issue before the FRS
They are waiting for the experts
08:31 Statistics ok - attachment 1
Temperature ok - attachment 2
Bias and leakage currrents ok - attachment 3
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
Currently on file R5_771
Analysis of file R5_770 (No beam) - attachment 4
Around 0.25% deadtime on AIDA02 rest even less
09:02 Current free HDD space 965 GB
Current tape server rate 4979 kB/s
Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours
11:41 Statistics ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
FPGA Timestamp error counter test result: Passed 7, Failed 1
Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently? |
|
Mon May 27 17:20:49 2024, TD, Monday 27 May 9x
|
17.50 bPlas cabling, grounds etc connected, power OFF
CC reports that bPlas power is actually ON
FEE64 power ON
DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ASIC settings 2024Apr26-01.29.53
p+n FFEE64 slow comparator 0xa, n+n Ohmic FEE64 0xf
System wide checks OK *except* aida02 WR decoder status
ADC data item stats - attachment 3
9/16 < 20k, max 234k (aida08)
per FEE64 Rate spectra - attachments 4-6
per FEE64 1.8.W spectra - 20us FSR - attachments 7-8
WR timestamps OK - attachment 9
18.50 FEE64 power OFF |
|
Mon Apr 26 09:41:21 2021, TD, Monday 26 April
|
10.42 All system wide checks OK *except*
Base Current Difference
aida01 fault 0xc960 : 0xc962 : 2
aida02 fault 0x8bbf : 0x8bc1 : 2
aida03 fault 0xae9e : 0xaea0 : 2
aida04 fault 0x6e3 : 0x6e5 : 2
aida05 fault 0x31a4 : 0x31ab : 7
aida05 : WR status 0x10
aida06 fault 0x22ea : 0x22f1 : 7
aida07 fault 0x438f : 0x4396 : 7
aida08 fault 0x61ed : 0x61f4 : 7
aida09 fault 0x1c32 : 0x1c33 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 25 8 28 8 3 2 2 3 2 3 4 : 27940
aida02 : 24 9 3 5 1 3 1 3 3 3 6 : 36408
aida03 : 24 6 5 4 3 3 1 3 3 3 6 : 36512
aida04 : 23 6 6 1 3 3 1 3 3 3 6 : 36428
aida05 : 15 21 27 7 4 2 2 3 3 3 4 : 29044
aida06 : 24 8 2 4 1 3 1 3 3 3 6 : 36352
aida07 : 29 24 28 9 3 3 2 2 2 3 4 : 27732
aida08 : 17 10 1 1 1 4 1 3 3 3 6 : 36356
aida09 : 41 49 18 14 3 4 3 4 2 3 4 : 29388
aida10 : 3 1 6 4 2 3 3 3 2 3 6 : 35828
aida11 : 22 7 5 4 2 1 2 4 2 3 6 : 35936
aida12 : 65 30 23 9 3 1 2 3 2 3 4 : 28100
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sat Apr 24 03:57:22 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
10.47 DSSSD bias |& leakage currents Ok - attachment 1
FEE64 temps OK - attachment 2
good event stats - attachment 3
merger time seq errors - attachment 4
Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5
DAQ continues OK - no data storage |
|
Mon May 24 09:57:07 2021, TD, Monday 24 May 19x
|
07.30 Hot HEC channels multiple FEE64s
ASIC check fixes
10.56 DAQ continues OK file S496/R18_75
System wide checks OK * except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b1 : 1
aida02 fault 0xfe72 : 0xfe73 : 1
aida03 fault 0x97f4 : 0x97f5 : 1
aida04 fault 0x950 : 0x951 : 1
aida05 fault 0x8e93 : 0x8e94 : 1
aida06 fault 0x545a : 0x545b : 1
aida07 fault 0xc04d : 0xc04e : 1
aida08 fault 0xa0f7 : 0xa0f8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 2 7 2 1 4 3 2 1 4 7 : 40452
aida02 : 13 3 3 1 3 4 2 2 2 4 7 : 41308
aida03 : 17 6 2 2 1 4 2 3 2 3 7 : 39700
aida04 : 8 6 1 0 1 3 1 4 2 3 7 : 39712
aida05 : 10 3 5 3 4 3 3 2 1 4 7 : 40560
aida06 : 4 4 2 0 2 3 2 3 3 4 7 : 42576
aida07 : 5 3 2 3 1 3 1 3 2 4 7 : 41324
aida08 : 11 5 1 1 2 3 3 2 2 4 7 : 41348
aida09 : 7 4 3 0 2 3 1 3 2 4 7 : 41324
aida10 : 11 4 2 2 2 3 2 4 1 4 7 : 41132
aida11 : 18 5 0 2 2 4 2 2 1 4 7 : 40240
aida12 : 2 3 0 1 2 3 2 2 2 4 7 : 41024
aida13 : 4 5 1 2 2 3 2 4 1 4 7 : 41096
aida14 : 9 3 1 2 1 4 2 2 2 4 7 : 41164
aida15 : 2 3 1 3 2 2 3 3 2 4 7 : 41744
aida16 : 5 5 2 0 3 3 2 4 1 4 7 : 41116
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
1.8.H spectra - attachments 6-8
1.8.L spectra - attachments 9 & 10
Stat spectra - attachments 11 & 12
1.8.W spectra - 20us FSR - attachments 13 & 14
ADC data items - attachment 15
FEE64 temperatures OK - attachment 16
DSSSD bias & leakage currents OK - attachment 17
Merger/Tape Server/Merger statistics - attachment 18
no merger errors reported since previous restart
analysis of file R18_74
max dead time 0.05%
10.17 ASIC check
All histograms, statistics, merger statistics zero'd |
|
Mon May 23 16:41:45 2022, TD, Monday 23 May
|
17.40 DAQ stopped
Attachment 1 - DSSSD bias & leakage current
Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased
Attachment 3 - FEE64 temps - OK |
|
Mon Dec 23 10:54:46 2024, TD, Monday 23 December 2024
|
11.54 re-established remote access via AnyDesk - details to follow
confirmed AIDA FEE64 power OFF, detector HV OFF
1. Establish port (to access carme-gsi) ssh -L 8080:proxy.gsi.de:8080 carme@atppc025
Establish port (to access aida-gsi) ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin
Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes
2. Check AnyDesk password for remote access has been set up
3. Close anydesk and check it is *really* closed using
ps -o pid= -C anydesk
kill -9 any process (as root)
4 Restart AnyDesk
systemctl restart anydesk (as npg) |
|
Mon Apr 12 16:49:40 2021, TD, Monday 22 April - pulser walkthrough & alpha background 6x
|
16.54 Pulser walkthrough
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xa
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
file NULL/R4
BNC PB-5 settings changed by terminal command line using the commands
set rep rate 25
set attenuation 5
set amplitude 10
set amplitude 9
:
:
set amplitude 1
set attenuation 1
set rep rate 2
aida07 & aida08 1.8.L spectra - see attachment 1
17.26 run ends OK file NULL/R4_27
17.26 file NULL/R5
amplitudes 0.6, 1, 1.4, 1.8, 2.2, 2.6, 3
aida07 & aida08 1.8.L spectra - see attachment 2
17.47 run ends OK NULL/R5_23
17.53 system wide checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 11 3 2 0 2 4 2 3 4 11 : 58888
aida02 : 7 11 4 2 0 4 3 2 3 3 7 : 40436
aida03 : 38 7 7 2 6 4 1 3 3 3 6 : 36864
aida04 : 19 10 5 2 1 3 1 4 2 4 15 : 74732
aida05 : 30 5 4 3 2 3 4 3 3 4 15 : 76096
aida06 : 24 4 3 3 1 4 2 3 2 4 7 : 41808
aida07 : 22 7 11 7 1 6 3 4 3 3 6 : 37984
aida08 : 13 7 5 2 1 4 3 3 1 4 6 : 36924
aida09 : 33 9 10 6 4 4 2 2 2 4 6 : 37676
aida10 : 15 7 7 1 1 3 1 3 2 3 7 : 39364
aida11 : 30 13 8 3 4 4 3 3 3 3 6 : 37312
aida12 : 17 9 4 3 2 3 2 2 3 3 7 : 40236
17.57 Slow comparator 0xa -> 0x64
BNC PB-5 Pulser OFF
file NULL/R9
alpha background
FEE64 temperatures OK - see attachment 3
statistics OK - see attachment 4
DSSSD bias & leakage currents OK - see attachment 5
tape server & merger - attachment 6
11:31 13.04.21
file NULL/R9 stopped for work in day
will resume alpha over night again
|
|
Sun Apr 21 23:20:45 2024, TD, Monday 22 April 38x
|
00.15 Data link to MBS was closed by MBS - reason(s) unknown
NH restarted MBS receiver and data transfer restarted
01.34 DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ADC data item stats OK - attachment 3
Merger, TapeServer etc - attachment 4
04.36 DSSSD bias & leakage current OK - attachment 5
FEE64 temperatures OK - attachment 6
ADC data item stats OK - attachment 7
Merger, TapeServer etc - attachment 8
07.54 data storage to local disk enabled
file aida-gsi:/TapeData/S100_alpha/R14_
09.36 DAQ continues, current data file R14_46
168Er setting
DSSSD bias & leakage current OK - attachments 9 & 10
leakage current DSSSD#1 ramping and cycling with beam spills
FEE64 temperatures OK - attachment 11
System wide checks OK *except* WR/FPGA errors - attachments 12-13
WR timestamps - attachment 14
aida04 out of seq
ADC data item stats - attachment 15
high rates observed all FEE64s beam on, beam off rates OK
per FEE64 Rate spectra - attachment 16
per FEE64 1.8.L spectra - attachments 17-18
per FEE64 1.8.W spectra - attachments 19-20
per FEE^4 1.8.H spectra - attachments 21-22
Merger, TapeServer etc - attachments 23-25
data rates to c. 10Mb/s beam on
09.55 aida04 rebooted - https://elog.ph.ed.ac.uk/DESPEC/586
10.01 DAQ restarted
data file R15
ADC data item stats - attachments 26-27
large variation in rates beam off/on
c. 10.30 beam off (for new FRS setting?)
11.25 analysis of data files R14_19 (beam ON) and R15_24 (beam OFF) - attachments 28-29
For beam ON max deadtime c. 11% (aida04), beam OFF max deadtime c. 0.1% (aida04)
Zero timewarps observed
Notice large # of PAUSE/RESUME data items for aida03 & aida04 presumably due to additional overhead of processing the time machine scalers
12.35 Beam off
DSSSD bias & leakage current OK - attachment 30
Can observe beam spill structure in DSSSD leakage currents - now slowly recovering with beam off
FEE64 temperatures OK - attachment 31
ADC data item stats OK - attachment 32
Merger, TapeServer etc - attachment 33
12.55 New ASIC settings 2024Apr22-12.55.38
*all* discs disabled to reduce load on aida04
slow comparator 0x14
BNC PB-5 settings (since yesterday evening)
Amplitude 10.0V
Attenuation x10
tau_d 1ms
Frequency 2Hz
Tail pulse
Polarity +
13.46 all histograms & stats zero'd
13.48 Beam off
per FEE64 Rate spectra - attachment 34
2Hz pulser to p+n FEE64s
per FEE64 1.8.W spectra - attachments 35-36
p+n FEE64 threshold = 6000 to capture pulser events
n+n FEE64 threshold = 8000 to capture noise
per p+n FEE64 1.8.L spectra - attachments 37-38
common x/y-scale - peak height proxy for peak width
aida09 pulser peak width 57 ch FWHM ~40keV FWHM ( 5 x sigma ~ 85keV )
14.28 168Eu setting (check only - setup still in progress) |
|