 Mon May 6 07:10:26 2024, TD, Monday 6 May 
|
08.11 DSSSD#1 bias -120V leakage current -17.0uA
DSSSD#2 bias -120V leakage current -10.5uA
S4 temperature 25 deg C
cf. 04.00 30.4.24 S4 temperature 25 deg C
DSSSD#1 -21.9uA
DSSSD#2 -13.0uA
Leakage currents currently c. 80% of 30.4.24 values at same temperature |
|
Mon Jul 5 14:36:15 2021, NH, Monday 5 July
|
AIDA Unbiased and powered down
All FEEs were running before stoppage. None had rebooted.
Aida05 did not respond until merger was restarted, all other DAQs responded fine.
Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup
AIDA will remain in Snout in lab for time being.
FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now |
|
Mon Mar 4 11:35:30 2024, TD, Monday 4 March 11x
|
12.30 FEE64 41:d7:cd ASIC mezzanine u/s, no data, ASIC temperature c. 20 deg C ( ambient ) low
Replaced ASIC mezzanine with new ASIC mezzanine
Installed as aida08
Water temperature & pressure as measured outside S4 area - OK
Manually power FEE64s
BNC PB-5 local control/ON
Amplitude 1.0V
Attenuation x10
Polarity +
tau_d 1ms
Frequency 22Hz
FEE64 temps - attachment 1
aida02 with new ASIC mezzanine continues to ramp to c. 512 deg C - initially reads 0, next refresh c. 70 and third refresh c. 512 deg C
aida02 Virtex and PSU temps OK, aida02 cooling plate ambient to touch
aida08 ASIC temp c. 45 deg C as expected, Virtex and PSU temps OK
All system wide checks OK
WR timestamp OK - attachment 2
aida04 ASIC settings - attachment 3
ADC data item stats OK - attachment 4
aida04 Rate spectrum - attachment 5
aida04 *.*.L spectra - attachments 6-9
1.8.L pulser peak width 13 ch FWHM
aida04 1.8.W spectra - 20us FSR - attachment 10-11 |
|
Mon Apr 4 09:19:49 2022, TD, Monday 4 April 12x
|
AIDA adaptor PCB config - all LKs removed *except*
aida04 & aida08 LK1
aida02 & aida07 LK2, LK3 & LK4
aida09, aida01, aida10, aida11, aida15, aida12 & aida16 LK2 & LK4
10.05 check all ASICs all FEE64s
zero histograms
Attachment 1 per FEE64 rate spectra
Attachment 2 per FEE64 p+n 1.8.L spectra - aida10 peak width 79 ch FWHM
Attachment 3 per FEE64 p+n 1.8.W spectra 20us FSR
Attachment 4 per FEE64 n+n 1.8.W spectra 20us FSR
Attachments 5-8 - system wide checks - all OK *except*
FPGA timestamp errors no longer works
aida05 & aida09 report WR decoder error 0x10
Attachment 9 ADC data item stats
Attachment 10 FEE64 temps OK
Attachment 11 DSSSD bias & leakage current OK
Attachment 12 per FEE64 n+n 1.8.L spectra - aida08 peak width 369 ch FWHM
14.25 DAQ STOP
DSSSD bias OFF
AIDA triple snout removed from AIDA stand and moved to table by aida-gsi (Messeheute) |
|
Mon Jun 3 07:35:56 2024, TD, Monday 3 June 8x
|
08.28 DSSSD bias & leakage current - attachments 1-2
FEE64 temps OK - attachment 3
ADC data item stats - attachment 4
per FEE64 Stat spectra - attachment 5
per FEE64 1.8.L spectra - attachment 6
per FEE64 1.8.W spectra - 0us FSR - attachments 7-8
08.39 DAQ stops file S181/R1_36
FEE64 power OFF
DSSSD bias ON
11.12 DSSSD bias OFF |
|
Mon Apr 29 08:11:16 2024, TD, Monday 29 April 41x
|
08.52 Beam off
09.11 New data file R22
Monitor accumulated implanted activity in bPlas and DSSSDs
ASIC settings 2024Apr26-01.29.53
DSSSD bias & leakage current OK - attachments 1-3
FEE64 temperatures OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
Merger etc - attachment 7
analysis data file R22_19 - attachment 8
all deadtimes 0.3%, or less
no timewarps
HEC data item rate 390Hz
10.00 all histograms & stats zero'd
13.24 DSSSD bias & leakage current OK - attachments 9-10
FEE64 temperatures OK - attachment 11
ADC data item stats - attachment 12
per FEE64 Rate spectra - attachment 13
per p+n FEE64 1.8.L spectra - attachments 14-16
common x/y axes - peak height proxy for width
aida09 pulser peak width 58 ch FWHM (~41keV FWHM) cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
electronic noise today very similar to electronic noise Monday 22 April
per FEE64 1.8.W spectra - 20us FSR - attachments 17-19
Merger etc - attachment 20
analysis data file R22_60 - attachment 21
max deadtime 1.7% (aida11), all others <1%
no timewarps
HEC data item rate 710Hz
14.13 ucesb - attachment 22
14.15 aida04 not producing data
14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
15.27 *all* FEE64s stop producing data - reason unknown
16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
16.29 DSSSD bias & leakage current OK - attachments 23-24
FEE64 temperatures OK - attachment 25
ADC data item stats - attachment 26
per FEE64 Rate spectra - attachment 27
Merger etc - attachment 28
20.45 DSSSD bias & leakage current OK - attachments 29-30
FEE64 temperatures OK - attachment 32
ADC data item stats - attachment 31
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
20.54 analysis data file R22_145 - attachment 35
max deadtime 1.5% (aida11), all others <1%
no timewarps
HEC data item rate 775Hz
03.45 DSSSD bias & leakage current OK - attachments 36-37
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41 |
|
Mon Jun 27 17:03:09 2022, MA, Monday 27th June 16:00-00:00 6x
|
16:00 Took over the shift from OH no beam yet.
18:00 Still no beam yet.
Statistics, Temperature, Current are checked and attached 1-3
system wide checks same as last updated in the previoues shift.
22:00 The beam is back but not taking data yet! FRS team doing some checkings
Statistics, Temperature, Current are checked and attached 4-6
system wide checks same as last updated in the previoues shift.
23:30 beam is back and taking data |
|
Mon Jun 27 06:45:22 2022, OH, Monday 27th June 08:00-16:00
|
07:45 Spoke to David and the beam has been gone since about 05:30
Reason for the loss of beam is a vacuum issue before the FRS
They are waiting for the experts
08:31 Statistics ok - attachment 1
Temperature ok - attachment 2
Bias and leakage currrents ok - attachment 3
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
Currently on file R5_771
Analysis of file R5_770 (No beam) - attachment 4
Around 0.25% deadtime on AIDA02 rest even less
09:02 Current free HDD space 965 GB
Current tape server rate 4979 kB/s
Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours
11:41 Statistics ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
FPGA Timestamp error counter test result: Passed 7, Failed 1
Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently? |
|
Mon May 27 17:20:49 2024, TD, Monday 27 May 9x
|
17.50 bPlas cabling, grounds etc connected, power OFF
CC reports that bPlas power is actually ON
FEE64 power ON
DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ASIC settings 2024Apr26-01.29.53
p+n FFEE64 slow comparator 0xa, n+n Ohmic FEE64 0xf
System wide checks OK *except* aida02 WR decoder status
ADC data item stats - attachment 3
9/16 < 20k, max 234k (aida08)
per FEE64 Rate spectra - attachments 4-6
per FEE64 1.8.W spectra - 20us FSR - attachments 7-8
WR timestamps OK - attachment 9
18.50 FEE64 power OFF |
|
Mon Apr 26 09:41:21 2021, TD, Monday 26 April
|
10.42 All system wide checks OK *except*
Base Current Difference
aida01 fault 0xc960 : 0xc962 : 2
aida02 fault 0x8bbf : 0x8bc1 : 2
aida03 fault 0xae9e : 0xaea0 : 2
aida04 fault 0x6e3 : 0x6e5 : 2
aida05 fault 0x31a4 : 0x31ab : 7
aida05 : WR status 0x10
aida06 fault 0x22ea : 0x22f1 : 7
aida07 fault 0x438f : 0x4396 : 7
aida08 fault 0x61ed : 0x61f4 : 7
aida09 fault 0x1c32 : 0x1c33 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 25 8 28 8 3 2 2 3 2 3 4 : 27940
aida02 : 24 9 3 5 1 3 1 3 3 3 6 : 36408
aida03 : 24 6 5 4 3 3 1 3 3 3 6 : 36512
aida04 : 23 6 6 1 3 3 1 3 3 3 6 : 36428
aida05 : 15 21 27 7 4 2 2 3 3 3 4 : 29044
aida06 : 24 8 2 4 1 3 1 3 3 3 6 : 36352
aida07 : 29 24 28 9 3 3 2 2 2 3 4 : 27732
aida08 : 17 10 1 1 1 4 1 3 3 3 6 : 36356
aida09 : 41 49 18 14 3 4 3 4 2 3 4 : 29388
aida10 : 3 1 6 4 2 3 3 3 2 3 6 : 35828
aida11 : 22 7 5 4 2 1 2 4 2 3 6 : 35936
aida12 : 65 30 23 9 3 1 2 3 2 3 4 : 28100
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sat Apr 24 03:57:22 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
10.47 DSSSD bias |& leakage currents Ok - attachment 1
FEE64 temps OK - attachment 2
good event stats - attachment 3
merger time seq errors - attachment 4
Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5
DAQ continues OK - no data storage |
|
Mon May 24 09:57:07 2021, TD, Monday 24 May 19x
|
07.30 Hot HEC channels multiple FEE64s
ASIC check fixes
10.56 DAQ continues OK file S496/R18_75
System wide checks OK * except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b1 : 1
aida02 fault 0xfe72 : 0xfe73 : 1
aida03 fault 0x97f4 : 0x97f5 : 1
aida04 fault 0x950 : 0x951 : 1
aida05 fault 0x8e93 : 0x8e94 : 1
aida06 fault 0x545a : 0x545b : 1
aida07 fault 0xc04d : 0xc04e : 1
aida08 fault 0xa0f7 : 0xa0f8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 2 7 2 1 4 3 2 1 4 7 : 40452
aida02 : 13 3 3 1 3 4 2 2 2 4 7 : 41308
aida03 : 17 6 2 2 1 4 2 3 2 3 7 : 39700
aida04 : 8 6 1 0 1 3 1 4 2 3 7 : 39712
aida05 : 10 3 5 3 4 3 3 2 1 4 7 : 40560
aida06 : 4 4 2 0 2 3 2 3 3 4 7 : 42576
aida07 : 5 3 2 3 1 3 1 3 2 4 7 : 41324
aida08 : 11 5 1 1 2 3 3 2 2 4 7 : 41348
aida09 : 7 4 3 0 2 3 1 3 2 4 7 : 41324
aida10 : 11 4 2 2 2 3 2 4 1 4 7 : 41132
aida11 : 18 5 0 2 2 4 2 2 1 4 7 : 40240
aida12 : 2 3 0 1 2 3 2 2 2 4 7 : 41024
aida13 : 4 5 1 2 2 3 2 4 1 4 7 : 41096
aida14 : 9 3 1 2 1 4 2 2 2 4 7 : 41164
aida15 : 2 3 1 3 2 2 3 3 2 4 7 : 41744
aida16 : 5 5 2 0 3 3 2 4 1 4 7 : 41116
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
1.8.H spectra - attachments 6-8
1.8.L spectra - attachments 9 & 10
Stat spectra - attachments 11 & 12
1.8.W spectra - 20us FSR - attachments 13 & 14
ADC data items - attachment 15
FEE64 temperatures OK - attachment 16
DSSSD bias & leakage currents OK - attachment 17
Merger/Tape Server/Merger statistics - attachment 18
no merger errors reported since previous restart
analysis of file R18_74
max dead time 0.05%
10.17 ASIC check
All histograms, statistics, merger statistics zero'd |
|
Mon May 23 16:41:45 2022, TD, Monday 23 May
|
17.40 DAQ stopped
Attachment 1 - DSSSD bias & leakage current
Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased
Attachment 3 - FEE64 temps - OK |
|
Mon Dec 23 10:54:46 2024, TD, Monday 23 December 2024
|
11.54 re-established remote access via AnyDesk - details to follow
confirmed AIDA FEE64 power OFF, detector HV OFF
1. Establish port (to access carme-gsi) ssh -L 8080:proxy.gsi.de:8080 carme@atppc025
Establish port (to access aida-gsi) ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin
Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes
2. Check AnyDesk password for remote access has been set up
3. Close anydesk and check it is *really* closed using
ps -o pid= -C anydesk
kill -9 any process (as root)
4 Restart AnyDesk
systemctl restart anydesk (as npg) |
|
Mon Apr 12 16:49:40 2021, TD, Monday 22 April - pulser walkthrough & alpha background 6x
|
16.54 Pulser walkthrough
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xa
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
file NULL/R4
BNC PB-5 settings changed by terminal command line using the commands
set rep rate 25
set attenuation 5
set amplitude 10
set amplitude 9
:
:
set amplitude 1
set attenuation 1
set rep rate 2
aida07 & aida08 1.8.L spectra - see attachment 1
17.26 run ends OK file NULL/R4_27
17.26 file NULL/R5
amplitudes 0.6, 1, 1.4, 1.8, 2.2, 2.6, 3
aida07 & aida08 1.8.L spectra - see attachment 2
17.47 run ends OK NULL/R5_23
17.53 system wide checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 11 3 2 0 2 4 2 3 4 11 : 58888
aida02 : 7 11 4 2 0 4 3 2 3 3 7 : 40436
aida03 : 38 7 7 2 6 4 1 3 3 3 6 : 36864
aida04 : 19 10 5 2 1 3 1 4 2 4 15 : 74732
aida05 : 30 5 4 3 2 3 4 3 3 4 15 : 76096
aida06 : 24 4 3 3 1 4 2 3 2 4 7 : 41808
aida07 : 22 7 11 7 1 6 3 4 3 3 6 : 37984
aida08 : 13 7 5 2 1 4 3 3 1 4 6 : 36924
aida09 : 33 9 10 6 4 4 2 2 2 4 6 : 37676
aida10 : 15 7 7 1 1 3 1 3 2 3 7 : 39364
aida11 : 30 13 8 3 4 4 3 3 3 3 6 : 37312
aida12 : 17 9 4 3 2 3 2 2 3 3 7 : 40236
17.57 Slow comparator 0xa -> 0x64
BNC PB-5 Pulser OFF
file NULL/R9
alpha background
FEE64 temperatures OK - see attachment 3
statistics OK - see attachment 4
DSSSD bias & leakage currents OK - see attachment 5
tape server & merger - attachment 6
11:31 13.04.21
file NULL/R9 stopped for work in day
will resume alpha over night again
|
|
Sun Apr 21 23:20:45 2024, TD, Monday 22 April 38x
|
00.15 Data link to MBS was closed by MBS - reason(s) unknown
NH restarted MBS receiver and data transfer restarted
01.34 DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ADC data item stats OK - attachment 3
Merger, TapeServer etc - attachment 4
04.36 DSSSD bias & leakage current OK - attachment 5
FEE64 temperatures OK - attachment 6
ADC data item stats OK - attachment 7
Merger, TapeServer etc - attachment 8
07.54 data storage to local disk enabled
file aida-gsi:/TapeData/S100_alpha/R14_
09.36 DAQ continues, current data file R14_46
168Er setting
DSSSD bias & leakage current OK - attachments 9 & 10
leakage current DSSSD#1 ramping and cycling with beam spills
FEE64 temperatures OK - attachment 11
System wide checks OK *except* WR/FPGA errors - attachments 12-13
WR timestamps - attachment 14
aida04 out of seq
ADC data item stats - attachment 15
high rates observed all FEE64s beam on, beam off rates OK
per FEE64 Rate spectra - attachment 16
per FEE64 1.8.L spectra - attachments 17-18
per FEE64 1.8.W spectra - attachments 19-20
per FEE^4 1.8.H spectra - attachments 21-22
Merger, TapeServer etc - attachments 23-25
data rates to c. 10Mb/s beam on
09.55 aida04 rebooted - https://elog.ph.ed.ac.uk/DESPEC/586
10.01 DAQ restarted
data file R15
ADC data item stats - attachments 26-27
large variation in rates beam off/on
c. 10.30 beam off (for new FRS setting?)
11.25 analysis of data files R14_19 (beam ON) and R15_24 (beam OFF) - attachments 28-29
For beam ON max deadtime c. 11% (aida04), beam OFF max deadtime c. 0.1% (aida04)
Zero timewarps observed
Notice large # of PAUSE/RESUME data items for aida03 & aida04 presumably due to additional overhead of processing the time machine scalers
12.35 Beam off
DSSSD bias & leakage current OK - attachment 30
Can observe beam spill structure in DSSSD leakage currents - now slowly recovering with beam off
FEE64 temperatures OK - attachment 31
ADC data item stats OK - attachment 32
Merger, TapeServer etc - attachment 33
12.55 New ASIC settings 2024Apr22-12.55.38
*all* discs disabled to reduce load on aida04
slow comparator 0x14
BNC PB-5 settings (since yesterday evening)
Amplitude 10.0V
Attenuation x10
tau_d 1ms
Frequency 2Hz
Tail pulse
Polarity +
13.46 all histograms & stats zero'd
13.48 Beam off
per FEE64 Rate spectra - attachment 34
2Hz pulser to p+n FEE64s
per FEE64 1.8.W spectra - attachments 35-36
p+n FEE64 threshold = 6000 to capture pulser events
n+n FEE64 threshold = 8000 to capture noise
per p+n FEE64 1.8.L spectra - attachments 37-38
common x/y-scale - peak height proxy for peak width
aida09 pulser peak width 57 ch FWHM ~40keV FWHM ( 5 x sigma ~ 85keV )
14.28 168Eu setting (check only - setup still in progress) |
|
Mon Jun 21 10:21:50 2021, NH, Monday 21 June 8x
|
11.21 - DAQ continues OK (no storage mode)
11.21 - System checks ok *except*
Base Current Difference
aida01 fault 0x8944 : 0x8963 : 31
aida02 fault 0x9b36 : 0x9b55 : 31
aida03 fault 0xf26b : 0xf28a : 31
aida04 fault 0x57a0 : 0x57bf : 31
aida05 fault 0x4d3c : 0x4d95 : 89
aida05 : WR status 0x10
aida06 fault 0x640c : 0x6425 : 25
aida07 fault 0x255e : 0x2577 : 25
aida08 fault 0xffbd : 0xffd6 : 25
aida09 fault 0x87e : 0x895 : 23
aida10 fault 0xc557 : 0xc56e : 23
aida11 fault 0x683e : 0x6855 : 23
aida12 fault 0x1c35 : 0x1c4c : 23
aida13 fault 0x8f41 : 0x8f58 : 23
aida14 fault 0x1b47 : 0x1b5e : 23
aida15 fault 0x4e4 : 0x127e : 3482
aida16 fault 0xf570 : 0xf587 : 23
White Rabbit error counter test result: Passed 0, Failed 16
Base Current Difference
aida01 fault 0x0 : 0x2 : 2
aida02 fault 0x0 : 0x2 : 2
aida05 fault 0x0 : 0x3 : 3
aida06 fault 0x0 : 0x2 : 2
aida09 fault 0x0 : 0x2 : 2
aida10 fault 0x0 : 0x2 : 2
aida11 fault 0x0 : 0x2 : 2
aida12 fault 0x0 : 0x9560 : 38240
aida13 fault 0x0 : 0x15b9 : 5561
aida14 fault 0x0 : 0x2 : 2
aida15 fault 0x0 : 0x2 : 2
aida16 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 33 12 14 2 4 4 2 2 2 4 6 : 37636
aida02 : 40 7 15 1 3 4 3 2 2 4 6 : 37800
aida03 : 39 7 9 6 5 3 1 3 2 4 6 : 37860
aida04 : 27 7 5 3 3 4 2 3 2 4 6 : 37908
aida05 : 33 5 5 3 6 4 3 3 2 4 6 : 38364
aida06 : 45 12 3 4 4 2 2 3 4 3 6 : 37828
aida07 : 31 11 5 4 4 2 3 2 2 4 6 : 37540
aida08 : 31 6 8 5 3 4 3 2 2 4 6 : 37772
aida09 : 45 4 9 3 2 3 3 2 2 4 6 : 37572
aida10 : 37 11 10 2 3 4 3 3 3 3 6 : 37260
aida11 : 40 9 11 3 3 4 3 3 1 4 6 : 37304
aida12 : 44 11 6 5 3 4 3 3 1 4 6 : 37320
aida13 : 41 11 9 3 5 3 3 3 1 4 6 : 37292
aida14 : 31 12 4 6 5 3 3 3 3 3 6 : 37276
aida15 : 17 9 7 1 2 2 3 2 1 4 7 : 40348
aida16 : 41 12 5 4 5 3 3 3 1 4 6 : 37268
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
11.23 - figures
Grafana temp & leakage - attachment 1
Weather cooler this week in Germany, expect S4 to cool down eventually
Lost activity monitor - attachment 2
1.8.W spectra - 20us FSR - attachments 3 & 4
aida05 looks very noisy (?)
ADC data items - attachment 5
FEE64 temperatures OK - attachment 6
DSSSD bias & leakage currents - attachment 7
Merger/Tape Server/Merger statistics - attachment 8
Looks some new time seq error since Thursday
Experiment is runnign in S4 so no access possible this week
Data is being archived to lustre (GSI PB file storage) |
|
Mon Jun 20 09:26:23 2022, OH, TD, Monday 20 June 11x
|
10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
Merger and DAQ restarted - All happily sending data
Has been running for 10 minutes with no WR errors.
Rates from yesterday achieved again.
DTAS will move back into position at some point.
Once this is done, will perform pulser walkthrough.
11:14 DTAS now back in position for experiment
Slight increase in noise noticed across a few FEEs
Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
Less frequently gaining FPGA errors
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
Upon powering up again rates in AIDA08 slightly worse - attachment 2
13:26 No WR errors since the last power cycle.
21:16 Baseline system wide checks, zero histograms & stats
All system wide checks OK
per p+n FEE64 1.8.L spectra - attachment 3
aida01 pulser peak width 82 channels FWHM
per FEE64 rate spectra - attachment 4
adc, pause, resume and correlation scaler data item stats - attachments 5-8
FEE64 temps OK - attachment 9
DSSSD bias & leakage currents OK - attachments 10 & 11 |
|
Mon Apr 19 16:58:29 2021, LJW, Monday 19th April 16:00-20:00 Shift 6x
|
System Checks @ ~18:10 :
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0xd052 : 0xd05a : 8
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x23 : 35
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 6 4 0 1 5 2 3 2 3 6 : 35636
aida02 : 27 3 2 0 2 3 2 3 3 3 6 : 36516
aida03 : 33 9 2 1 1 3 2 3 3 3 6 : 36556
aida04 : 11 6 5 4 2 4 3 3 3 3 6 : 37036
aida05 : 15 8 7 4 3 4 4 2 3 3 6 : 36908
aida06 : 13 9 7 3 3 4 2 3 3 3 6 : 36876
aida07 : 1 3 3 2 2 3 1 3 3 3 6 : 36236
aida08 : 29 5 2 1 1 2 2 3 3 3 6 : 36380
aida09 : 3 8 2 0 2 3 2 2 2 4 6 : 36972
aida10 : 9 5 3 3 1 4 2 4 2 3 6 : 36124
aida11 : 5 7 3 2 2 2 2 4 2 3 6 : 35900
aida12 : 24 6 3 1 2 3 2 2 3 3 6 : 36064
System Checks @ ~19:55
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xdb9a : 0xdb9d : 3
aida06 fault 0xb74f : 0xb752 : 3
aida07 fault 0xd052 : 0xd05e : 12
aida08 fault 0xe4c1 : 0xe4c4 : 3
aida09 fault 0xaf4b : 0xaf4d : 2
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x27 : 39
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 18 10 2 2 1 5 2 3 2 3 6 : 35768
aida02 : 21 9 2 1 2 2 2 3 3 3 6 : 36444
aida03 : 13 5 6 1 2 2 2 3 3 3 6 : 36444
aida04 : 30 11 4 3 2 4 3 3 3 3 6 : 37104
aida05 : 10 9 8 4 4 4 4 2 3 3 6 : 36976
aida06 : 22 17 10 3 2 3 1 4 3 3 6 : 37088
aida07 : 20 8 4 1 2 4 1 2 3 3 6 : 35952
aida08 : 14 4 3 0 1 2 2 3 3 3 6 : 36296
aida09 : 24 8 4 0 1 3 3 3 3 3 6 : 36768
aida10 : 19 8 4 1 1 4 2 3 3 3 6 : 36652
aida11 : 6 6 1 2 3 1 2 4 2 3 6 : 35800
aida12 : 15 7 1 1 2 4 1 2 3 3 6 : 35876 |
|
Mon Apr 19 13:27:08 2021, RDP, Monday 19th April 12 noon shift
|
12.30 There appear to be fewer HEC events in aida09 cf. aida11 and upstream DSSSDs
Decrease aida09 asic #1-4 HEC fast comparator 0x2 -> 0x1
Online/nearline analyses of daat files indicates that this does not resolve the issue
Some timestamp issues
MERGE Data Link (23725): bad timestamp 11 3 0x8b620008 0x0287be94 0x00003dd78287be94 0x16773dd78287be94 0x16773dd78287beb2
MERGE Data Link (23725): bad timestamp 11 3 0x8b610040 0x073302ce 0x00003df3e73302ce 0x16773df3e73302ce 0x16773df3e73302ec
MERGE Data Link (23725): bad timestamp 11 3 0x8b610020 0x0a54b794 0x00003e4e4a54b794 0x16773e4e4a54b794 0x16773e4e4a54b7b2
MERGE Data Link (23725): bad timestamp 11 3 0x8b610020 0x052da21e 0x00003f68f52da21e 0x16773f68f52da21e 0x16773f68f52da232
MERGE Data Link (23725): bad timestamp 11 3 0x8b620008 0x0061489e 0x000040aa8061489e 0x167740aa8061489e 0x167740aa806148a8
MERGE Data Link (23725): bad timestamp 11 3 0x8b610008 0x04029d5e 0x00004169c4029d5e 0x16774169c4029d5e 0x16774169c4029d68
MERGE Data Link (23725): bad timestamp 11 3 0x8b630004 0x08dfcb14 0x0000417fd8dfcb14 0x1677417fd8dfcb14 0x1677417fd8dfcb28
MERGE Data Link (23725): bad timestamp 11 3 0x8b630020 0x08dfcb1e 0x0000417fd8dfcb1e 0x1677417fd8dfcb1e 0x1677417fd8dfcb28
13.50 Increased DSSSD# 3 bias from -100V
-100V -6.325uA
-110V -6.500uA
-120V -6.800uA
|
|
Mon Apr 19 01:08:40 2021, MA, Monday 19th April 00:00-08:00 8x
|
03:09 General check
Rates, temptature, voltages are OK attached 1, 2, 3, 4
****Clock Ckeck******
OK
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
****** White Rabbit check******
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c2c : 48
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
*****FPGA check ******
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x10 : 6
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
*****Memory check*****
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 6 12 11 7 2 3 2 4 2 3 6 : 36360
aida02 : 21 15 7 3 2 3 1 4 2 3 6 : 35996
aida03 : 9 10 10 2 4 3 3 3 2 3 6 : 36052
aida04 : 17 17 17 4 0 2 3 4 2 3 6 : 36444
aida05 : 25 7 3 1 1 1 2 3 2 4 6 : 37292
aida06 : 29 15 7 2 3 4 1 3 3 3 6 : 36700
aida07 : 17 17 6 5 2 3 2 3 3 3 6 : 36812
aida08 : 22 6 10 5 3 1 3 4 2 3 6 : 36360
aida09 : 18 28 14 4 2 3 3 3 2 3 6 : 36232
aida10 : 26 16 4 3 3 1 3 4 2 3 6 : 36296
aida11 : 16 10 4 0 3 3 1 4 2 3 6 : 35856
aida12 : 19 14 4 2 4 4 1 3 3 3 6 : 36668
04:15 General Check
Rates, Tempratures, Voltages are ok, attached 5,6,7,8
******Clock Check*******
OK
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******ADC******
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******White Rabbit check******
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c39 : 61
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
******FPGA Check******
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x11 : 7
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
****** Memorey check******
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 21 10 7 1 3 2 4 2 3 6 : 36436
aida02 : 28 21 3 3 1 3 1 4 2 3 6 : 35944
aida03 : 15 8 11 3 3 3 3 3 2 3 6 : 36044
aida04 : 25 19 14 4 1 3 4 3 2 3 6 : 36380
aida05 : 12 6 5 1 0 2 2 3 2 4 6 : 37328
aida06 : 29 21 4 2 3 4 1 3 3 3 6 : 36700
aida07 : 19 14 7 6 2 3 2 2 3 3 6 : 36332
aida08 : 11 8 8 6 3 1 3 4 2 3 6 : 36332
aida09 : 16 25 11 3 1 2 2 4 2 3 6 : 36184
aida10 : 14 13 4 2 3 2 2 4 2 3 6 : 36064
aida11 : 4 9 4 0 3 3 1 4 2 3 6 : 35800
aida12 : 38 17 4 2 4 4 1 3 3 3 6 : 36768
|
|