 Sat Jun 5 11:20:15 2021, TD, Saturday 5 June 9x
|
12.25 DAQ continues OK - file S496/R32_48
alpha background
All system wide checks OK *except*
Base Current Difference
aida01 fault 0x8944 : 0x8946 : 2
aida02 fault 0x9b36 : 0x9b38 : 2
aida03 fault 0xf26b : 0xf26d : 2
aida04 fault 0x57a0 : 0x57a2 : 2
aida05 fault 0x4d3c : 0x4d3d : 1
aida05 : WR status 0x10
aida06 fault 0x640c : 0x640d : 1
aida07 fault 0x255e : 0x255f : 1
aida08 fault 0xffbd : 0xffbe : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 10 6 1 2 4 4 2 2 1 4 7 : 40360
aida02 : 4 3 3 3 3 4 2 2 1 4 7 : 40312
aida03 : 13 5 4 4 2 4 1 3 2 3 7 : 39580
aida04 : 16 5 3 3 2 4 1 3 2 3 7 : 39544
aida05 : 8 6 1 2 3 3 1 3 1 4 7 : 40416
aida06 : 4 3 2 1 3 3 1 3 1 4 7 : 40360
aida07 : 6 6 2 4 4 2 3 2 1 4 7 : 40424
aida08 : 6 4 1 1 4 3 3 2 1 4 7 : 40424
aida09 : 16 1 3 1 0 2 2 3 1 4 7 : 40344
aida10 : 2 5 1 4 1 3 2 4 2 3 7 : 40064
aida11 : 14 3 2 1 4 4 3 3 2 3 7 : 40080
aida12 : 11 2 2 4 4 4 3 3 2 3 7 : 40156
aida13 : 11 3 2 2 3 4 3 3 2 3 7 : 40036
aida14 : 6 5 0 4 3 4 3 3 2 3 7 : 40064
aida15 : 11 6 3 1 3 1 2 4 1 4 7 : 40940
aida16 : 12 5 0 4 4 4 3 3 2 3 7 : 40152
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
1.8.W spectra - 20us FSR - attachments 2 & 3
ADC data items - attachment 4
FEE64 temperatures OK - attachment 5
DSSSD bias & leakage currents OK - attachment 6
Merger/Tape Server/Merger statistics - attachment 7
no merger errors reported since previous restart
analysis of file S496/R32_47 - attachment 8
max deadtime 0.35% aida04
13.56 Lost activity monitor - see attachment 9 |
|
Sat May 4 07:36:07 2024, TD, Saturday 4 May 11x
|
08.32 DSSSD bias & leakage currents - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC, correlation scalers data item stats OK attachments 4-5
11/16 <20k max aida16 174k
per FEE64 Rate spectra - attachmnt 6
09.44 per p+n FEE64 1.8.L spectra - attachment 7
aida09 pulser peak width 58 ch FWHM
per FEE64 1.8.W spectra - attachments 8-10
09.47 bias cycle - off and then on
no significant change in leakage current
09.57 FEE64 power OFF
DSSSD bias ON |
|
Sat May 29 13:13:45 2021, TD, Saturday 29 May 9x
|
14.10 DAQ continues - file S496/R18_539
alpha background
increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem
All system wide checks OK *except*
Base Current Difference
aida01 fault 0x82b0 : 0x82b7 : 7
aida02 fault 0xfe72 : 0xfe79 : 7
aida03 fault 0x97f4 : 0x97fb : 7
aida04 fault 0x950 : 0x957 : 7
aida05 fault 0x8e93 : 0x8e97 : 4
aida06 fault 0x545a : 0x545e : 4
aida07 fault 0xc04d : 0xc051 : 4
aida08 fault 0xa0f7 : 0xa0fb : 4
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 8 7 2 2 2 2 2 2 4 6 : 37036
aida02 : 14 3 5 2 2 4 3 3 2 4 6 : 37984
aida03 : 15 11 8 6 3 3 2 3 3 3 6 : 36884
aida04 : 43 13 5 5 3 4 2 3 3 3 6 : 37060
aida05 : 19 3 5 4 2 2 2 2 2 4 6 : 37044
aida06 : 17 4 1 3 2 3 3 4 2 4 6 : 38356
aida07 : 16 6 2 1 0 2 2 3 2 4 6 : 37296
aida08 : 21 8 1 1 3 4 4 3 2 4 6 : 38276
aida09 : 13 10 2 1 2 3 2 2 2 4 6 : 37060
aida10 : 13 9 4 1 1 3 3 3 1 4 6 : 36764
aida11 : 13 9 2 1 2 3 3 3 1 4 6 : 36796
aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724
aida13 : 7 6 6 2 1 3 2 2 2 4 6 : 37036
aida14 : 9 7 7 1 1 3 2 2 2 4 6 : 37036
aida15 : 14 5 2 2 1 3 3 4 2 4 6 : 38272
aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.24 All histograms, statistics and merger statistics zero'd
16.16 analysis of file S496/R18_541 - attachment 9
max deadtime 0.12% aida04
18.40 DAQ continues - file S496/R18_551
alpha background
increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem |
|
Sat Jun 25 06:55:07 2022, MS, Saturday 25 June 2022 8:00-16:00 15x
|
Took over the night shift from Tom.
7:00 The beam is back.
8:00 : attachments 1-3
10:00 : attachments 4-6
12:00 : attachments 7-9
14:00 : attachments 10-12
16:00 : attachments 13-15
|
|
Sat Jun 25 17:06:42 2022, BA, AA, Saturday 25 June 2022 16:00-00:00 9x
|
Took over the night shift from Magda
18:00 : attachments 1-3
20:00 : attachments 4-6
22:00 : attachments 7-9
00:00 : attachments 10-12
|
|
Fri Jun 24 22:36:45 2022, TD, Saturday 25 June 00:00-08:00 31x
|
23:36 Check ASIC contorol
Zero stats & all histograms
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 -> 0xa
all waveform AD9252 ADCs disabled
all fast discs disabled
BNC PB-5 settings (to p+n FEE64s only)
amplitude 1.0V
attenuator x1
decay time 1ms
polarity +
frequency 22Hz
analysis of file S505/R5_154 - attachment 1
zero timewarps
max deadtime aida02 c. 17%, all other FEE64s < 5%
stats - attachments 2-6
per FEE64 1.8.H spectra - attachments 7 & 8
per p+n FEE64 1.8.L spectra - attachment 9
aida01 pulser peak width 101 ch FWHM
per FEE64 stat & rate spectra - attachments 10 & 11
adc data item stats - attachment 12
FEE64 temps OK - attachment 13
DSSSD bias & leakage currents OK - attachments 14 & 15
00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17
03:03
analysis of file S505/R5_189 - attachment 18
zero timewarps
max deadtime aida02 c. 17%, all other FEE64s < 5%
all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20
adc data item stats - attachment 21
FEE64 temps OK - attachment 22
DSSSD bias & leakage currents OK - attachment 23
03:30
per FEE64 1.8.H spectra - attcahments 24 & 25
06:09 no beam
analysis of file S505/R5_221 - attachment 26
zero timewarps
all dead times << 1%
all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28
adc data item stats - attachment 29
FEE64 temps OK - attachment 30
DSSSD bias & leakage currents OK - attachment 31 |
|
Sat Apr 24 10:21:06 2021, NH, Saturday 24.04.2021
|
I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again
Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)
System Wide Checks
Clocks: OK
FADC: OK
WR:
Base Current Difference
aida05 fault 0x31a2 : 0x31a2 : 0
aida05 : WR status 0x10
aida06 fault 0x22e7 : 0x22e8 : 1
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK
aida05 ASIC maybe didn't complete setup properly (WR 0x10?)
Timestamps: OK (1)
Temps: OK
Will start gzipping S460 runs for migration to lustre & tape |
|
Fri May 21 23:19:23 2021, TD, Saturday 22 May 00:00-08:00 29x
|
00.16 413Gb remaining /TapeData filesystem
estimate 413/2 x 177s = 10.15h to filesystem full - OK
00.18 All histograms, statistics and merger statistics zero'd
00.57 All system wide checks OK *except*
Base Current Difference
aida05 fault 0x500 : 0x572 : 114
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 9 3 4 1 5 2 2 3 3 6 : 36356
aida02 : 4 8 7 2 4 3 3 3 2 3 6 : 35968
aida03 : 12 9 6 2 5 3 2 3 3 3 6 : 36824
aida04 : 11 5 9 2 2 2 2 4 3 3 6 : 37028
aida05 : 24 9 2 1 2 3 2 3 4 3 6 : 37608
aida06 : 25 6 3 4 2 4 3 3 3 3 6 : 37060
aida07 : 3 3 3 1 3 3 2 2 2 2 7 : 37044
aida08 : 20 4 2 3 1 3 2 2 3 3 6 : 36016
aida09 : 15 8 2 5 0 3 3 2 3 3 6 : 36284
aida10 : 16 6 4 1 2 3 3 2 2 4 6 : 37328
aida11 : 20 2 2 1 1 3 2 2 3 3 6 : 35936
aida12 : 1 2 1 2 3 4 1 2 4 3 6 : 36900
aida13 : 25 10 4 1 0 3 2 3 3 3 6 : 36500
aida14 : 20 6 9 2 2 5 1 2 2 4 6 : 37200
aida15 : 31 9 5 2 3 3 1 3 3 3 6 : 36500
aida16 : 24 4 2 3 1 4 2 3 3 3 6 : 36672
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
01.12 Lost activity monitor - attachment 1
1.8.H spectra - attachments 2 & 3
1.8.L spectra - attachments 4-6
aida01 pulser peak width 120 ch FWHM
Rate specttra - attachments 7 & 8
01.06 Lost activity monitor - attachment 9
ucesb - attachment 10
Grafana - DSSSD bias & leakage currents past 7 days - attachment 11
statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17
FEE64 temperatures OK - attachment 18
DSSSD bias & leakage currents OK - attachment 19
Merger/Tape Server/Merger stats - attachment 20
no merger warning/error messages since last restart
01.26 Analysis file S496/R14_1798 - attachment 21
aida08 5.5% deadtime
01.17 Disk space remaining /TapeData filesystem
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 706M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.4T 395G 95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 395/2 x 177s = 9.7h to filesystem full - OK
01.30 despec shift reports no data reported by ucesb
aida stats & tape server ok - no data being forwarded to MBS by data relay
01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811
04.35 System wide checks all OK *except*
Base Current Difference
aida05 fault 0x500 : 0x573 : 115
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 661M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.5T 337G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 337/2 x 177s = 8.3h to filesystem full - OK
FEE64 temperatures OK - attachment 22
ADC data item statistics OK - attachment 24
DSSSD bias & leakage current OK - attachment 26
analysis of file S496/R14_1920 - attachment 28
aida04 deadtime 5.0%
07.10 Ssystem wide checks all OK *except*
Base Current Difference
aida05 fault 0x500 : 0x573 : 115
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 662M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.5T 287G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 287/2 x 178s = 7.1h to filesystem full - OK
FEE64 temperatures OK - attachment 23
ADC data item statistics OK - attachment 25
DSSSD bias & leakage current OK - attachment 27
analysis of file S496/R14_1920 - attachment 29
aida04 deadtime 4.4%
07.35 Found MBS data monitor kill'd
Restarted MBS data relay and data monitor OK
08.00 beam off - end of S496
08.28 Tape Server -> no storage mode
DAQ continues
09.58 All slow comparators -> 0x64
BNC PB-5 OFF
file S496/R15
alpha background
Tape server 200kb/s, merger 260k data items/s |
|
Sat May 22 13:40:56 2021, TD, Saturday 22 May 23x
|
14.37 DAQ continues file S496/R15_14
Grafana - DSSSD bias & leakage current - attachment 1
1.8.L spectra - attachments 2 & 3
Stat spectra - attachments 4 & 5
FEE64 temperatures OK - attachment 6
stats - ADC data items, pause items OK - attachments 7 & 8
DSSSD bias & leakage current OK - attachment 9
Merger/Tape Server/Merger stats - attachment 10
analysis of file S496/R14_1920 - attachment 11
all FEE64 deadtime 0% as expected
17.30 aida01, 3, 5, 9 & 11 have hot HEC channels
ASIC check fixes
All histogram, statistics & merger statistics zero'd
DAQ continues
Note gamma-ray source tests have started with FATIMA and DEGAS
20.20 Analysis file R15_19 - attachment 12
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 665M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.3T 530G 93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 90/2 x 3168s = 40h to filesystem full - OK
20.22 All system wide checks OK *except*
Base Current Difference
aida01 fault 0xf82c : 0xf82f : 3
aida02 fault 0xdda6 : 0xdda9 : 3
aida03 fault 0xb13d : 0xb140 : 3
aida04 fault 0xf4cd : 0xf4d0 : 3
aida05 fault 0x500 : 0x575 : 117
aida06 fault 0xc6e7 : 0xc6e9 : 2
aida07 fault 0x4920 : 0x4922 : 2
aida08 fault 0xbd9a : 0xbd9c : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 8 4 6 6 1 4 3 3 4 3 6 : 38048
aida02 : 11 8 10 3 4 4 3 4 3 3 6 : 37740
aida03 : 2 8 3 3 2 2 1 4 2 4 6 : 37720
aida04 : 26 15 17 4 4 2 1 4 2 4 6 : 38256
aida05 : 15 6 4 1 1 2 1 3 3 4 6 : 38156
aida06 : 7 7 16 6 3 4 2 2 3 4 6 : 38612
aida07 : 11 5 12 3 3 3 2 3 1 3 7 : 38836
aida08 : 13 6 7 2 1 2 2 2 3 4 6 : 37972
aida09 : 23 3 8 5 0 5 3 3 4 3 6 : 38164
aida10 : 12 5 7 1 2 2 4 3 2 4 6 : 37992
aida11 : 19 2 7 1 2 2 1 2 3 4 6 : 37740
aida12 : 21 5 3 3 3 3 1 2 3 4 6 : 37964
aida13 : 18 7 7 1 2 4 2 3 2 4 6 : 37776
aida14 : 8 3 13 7 2 4 2 3 2 4 6 : 37992
aida15 : 11 5 14 1 3 4 2 3 2 4 6 : 37908
aida16 : 17 6 6 2 2 3 1 2 3 4 6 : 37908
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
19.30 Lost activity monitors - attachments 13-15
20.54 1.8.W spectra 20us FSR attachments 16 & 23
1.8.W spectra 200us FSR attachments 17 & 22
1.8.W spectra 2ms FSR attachments 18 & 21
1.8.W spectra 20ms FSR attachments 19 & 20 |
|
Sat May 21 15:30:45 2022, TD, Saturday 21 May
|
16.31 DAQ stopped
Attachment 1 - DSSSD bias & leakage current
Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased
Attachment 3 - FEE64 temps - OK |
|
Sat Apr 20 17:52:27 2024, TD, NH, Saturday 20 April contd.
|
Note 40kBq 60Co source placed on bottom surface of snout centred on bPlas detectors and DSSSDs
18.50 Primary beam 750MeV/u Er to S4
Primary beam stopped by flange (5cm s/steel in position upstream of implantation stack)
Observe c. 2MeV dE peak in AIDA DSSSD#1 and DSSSD#2 using MIDASsort online analysis - attachment 1
21.50 Primary beam 1000MeV/u Er to S4 - attachment 2
per DSSSD e_front versus e_back, 20keV/channel
DSSSD #1 x & y -axis projections
10.50 Overnight stats - attachment 3
Primary beam ?
per DSSSD e_front versus e_back, 20keV/channel
DSSSD #1 x & y -axis projections
note multiple dE peaks observed in both DSSSDs |
|
Sat Apr 20 03:20:23 2024, TD, Saturday 20 April 23x
|
Note 60Co source (current activity c. 43kBq) was installed yesterday evening - assume it is positioned on the bottom side of the snout as per the other sources used this week/
04.17 DSSSD bias & leakage current OK - Grafana - attachment 1
FEE64 temperatures OK - attachment 2
aida02 ASIC temp known to be u/s
ADC data item stats - attachment 3
all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 5
11.24 DSSSD bias & leakage current OK - Grafana - attachment 6
FEE64 temperatures OK - attachment 7
aida02 ASIC temp known to be u/s
ADC data item stats - attachment 8
all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs
per FEE64 Rate spectra - attachment 9
Merger etc - attachment 10
17.50 DSSSD bias & leakage current OK - Grafana - attachment 11
FEE64 temperatures OK - attachment 12
aida02 ASIC temp known to be u/s
ALl syhstem wide checks OK *except* WR/FPGA errors - attachments 13-14
WR timestamps OK - attachment 15
per FEE64 Rate spectra - attachment 16
per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
Merger, TapeServer, Data Links - attachments 19-20
no data storage
ADC data item stats - attachment 21
all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs
18.01 Grafana alerting shows detail of DSSSD leakage current and S4 temperature variations over time - attachment 22
DEGAS LN2 fill cycle every 6h 10.00 - 16.00 - 22.00 ... etc
https://despec-vm-01.gsi.de/grafana/d/tHRVJYfGz/aida-alerting?orgId=1 |
|
Sat Apr 2 10:42:49 2022, TD, Saturday 2 April 21x
|
11.10 Found aida08 not running - merger/TapeServer stalled
Restart required power cycle of all FEE64s
*all* ADC calibrations failed during system wide checks it was necessary to use FADC align & control (act on all FEE64 modules selected) to calibrate - first pass
12/16 calibrated, successive individual re-calibrations calibrated remaining FEE64s
ASIC settings 2021Apr29-13-16-00
slow comparator 0x2f
BNC PB-5
amplitude 1.0V
attenuation x1
polarity +
tail pulse
tau_d 1ms
rate 25Hz
Attachments 1-4 1.8.W n+n spectra 20us, 200us, 2ms, 20ms FSR
Attachments 5-8 1.8.W p+n spectra 20ms, 2ms, 200us, 20us FSR
Attachment 9 per FEE64 1.8.L spectra n+n strips
aida04 pulser peak width 387 ch FWHM
Attachment 10 per FEE64 1.8.L spectra p+n strips
aida10 pulser peak width 93 ch FWHM
Attachment 11 per FEE64 Rate spectra
Attachments 12-15 all system wide checks OK
Attachment 16 ADC data item stats
Attachment 17 FEE64 temps OK
Attachment 18 DSSSD bias & leakage currents OK
15.45 check of ASIC settings cf. https://www2.ph.ed.ac.uk/~td/AIDA/Information/Default%20AIDA%20ASIC%20Parameters.txt
Differences from default AIDA ASIC parameters
---------------------------------------------
Input Polarity
Offset Title Positive Negative Notes
12 slow comparator threshold 0x2f->0xa 100keV nominal
14 fast comparator threshold HEC 0xf->0x02 200MeV nominal
15 fast comparator threshold LEC/MEC 0xf->0xff > LEC/MEC nominal FSR
18 preAmp reference 0xb2 0x20->0x30
21 Ibias LF feedback 0x08 0xb->0xf
26 diode link threshold 0xca->0xbf 0x11->0x23 decrease switch threshold
(front) p+n junction strips - positive input polarity
(back) n+n ohmic strips - negative input polarity
All looks OK - variations from default settings are those used at RIKEN
16:45 FEE64 adaptor PCB config
LK1 LK2 LK3 LK4
p+n n/c o * o
n+n o o n/c o
o = jumper fitted
n/c = jumper not fitted
* = top, middle FEE64s - n+n bias ring ground for Si wafer #2
bottom, middle FEE64s - DSSSD PCB gnd
DSSSD#1 & #2 bias via standard coaxial cable to top p+n FEE64 adaptor PCBs
17.05 aida09 or aida13 cable braid not secured to ground - cable tag dia too large - now sandwiched between correctly sized cable tags
no obvious difference to adc data item rates
17.20 LK1 removed for all n+n PCBs
attachment 19 - adc data item stats
18.55 LK2/LK4 removed for all n+n PCBs
attachment 20 - adc data item stats
19.10 power cycle FEE64s
attachment 21 - adc data item stats |
|
Sat Apr 17 11:27:38 2021, JS, TD, Saturday 17th April 12:00-16:00 10x
|
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
13:55 CEST
Statistics : ok elog:234/4
Temp : ok elog:234/5
Bias : ok elog:234/6
ucesb : ok
DB: No faults found
ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and
Control browser page to rerun calibration for that module
White Rabbit Check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
14:05 no beam
14:10 beam back
14:15 no beam, beam current being optimised, going to thicker degrader when beam returns
14:37 CEST
Statistics : ok
Temp : ok
Bias : ok (ch3 now over 6uA)
ucesb : ok
DB: No faults found
15:03 CEST
Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found
15:31 CEST
Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found
15:50
Statistics : ok elog:234/7
Temp : ok elog:234/8
Bias : ok elog:234/9
ucesb : ok elog:234/10
DB: No faults found
ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last |
|
Sat Apr 17 19:39:56 2021, DJ, TD, Saturday 17 April 20.00-00.00 13x
|
19.37 per FEE64 rate spectra - attachments 1 & 2
1.8.L spectra - attachments 3 & 4
1.8.H spectra - attachments 5-8
1.8.W spectra - attachments 9 & 10
20.42 DAQ contrinues file NULL/R30_233
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bd7 : 10
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x4 : 4
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
-------
23:48
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bde : 17
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x7 : 7
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 5 6 2 1 3 1 3 3 3 6 : 36336
aida02 : 7 9 10 3 2 2 2 4 2 3 6 : 36068
aida03 : 26 7 8 4 3 4 2 4 2 3 6 : 36448
aida04 : 18 8 2 1 2 2 3 4 2 3 6 : 36168
aida05 : 24 7 9 4 3 1 3 2 2 4 6 : 37352
aida06 : 17 10 3 1 3 5 1 3 3 3 6 : 36644
aida07 : 7 11 2 2 3 2 3 2 4 3 6 : 37268
aida08 : 23 4 1 5 1 1 2 3 3 3 6 : 36332
aida09 : 14 6 3 2 1 3 2 3 3 3 6 : 36504
aida10 : 2 2 2 2 1 2 2 2 3 3 6 : 35768
aida11 : 2 2 1 2 2 2 2 4 2 3 6 : 35816
aida12 : 26 8 5 5 4 3 1 3 3 3 6 : 36632
---- |
|
Sat Apr 17 15:29:27 2021, MS, Saturday 17 April 16:00-20:00 2021 6x
|
16:21
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Sat Apr 17 06:14:30 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
18:29
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bd1 : 4
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 21 3 4 4 1 3 1 3 3 3 6 : 36332
aida02 : 4 3 13 2 2 3 2 4 2 3 6 : 36152
aida03 : 12 7 5 2 2 3 2 4 2 3 6 : 36088
aida04 : 17 6 9 5 1 1 2 3 3 3 6 : 36452
aida05 : 26 8 0 3 1 1 2 3 3 3 6 : 36296
aida06 : 22 6 9 4 3 4 2 3 3 3 6 : 36952
aida07 : 17 7 3 1 2 3 3 2 3 3 6 : 36300
aida08 : 24 5 4 4 1 1 2 3 3 3 6 : 36360
aida09 : 15 11 3 2 1 3 1 3 3 3 6 : 36292
aida10 : 2 3 5 2 1 2 2 2 3 3 6 : 35824
aida11 : 2 2 0 0 3 2 2 4 2 3 6 : 35800
aida12 : 18 15 7 5 4 3 1 3 3 3 6 : 36688
|
|
Fri Apr 16 23:13:57 2021, CA, MA, BA, Saturday 16th 00:00 - 08:00 14x
|
00:05 DESPEC tuning beam, changing spill length
00:21 DAQ continues ok
Merger/TapeServer ok
00:25 still no beam in AIDA
00:30 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok - attachment 1
00:52 beam in AIDA - rates spectra - attachment 2
01:05 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok - attachment 3
beam is off again
01:30 beam back
01:32 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok - attachment 4
01:37 beam off - FRS increasing spill length to 5s
01:48 beam back with new spill setting - 5s on spill, 2s off spill
02:07 system wide checks ok except;
WR decoder status
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb42 : 8
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA check :
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
02:10 FEE64 Temps ok - attachment 5
02:20 good event statistics ok - attachment 6
02:27 detector bias / leakage currents ok - attachment 7
02:30 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok
beam off
03:00 stats/ucesb/database checks ok
03:30 stats/ucesb/database checks ok
still no beam in AIDA
05:11 system is checked, beam is back with low intensity
05:13 FEE64 Temps ok - attachment 8
05:13 good event statistics ok - attachment 9
05:13 detector bias / leakage currents ok - attachment 10
05:33 stats/ucesb/database checks ok
06:00 AIDA crashes, lose connection to aida09
06:20 aida08 does not respond on powercycle and reset
06:30 Called TD
06:50 successfully reset AIDA. Writing to file R30
synchronise ASIC clocks:
FEE64 module aida01 => 7
FEE64 module aida02 => 7
FEE64 module aida03 => 7
FEE64 module aida04 => 7
FEE64 module aida05 => 7
FEE64 module aida06 => 7
FEE64 module aida07 => 7
FEE64 module aida08 => 7
FEE64 module aida09 => 7
FEE64 module aida10 => 7
FEE64 module aida11 => 7
FEE64 module aida12 => 7
ASIC Clock Timestamp check test result: Passed 12, Failed 0
Database options file size check:
If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Sat Apr 17 06:14:30 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
System wide checks ok *except*
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
re-calibration of these FEE modules unsuccessful in FADC align and control.
07:05 FEE64 Temps ok - attachment 12
good event statistics ok - attachment 13
detector bias / leakage currents ok - attachment 14
07:10 stats/ucesb/database checks ok
07:30 stats/ucesb/database checks ok
|
|
Sat Jun 15 00:28:01 2024, TD, Saturday 15 June 45x
|
Beam lost due to septum failure c. 18.15 Friday 14 June
Cooling of recirculating water failed c. 19.00 - water temperature increased from the normal 17-18deg C to 25-26 deg C
FEE64 temperatures increased and FEE64s powered off and on-call engineer requested.
Cooling of recirculating water restored and AIDA restarted
NH started background run 23.37 - data file S181/R9
01.21 DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ADC data item stats OK - attachment 3
8x < 20k, max 238k aida08
per FEE64 Rate spectra - attachment 4
2x Check ASIC load
Merger, TapeServer etc - attachments 5-6
analysis data file R9_27 - attachment 7
max deadtime 1.9%, all other FEE64s < 1%
ADC data rate 951k, HEC data rate 1.6kHz
04.28 DSSSD bias & leakage current OK - attachment 8
FEE64 temperatures OK - attachment 9
ADC data item stats OK - attachment 10
8x < 20k, max 261k aida08
per FEE64 Rate spectra - attachment 11
08.30 DSSSD bias & leakage current OK - attachments 12-13
FEE64 temperatures OK - attachment 14
ADC data item stats OK - attachment 15
all rates increased - gamma source in use?
per FEE64 Rate spectra - attachments 16-17
note aida08 asic #3 hot HEC channel
per FEE64 1.8.L spectra - attachments 18-20
aida09 pulser peak width 59 ch FWHM
decay alphas aida05, aida12, aida02, aida04
per FEE64 1.8.W spectra - 20us FSR - attachments 21-23
Merger, TapeServer etc - attachments 25-25
analysis data file R9_152 - attachment 26
max deadtime 2.4%, all other FEE64s < 1%
ADC data rate 1.215M, HEC data rate 2.1kHz
12.22 DSSSD bias & leakage current OK - attachment 27
FEE64 temperatures OK - attachment 28
ADC data item stats OK - attachment 29
per FEE64 Rate spectra - attachment 30
15.27 DSSSD bias & leakage current OK - attachment 31
FEE64 temperatures OK - attachment 32
ADC data item stats OK - attachment 33
all rates increased - 370kBq 152Eu in use
per FEE64 Rate spectra - attachments 34
per FEE64 1.8.L spectra - attachments 35-36
aida09 pulser peak width 59 ch FWHM
decay alphas aida05, aida12, aida02, aida04
analysis data file R9_275 - attachment 37
max deadtime 4.4%, all other FEE64s < 2%
ADC data rate 1.520M, HEC data rate 2.3kHz
18.52 DSSSD bias & leakage current OK - attachment 38
FEE64 temperatures OK - attachment 39
ADC data item stats OK - attachment 40
per FEE64 Rate spectra - attachment 41
22.50 DSSSD bias & leakage current OK - attachment 42
FEE64 temperatures OK - attachment 43
ADC data item stats OK - attachment 44
per FEE64 Rate spectra - attachment 45 |
|
Sat Mar 14 07:13:59 2020, DK, TD, LS, CG, Saturday 14th March 08:00 - 24:00 41x
|
8:15
All system wide checks passed
Leak currents are continuing to creep up - attachment #1
FEE temperatures fine - attachment #2
Stats about where they have been - attachment #3
Implants look as previously (5ish counts per channel in SSD1 and SSD2) - attachment #4
Merger fine, 3 Mega items / sec
TapeServer still writing about 25 MB/s
09.37 analysis of data file S480/R12_443
deadtime aida09 c. 5%, all other FEE64s <<1%
See attachment #5
09.54.51
biases, SSD1 and 2 over 2 uA each now - attachment #6
11:20
All system-wide checks okay
Merger okay, 3 Mega items / sec
TapeServer normal, 27 MB/sec
Biases as attachment #7
Stats as #8
Temps as #9
Histos as #10
14:10
All system wide checks okay
Stats #11
Temps #12
Implants #13
Leak currents are getting a bit higher #14
15:30
All system wide checks passed
Stats attachment #15
Temps attachment #16
Implants attachment #17
Biases / leak currents attachment #18
Merger okay, 3 Mega items / sec
TapeServer okay, writing 27 MB/s
16.12 all online spectra zero'd
16.31 stat & rate spectra - attachments 19-22
1.8.L spectra - attachments 23 & 24
aida03 pulser peak 57 ch FWHM
1.8.H spectra - attachments 25 & 26
NewMerger & TapeServer - attachments 27 & 28
18.29
system wide checks all okay
FEE temperatures normal (attachment 29)
Leakage currents okay and recorded to spreadsheet (attachment 30)
Good event statistics okay (attachment 31)
Merger running at 3M items/sec
Tape server running at 25MB/sec
20.25
system wide checks all okay
FEE temperatures normal (attachment 32)
Leakage currents okay and recorded to spreadsheet (attachment 33)
Good event statistics okay (attachment 34)
Merger running at 3M items/sec
Tape server running at 26MB/sec
as of 20.39 just finished R12_961
21.36 beam is down confirmed by rates histogram (attachment 35), informed others on shift
was down for maybe 5 minutes is back on at 21.40 (attachment 36)
think between R12_1004 to R12_1008
22.10
system wide checks all okay
FEE temperatures normal (attachment 37)
Leakage currents okay and recorded to spreadsheet (attachment 38)
Good event statistics okay (attachment 39)
Merger running at 3M items/sec
Tape server running at 26MB/sec
23.03 someone comes to ask if we see an increase in implantation rate as they do their side
I checked the rates histogram and good events and see nothing that would indicate an increase (attachment 40 and 41)
FEE temperatures and leakage currents do not seem any different either
believe they are calling someone about the issue
23.30 leaving now to catch bus but people on night shift have been told how to check on AIDA and know that CG is on call if needed |
|
Fri Mar 13 23:06:39 2020, CA, OH, Saturday 14th March 00:00 - 08:00 19x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
00.00 DAQ momentarily set to NoStorage
OH changes symbolic link such that AIDA is writing directly to mounted hard-drives ("SecondDrive" and "ThirdDrive")
00.02 DAQ now writing to file again - /media/SecondDrive/TapeData/S480/R12_0
00.30 all system wide checks ok
FEE64 temperatures ok - attachment 1
good event statistics ok - attachment 2
detector bias and leakage currents ok - attachment 3
00:43 merger ok - writing at ~ 3 million items/s
TapeServer ok - writing at ~ 27 Mb/s
Data forwarding to MBS ok
02:14 all system wide checks ok
FEE64 temperatures ok - attachment 4
good event statistics ok - attachment 5
detector bias and leakage currents ok - attachment 6
02:17 Merger ok ~ 2.7 million items/s
TapeServer ok ~ 27 Mb/s
Data forwarding to MBS ok
02:18 currently writing to file /media/SecondDrive/TapeData/S480/R12_110
02:57 attachments 7 & 8 - high energy channel spectra
attachment 9 - hit rate spectra
04:21 FEE64 temperatures ok - attachment 10
good event statistics ok - attachment 11
detector bias and leakage currents ok - attachment 12
system wide checks ok - except WR decoder status (see attachment 13)
04:26 check ASIC control
04:30 system functions -> select required function -> collect all WhiteRabbit status error counters for baseline
WR decoder status check now ok
04:50 Merger ok ~ 3 million items/s
TapeServer ok ~ 25 Mb/s
Data forwarding to MBS ok
04:52 currently writing to file /media/SecondDrive/TapeData/S480/R12_231
06:11 all system wide checks ok
FEE64 temperatures ok - attachment 14
good event statistics ok - attachment 15
detector bias and leakage currents ok - attachment 16
06:41 Merger ok ~ 3.3 million items/s
TapeServer ok ~ 25 Mb/s
Data forwarding to MBS ok
06:42 currently writing to file /media/SecondDrive/TapeData/S480/R12_315
07:25 hit-rate spectra - attachment 17
07:36 low energy channel spectra (1.8.L) - attachments 18 & 19
|
|