| ID |
Date |
Author |
Subject |
|
352
|
Sun May 30 08:57:53 2021 |
TD | Sunday 30 May | 09.55 DAQ continues - file S496/R18_588
alpha background
increased data rate due to hot HEC channels since c. 06.00 this am - ASIC check fixes problem
All system wide checks OK *except*
Base Current Difference
aida01 fault 0x82b0 : 0x82b7 : 7
aida02 fault 0xfe72 : 0xfe79 : 7
aida03 fault 0x97f4 : 0x97fb : 7
aida04 fault 0x950 : 0x957 : 7
aida05 fault 0x8e93 : 0x8e98 : 5
aida06 fault 0x545a : 0x545f : 5
aida07 fault 0xc04d : 0xc052 : 5
aida08 fault 0xa0f7 : 0xa0fc : 5
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 6 9 5 1 3 2 2 2 2 4 6 : 37008
aida02 : 3 5 5 2 2 4 3 3 2 4 6 : 37956
aida03 : 22 12 7 5 3 3 2 3 3 3 6 : 36872
aida04 : 40 13 5 5 3 4 2 3 3 3 6 : 37048
aida05 : 15 5 5 4 2 2 2 2 2 4 6 : 37044
aida06 : 11 6 2 1 2 3 3 4 2 4 6 : 38300
aida07 : 17 5 4 3 1 3 3 2 2 4 6 : 37324
aida08 : 1 8 1 2 2 4 4 3 2 4 6 : 38164
aida09 : 10 8 6 1 2 4 3 3 1 4 6 : 36968
aida10 : 14 7 5 2 1 3 3 3 1 4 6 : 36800
aida11 : 7 9 3 1 2 3 3 3 1 4 6 : 36788
aida12 : 3 5 2 3 1 4 2 3 2 4 6 : 37620
aida13 : 12 4 4 2 1 3 2 2 2 4 6 : 37008
aida14 : 11 10 5 1 1 3 2 2 2 4 6 : 37036
aida15 : 11 5 1 2 1 3 3 4 2 4 6 : 38244
aida16 : 10 4 8 1 2 3 2 2 2 4 6 : 37096
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.24 All histograms, statistics and merger statistics zero'd
16.16 analysis of file S496/R18_541 - attachment 9
max deadtime 0.02% aida03
18.40 DAQ continues - file S496/R18_589
alpha background
increased data rate due to hot HEC channels since c. 18.00 this am - ASIC check fixes problem |
|
351
|
Sat May 29 13:13:45 2021 |
TD | Saturday 29 May | 14.10 DAQ continues - file S496/R18_539
alpha background
increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem
All system wide checks OK *except*
Base Current Difference
aida01 fault 0x82b0 : 0x82b7 : 7
aida02 fault 0xfe72 : 0xfe79 : 7
aida03 fault 0x97f4 : 0x97fb : 7
aida04 fault 0x950 : 0x957 : 7
aida05 fault 0x8e93 : 0x8e97 : 4
aida06 fault 0x545a : 0x545e : 4
aida07 fault 0xc04d : 0xc051 : 4
aida08 fault 0xa0f7 : 0xa0fb : 4
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 8 7 2 2 2 2 2 2 4 6 : 37036
aida02 : 14 3 5 2 2 4 3 3 2 4 6 : 37984
aida03 : 15 11 8 6 3 3 2 3 3 3 6 : 36884
aida04 : 43 13 5 5 3 4 2 3 3 3 6 : 37060
aida05 : 19 3 5 4 2 2 2 2 2 4 6 : 37044
aida06 : 17 4 1 3 2 3 3 4 2 4 6 : 38356
aida07 : 16 6 2 1 0 2 2 3 2 4 6 : 37296
aida08 : 21 8 1 1 3 4 4 3 2 4 6 : 38276
aida09 : 13 10 2 1 2 3 2 2 2 4 6 : 37060
aida10 : 13 9 4 1 1 3 3 3 1 4 6 : 36764
aida11 : 13 9 2 1 2 3 3 3 1 4 6 : 36796
aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724
aida13 : 7 6 6 2 1 3 2 2 2 4 6 : 37036
aida14 : 9 7 7 1 1 3 2 2 2 4 6 : 37036
aida15 : 14 5 2 2 1 3 3 4 2 4 6 : 38272
aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.24 All histograms, statistics and merger statistics zero'd
16.16 analysis of file S496/R18_541 - attachment 9
max deadtime 0.12% aida04
18.40 DAQ continues - file S496/R18_551
alpha background
increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem |
|
350
|
Fri May 28 10:29:07 2021 |
TD | Friday 28 May | 11.30 DAQ continues - file S496/R18_472
alpha background
All system wide checks OK *except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b2 : 2
aida02 fault 0xfe72 : 0xfe74 : 2
aida03 fault 0x97f4 : 0x97f6 : 2
aida04 fault 0x950 : 0x952 : 2
aida05 fault 0x8e93 : 0x8e95 : 2
aida06 fault 0x545a : 0x545c : 2
aida07 fault 0xc04d : 0xc04f : 2
aida08 fault 0xa0f7 : 0xa0f9 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 4 7 1 3 2 2 2 2 4 6 : 36980
aida02 : 3 1 5 1 2 4 3 3 2 4 6 : 37892
aida03 : 8 8 6 6 3 3 2 3 3 3 6 : 36800
aida04 : 37 12 5 6 3 4 2 3 3 3 6 : 37060
aida05 : 11 5 6 4 2 2 2 2 2 4 6 : 37044
aida06 : 4 3 3 3 2 2 2 3 3 4 6 : 38456
aida07 : 1 12 3 2 0 3 3 2 2 4 6 : 37204
aida08 : 13 8 1 2 3 4 3 3 2 4 6 : 38020
aida09 : 7 7 5 1 1 3 2 2 2 4 6 : 36996
aida10 : 12 7 9 2 1 3 3 3 1 4 6 : 36856
aida11 : 2 9 3 1 2 3 3 3 1 4 6 : 36768
aida12 : 11 4 3 3 2 4 2 3 2 4 6 : 37724
aida13 : 10 8 6 2 2 3 2 2 2 4 6 : 37128
aida14 : 16 7 7 1 0 4 2 2 2 4 6 : 37128
aida15 : 12 6 2 2 1 3 3 4 2 4 6 : 38272
aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.30 analysis of file S496/R18_471 - attachment 9
max deadtime 0.18% aida04 |
|
349
|
Thu May 27 12:25:53 2021 |
TD | Thursday 27 May | 13.29 DAQ continues - file S496/R18_443
alpha background
All system wide checks OK *except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b2 : 2
aida02 fault 0xfe72 : 0xfe74 : 2
aida03 fault 0x97f4 : 0x97f6 : 2
aida04 fault 0x950 : 0x952 : 2
aida05 fault 0x8e93 : 0x8e95 : 2
aida06 fault 0x545a : 0x545c : 2
aida07 fault 0xc04d : 0xc04f : 2
aida08 fault 0xa0f7 : 0xa0f9 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 2 7 7 1 3 2 2 2 2 4 6 : 37008
aida02 : 3 1 5 1 3 4 3 3 2 4 6 : 37956
aida03 : 2 6 5 6 2 4 2 3 3 3 6 : 36808
aida04 : 24 12 5 5 4 3 2 3 3 3 6 : 36912
aida05 : 8 1 9 3 2 2 2 2 2 4 6 : 37016
aida06 : 3 6 2 2 2 2 2 3 3 4 6 : 38428
aida07 : 6 17 1 2 0 3 3 2 2 4 6 : 37232
aida08 : 4 2 1 2 3 4 3 3 2 4 6 : 37936
aida09 : 15 7 5 2 3 4 3 3 1 4 6 : 37060
aida10 : 7 6 9 2 0 3 3 3 1 4 6 : 36764
aida11 : 19 10 2 0 2 3 3 3 1 4 6 : 36796
aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724
aida13 : 17 9 2 0 0 4 2 2 2 4 6 : 37036
aida14 : 15 4 7 1 1 3 2 2 2 4 6 : 37036
aida15 : 9 6 1 2 1 3 3 4 2 4 6 : 38244
aida16 : 11 11 8 2 0 3 2 2 2 4 6 : 37060
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.30 analysis of file S496/R18_442 - attachment 9
max deadtime 0.28% aida04 |
|
348
|
Wed May 26 13:21:59 2021 |
TD | Wednesday 26 May |
14.22 DAQ continues - file S496/R18_369
alpha background
All system wide checks OK *except*
Base Current Difference
aida01 fault 0x82b0 : 0x82b1 : 1
aida02 fault 0xfe72 : 0xfe73 : 1
aida03 fault 0x97f4 : 0x97f5 : 1
aida04 fault 0x950 : 0x951 : 1
aida05 fault 0x8e93 : 0x8e94 : 1
aida06 fault 0x545a : 0x545b : 1
aida07 fault 0xc04d : 0xc04e : 1
aida08 fault 0xa0f7 : 0xa0f8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 12 4 6 1 2 3 2 2 2 4 6 : 37072
aida02 : 12 2 4 1 3 4 3 3 2 4 6 : 37984
aida03 : 12 6 7 6 2 4 3 2 3 3 6 : 36624
aida04 : 36 11 4 6 4 3 2 3 3 3 6 : 36968
aida05 : 2 4 6 3 2 2 2 2 2 4 6 : 36968
aida06 : 8 5 3 2 2 2 2 3 3 4 6 : 38456
aida07 : 10 4 6 4 3 3 3 3 2 4 6 : 37992
aida08 : 21 8 3 2 3 3 3 3 2 4 6 : 37956
aida09 : 17 10 3 2 3 4 3 3 1 4 6 : 37060
aida10 : 3 6 10 2 1 3 3 3 1 4 6 : 36828
aida11 : 12 6 4 2 2 3 3 3 1 4 6 : 36832
aida12 : 2 9 5 1 3 3 2 3 2 4 6 : 37632
aida13 : 4 7 2 1 3 3 3 3 2 4 6 : 37832
aida14 : 15 6 6 1 0 4 2 2 2 4 6 : 37100
aida15 : 5 4 3 2 1 3 3 4 2 4 6 : 38244
aida16 : 17 6 6 2 1 3 2 2 2 4 6 : 37076
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - attachments 3 & 4
ADC data item stats - attachment 5
FEE64 temps OK - attachment 6
DSSSD bias & leakage current OK - attachment 7
Merger/Tape Server/Merger stats - attachment 8
no merger errors reported since previous restart
14.30 analysis of file S496/R18_340 - attachment 9
max deadtime 0.31% aida04 |
|
347
|
Tue May 25 11:16:54 2021 |
TD | Tuesday 25 May | 12.16 DAQ continues file S496/R18_324
alpha background
Tape server c. 2.5Mb/s
ASIC check
Tape server c. 300kb/s - presumably hot HEC channels
12.20 All system wide checks OK *except*
Base Current Difference
aida01 fault 0x82b0 : 0x82b1 : 1
aida02 fault 0xfe72 : 0xfe73 : 1
aida03 fault 0x97f4 : 0x97f5 : 1
aida04 fault 0x950 : 0x951 : 1
aida05 fault 0x8e93 : 0x8e94 : 1
aida06 fault 0x545a : 0x545b : 1
aida07 fault 0xc04d : 0xc04e : 1
aida08 fault 0xa0f7 : 0xa0f8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 2 6 6 2 1 3 2 2 2 4 6 : 37016
aida02 : 3 6 5 3 3 4 3 2 1 3 7 : 38572
aida03 : 21 10 8 6 3 3 2 3 2 4 6 : 37924
aida04 : 29 15 4 5 3 4 2 3 3 3 6 : 37004
aida05 : 17 5 2 2 2 2 2 2 2 4 6 : 36940
aida06 : 1 6 0 1 2 2 2 3 3 4 6 : 38356
aida07 : 11 6 7 4 3 3 3 3 2 4 6 : 38028
aida08 : 13 8 3 2 2 3 3 2 1 3 7 : 38372
aida09 : 7 9 2 1 3 3 3 3 2 4 6 : 37860
aida10 : 9 8 4 2 0 3 3 3 1 4 6 : 36708
aida11 : 16 6 3 2 2 3 3 3 1 4 6 : 36832
aida12 : 21 10 7 4 2 4 2 2 1 3 7 : 38420
aida13 : 8 3 3 1 3 3 3 3 2 4 6 : 37832
aida14 : 14 3 6 1 0 4 2 2 2 4 6 : 37072
aida15 : 2 4 2 2 1 3 3 4 2 4 6 : 38216
aida16 : 16 5 5 3 0 3 2 2 2 4 6 : 37016
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
Rate spectra - attachment 2
ADC data item stats - attachment 3
FEE64 temps OK - attachment 4
DSSSD bias & leakage current OK - attachment 5
Merger/Tape Server/Merger stats - attachment 6
no merger errors reported since previous restart
17.30 analysis of file S496/R18_340
max deadtime 0.36% aida04 |
|
346
|
Mon May 24 09:57:07 2021 |
TD | Monday 24 May |
07.30 Hot HEC channels multiple FEE64s
ASIC check fixes
10.56 DAQ continues OK file S496/R18_75
System wide checks OK * except*
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Base Current Difference
aida01 fault 0x82b0 : 0x82b1 : 1
aida02 fault 0xfe72 : 0xfe73 : 1
aida03 fault 0x97f4 : 0x97f5 : 1
aida04 fault 0x950 : 0x951 : 1
aida05 fault 0x8e93 : 0x8e94 : 1
aida06 fault 0x545a : 0x545b : 1
aida07 fault 0xc04d : 0xc04e : 1
aida08 fault 0xa0f7 : 0xa0f8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 2 7 2 1 4 3 2 1 4 7 : 40452
aida02 : 13 3 3 1 3 4 2 2 2 4 7 : 41308
aida03 : 17 6 2 2 1 4 2 3 2 3 7 : 39700
aida04 : 8 6 1 0 1 3 1 4 2 3 7 : 39712
aida05 : 10 3 5 3 4 3 3 2 1 4 7 : 40560
aida06 : 4 4 2 0 2 3 2 3 3 4 7 : 42576
aida07 : 5 3 2 3 1 3 1 3 2 4 7 : 41324
aida08 : 11 5 1 1 2 3 3 2 2 4 7 : 41348
aida09 : 7 4 3 0 2 3 1 3 2 4 7 : 41324
aida10 : 11 4 2 2 2 3 2 4 1 4 7 : 41132
aida11 : 18 5 0 2 2 4 2 2 1 4 7 : 40240
aida12 : 2 3 0 1 2 3 2 2 2 4 7 : 41024
aida13 : 4 5 1 2 2 3 2 4 1 4 7 : 41096
aida14 : 9 3 1 2 1 4 2 2 2 4 7 : 41164
aida15 : 2 3 1 3 2 2 3 3 2 4 7 : 41744
aida16 : 5 5 2 0 3 3 2 4 1 4 7 : 41116
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
1.8.H spectra - attachments 6-8
1.8.L spectra - attachments 9 & 10
Stat spectra - attachments 11 & 12
1.8.W spectra - 20us FSR - attachments 13 & 14
ADC data items - attachment 15
FEE64 temperatures OK - attachment 16
DSSSD bias & leakage currents OK - attachment 17
Merger/Tape Server/Merger statistics - attachment 18
no merger errors reported since previous restart
analysis of file R18_74
max dead time 0.05%
10.17 ASIC check
All histograms, statistics, merger statistics zero'd |
|
345
|
Sat May 22 23:40:32 2021 |
TD | Sunday 23 May |
00.40 Unable to connect to FEE64s 1-7
DAQ power cycled OK
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 *all* FEE64s
BNC PB-5 OFF
file S496/R16
alpha background
07.17 Some hot HEC channels
ASIC check fixes issue
System wide checks OK *except*
Base Current Difference
aida06 fault 0x5457 : 0x5457 : 0
aida06 : WR status 0x10
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x1cd : 461
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 9 4 3 1 4 3 2 1 4 7 : 40552
aida02 : 3 2 2 1 2 3 3 2 2 4 7 : 41308
aida03 : 14 5 0 1 1 3 2 4 2 3 7 : 40000
aida04 : 14 4 5 1 3 3 1 4 2 3 7 : 39944
aida05 : 12 4 1 3 2 3 3 2 2 4 7 : 41408
aida06 : 8 3 2 1 2 4 2 3 3 4 7 : 42744
aida07 : 10 4 3 2 1 2 2 3 2 4 7 : 41464
aida08 : 7 5 2 0 3 3 3 2 2 4 7 : 41380
aida09 : 10 5 1 1 1 2 2 3 2 4 7 : 41408
aida10 : 11 6 3 2 1 3 2 2 2 4 7 : 41100
aida11 : 12 8 3 2 2 2 3 2 1 4 7 : 40288
aida12 : 9 9 3 1 2 3 3 2 1 4 7 : 40380
aida13 : 10 4 0 1 1 2 1 3 2 4 7 : 41128
aida14 : 9 4 2 1 1 2 1 3 2 4 7 : 41156
aida15 : 5 4 0 3 3 3 3 3 2 4 7 : 41940
aida16 : 9 6 3 2 0 2 1 3 2 4 7 : 41156
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 00:29:48 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE64 temperatures OK - attachment 1
ADC data item stats - attachment 2
Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3
DSSSD bias & leakage currents OK - attachment 4
11.18 Analysis of file S496/R16_33 - attachment 5
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 685M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.0T 867G 88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
867/2 x 3279s = 16.5 days to filesystem full - OK
12.11 aida15 zero data/poll stats - re-loaded DAQ program OK
aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK
DAQ continues file S496/R18 |
|
344
|
Sat May 22 13:40:56 2021 |
TD | Saturday 22 May |
14.37 DAQ continues file S496/R15_14
Grafana - DSSSD bias & leakage current - attachment 1
1.8.L spectra - attachments 2 & 3
Stat spectra - attachments 4 & 5
FEE64 temperatures OK - attachment 6
stats - ADC data items, pause items OK - attachments 7 & 8
DSSSD bias & leakage current OK - attachment 9
Merger/Tape Server/Merger stats - attachment 10
analysis of file S496/R14_1920 - attachment 11
all FEE64 deadtime 0% as expected
17.30 aida01, 3, 5, 9 & 11 have hot HEC channels
ASIC check fixes
All histogram, statistics & merger statistics zero'd
DAQ continues
Note gamma-ray source tests have started with FATIMA and DEGAS
20.20 Analysis file R15_19 - attachment 12
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 665M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.3T 530G 93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 90/2 x 3168s = 40h to filesystem full - OK
20.22 All system wide checks OK *except*
Base Current Difference
aida01 fault 0xf82c : 0xf82f : 3
aida02 fault 0xdda6 : 0xdda9 : 3
aida03 fault 0xb13d : 0xb140 : 3
aida04 fault 0xf4cd : 0xf4d0 : 3
aida05 fault 0x500 : 0x575 : 117
aida06 fault 0xc6e7 : 0xc6e9 : 2
aida07 fault 0x4920 : 0x4922 : 2
aida08 fault 0xbd9a : 0xbd9c : 2
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 8 4 6 6 1 4 3 3 4 3 6 : 38048
aida02 : 11 8 10 3 4 4 3 4 3 3 6 : 37740
aida03 : 2 8 3 3 2 2 1 4 2 4 6 : 37720
aida04 : 26 15 17 4 4 2 1 4 2 4 6 : 38256
aida05 : 15 6 4 1 1 2 1 3 3 4 6 : 38156
aida06 : 7 7 16 6 3 4 2 2 3 4 6 : 38612
aida07 : 11 5 12 3 3 3 2 3 1 3 7 : 38836
aida08 : 13 6 7 2 1 2 2 2 3 4 6 : 37972
aida09 : 23 3 8 5 0 5 3 3 4 3 6 : 38164
aida10 : 12 5 7 1 2 2 4 3 2 4 6 : 37992
aida11 : 19 2 7 1 2 2 1 2 3 4 6 : 37740
aida12 : 21 5 3 3 3 3 1 2 3 4 6 : 37964
aida13 : 18 7 7 1 2 4 2 3 2 4 6 : 37776
aida14 : 8 3 13 7 2 4 2 3 2 4 6 : 37992
aida15 : 11 5 14 1 3 4 2 3 2 4 6 : 37908
aida16 : 17 6 6 2 2 3 1 2 3 4 6 : 37908
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
19.30 Lost activity monitors - attachments 13-15
20.54 1.8.W spectra 20us FSR attachments 16 & 23
1.8.W spectra 200us FSR attachments 17 & 22
1.8.W spectra 2ms FSR attachments 18 & 21
1.8.W spectra 20ms FSR attachments 19 & 20 |
|
343
|
Fri May 21 23:19:23 2021 |
TD | Saturday 22 May 00:00-08:00 |
00.16 413Gb remaining /TapeData filesystem
estimate 413/2 x 177s = 10.15h to filesystem full - OK
00.18 All histograms, statistics and merger statistics zero'd
00.57 All system wide checks OK *except*
Base Current Difference
aida05 fault 0x500 : 0x572 : 114
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 9 3 4 1 5 2 2 3 3 6 : 36356
aida02 : 4 8 7 2 4 3 3 3 2 3 6 : 35968
aida03 : 12 9 6 2 5 3 2 3 3 3 6 : 36824
aida04 : 11 5 9 2 2 2 2 4 3 3 6 : 37028
aida05 : 24 9 2 1 2 3 2 3 4 3 6 : 37608
aida06 : 25 6 3 4 2 4 3 3 3 3 6 : 37060
aida07 : 3 3 3 1 3 3 2 2 2 2 7 : 37044
aida08 : 20 4 2 3 1 3 2 2 3 3 6 : 36016
aida09 : 15 8 2 5 0 3 3 2 3 3 6 : 36284
aida10 : 16 6 4 1 2 3 3 2 2 4 6 : 37328
aida11 : 20 2 2 1 1 3 2 2 3 3 6 : 35936
aida12 : 1 2 1 2 3 4 1 2 4 3 6 : 36900
aida13 : 25 10 4 1 0 3 2 3 3 3 6 : 36500
aida14 : 20 6 9 2 2 5 1 2 2 4 6 : 37200
aida15 : 31 9 5 2 3 3 1 3 3 3 6 : 36500
aida16 : 24 4 2 3 1 4 2 3 3 3 6 : 36672
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
01.12 Lost activity monitor - attachment 1
1.8.H spectra - attachments 2 & 3
1.8.L spectra - attachments 4-6
aida01 pulser peak width 120 ch FWHM
Rate specttra - attachments 7 & 8
01.06 Lost activity monitor - attachment 9
ucesb - attachment 10
Grafana - DSSSD bias & leakage currents past 7 days - attachment 11
statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17
FEE64 temperatures OK - attachment 18
DSSSD bias & leakage currents OK - attachment 19
Merger/Tape Server/Merger stats - attachment 20
no merger warning/error messages since last restart
01.26 Analysis file S496/R14_1798 - attachment 21
aida08 5.5% deadtime
01.17 Disk space remaining /TapeData filesystem
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 706M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.4T 395G 95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 395/2 x 177s = 9.7h to filesystem full - OK
01.30 despec shift reports no data reported by ucesb
aida stats & tape server ok - no data being forwarded to MBS by data relay
01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811
04.35 System wide checks all OK *except*
Base Current Difference
aida05 fault 0x500 : 0x573 : 115
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 661M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.5T 337G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 337/2 x 177s = 8.3h to filesystem full - OK
FEE64 temperatures OK - attachment 22
ADC data item statistics OK - attachment 24
DSSSD bias & leakage current OK - attachment 26
analysis of file S496/R14_1920 - attachment 28
aida04 deadtime 5.0%
07.10 Ssystem wide checks all OK *except*
Base Current Difference
aida05 fault 0x500 : 0x573 : 115
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
[npg@aidas-gsi S496]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
50G 18G 30G 37% /
tmpfs 7.9G 662M 7.2G 9% /dev/shm
/dev/sda2 477M 42M 410M 10% /boot
/dev/sda1 200M 264K 200M 1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
401G 90G 292G 24% /home
/dev/mapper/vg_aidas2-lv_home
7.2T 6.5T 287G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive
/dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive
estimate 287/2 x 178s = 7.1h to filesystem full - OK
FEE64 temperatures OK - attachment 23
ADC data item statistics OK - attachment 25
DSSSD bias & leakage current OK - attachment 27
analysis of file S496/R14_1920 - attachment 29
aida04 deadtime 4.4%
07.35 Found MBS data monitor kill'd
Restarted MBS data relay and data monitor OK
08.00 beam off - end of S496
08.28 Tape Server -> no storage mode
DAQ continues
09.58 All slow comparators -> 0x64
BNC PB-5 OFF
file S496/R15
alpha background
Tape server 200kb/s, merger 260k data items/s |
|
342
|
Fri May 21 15:03:48 2021 |
CA | Friday 21st May 16:00 - 00:00 shift | 16:00 CA takes over
DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4
PULSER SETTINGS
---------------------------
Pulse is ON
Positive Tail Pulse
Trigger Source is Internal Clock
Trigger Threshold is 3.5
Amplitude : 2.0 Volts
Rep Rate : 2.0 hZ
Delay : 250.0 ns
Fall Time : 1 ms
Attenuation : 1
Display is : Volts
Equivalent keV is : 200.0
Ramp Start at 0.01 Volts
Ramp Stop at 9.99 Volts
Ramp Start at 1.0 keV
Ramp Stop at 999.0 keV
Ramp Time is 60 seconds
# Ramp Cycles is 1
16:14 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x570 : 112
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
16:17 temperatures ok - attachment 1
statistics ok - attachment 2
detector bias / leakage currents ok - attachment 3
analysis of R14_1625 - FEE7 deadtime at 4.5%
18:06 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x571 : 113
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
18:07 temperatures ok - attachment 4
statistics ok - attachment 5
detector bias / leakage currents ok - attachment 6
analysis of R14_1662 - FEE7 deadtime at 4.3%
19:07 DAQ continues ok - writing to file R14_1682
20:07 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x572 : 114
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
20:10 temperatures ok - attachment 7
statistics ok - attachment 8
detector bias / leakage currents ok - attachment 9
analysis of R14_1703 - max deadtime at 4.0% - attachment 10
20:35 rate spectra - attachment 11
HEC spectra - attachment 12 & 13
20:36 all histograms zeroed
20:37 DAQ continues ok - writing to file R14_1712
20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.
~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files
21:12 no beam - writing to file R14_1723
21:17 beam is back
22:07 system wide checks all ok - WR decoder fault same as entry at 20:07
22:08 temperatures ok - attachment 14
statistics ok - attachment 15
detector bias / leakage currents ok - attachment 16
analysis of R14_1740 - max deadtime at 4.2% - attachment 17
|
|
341
|
Fri May 21 07:03:50 2021 |
OH, TD | Friday 21 May 08:00-16:00 | 08:00 OH and TD take over
Current settings p+n sides at 0xc
DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4
PULSER SETTINGS
---------------------------
Pulse is ON
Positive Tail Pulse
Trigger Source is Internal Clock
Trigger Threshold is 3.5
Amplitude : 2.0 Volts
Rep Rate : 2.0 hZ
Delay : 250.0 ns
Fall Time : 1 ms
Attenuation : 1
Display is : Volts
Equivalent keV is : 200.0
Ramp Start at 0.01 Volts
Ramp Stop at 9.99 Volts
Ramp Start at 1.0 keV
Ramp Stop at 999.0 keV
Ramp Time is 60 seconds
# Ramp Cycles is 1
08:29 System wide checks ok - WR difference at 111
Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
Temperature ok - attachment 2
Bias and leakage current ok - attachment 3
09:58 System wide checks ok - WR difference at 111
Statistics ok - attachment 4
Temperature ok - attachment 5
Bias and leakage current ok - attachment 6
Max deadtime this morning has been around 5.5% on FEE8
12:32 System wide checks ok - WR difference at 111
Statistics ok - attachment 7
Temperature ok - attachment 9
Bias and leakage current ok - attachment 8 |
|
340
|
Fri May 21 06:09:28 2021 |
ML | system wide checks | System wide checks at 7am CET
All seems fine, nothing significant to report
Slight increase of dead time noted in FEE 3, 5 and 7.
-----
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
-----
ADC calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
-----
White Rabbit:
Base Current Difference
aida05 fault 0x500 : 0x56e : 110
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
-----
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
-----
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 5 6 5 0 4 1 3 3 3 6 : 36484
aida02 : 10 3 3 2 3 2 2 2 3 3 6 : 35952
aida03 : 2 5 1 4 5 2 2 3 3 3 6 : 36608
aida04 : 1 3 4 0 4 3 2 3 3 3 6 : 36572
aida05 : 19 9 3 2 2 3 1 3 4 3 6 : 37380
aida06 : 13 11 1 5 3 4 4 2 3 3 6 : 36860
aida07 : 12 2 4 3 3 2 1 3 2 2 7 : 37280
aida08 : 25 5 7 1 1 3 3 3 3 3 6 : 36828
aida09 : 22 5 6 4 1 3 2 3 3 3 6 : 36640
aida10 : 23 8 6 3 2 3 2 2 2 4 6 : 37212
aida11 : 24 14 7 3 3 2 3 3 3 3 6 : 36960
aida12 : 16 10 5 3 1 2 1 2 4 3 6 : 36736
aida13 : 23 6 4 2 2 3 2 3 3 3 6 : 36620
aida14 : 22 7 8 4 3 4 1 2 2 4 6 : 37200
aida15 : 10 12 9 2 1 3 1 3 3 3 6 : 36376
aida16 : 18 5 6 2 0 3 2 4 3 3 6 : 37008
|
|
339
|
Fri May 21 02:17:47 2021 |
ML | system wide checks | Wide check at 3am -
All fine.
Same output as at the beginning of the shift (see my previous entry) and attachments.
|
|
338
|
Thu May 20 23:26:11 2021 |
Marc Labiche (ML) | system wide checks | System Wide check at begining of the night shift:
-----
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
-----
ADC calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
-----
White rabbit status:
Base Current Difference
aida05 fault 0x500 : 0x56d : 109
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
-----
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
-----
Memory info from FEE cards
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 7 7 6 1 5 2 2 3 3 6 : 36472
aida02 : 19 5 4 4 4 3 3 3 2 3 6 : 36020
aida03 : 22 10 2 2 5 2 2 3 3 3 6 : 36680
aida04 : 12 7 1 0 4 3 1 4 3 3 6 : 36856
aida05 : 16 7 5 3 1 3 2 3 3 3 6 : 36584
aida06 : 20 7 5 6 2 4 4 2 3 3 6 : 36888
aida07 : 13 7 5 2 3 2 1 3 2 2 7 : 37308
aida08 : 18 7 5 3 0 2 3 3 3 3 6 : 36656
aida09 : 21 6 4 4 1 4 2 3 3 3 6 : 36740
aida10 : 21 10 2 3 2 3 2 2 2 4 6 : 37156
aida11 : 23 2 4 4 3 2 2 3 2 4 6 : 37612
aida12 : 24 1 4 3 2 2 1 2 4 3 6 : 36744
aida13 : 26 9 4 2 1 4 3 2 3 3 6 : 36464
aida14 : 19 9 9 2 3 4 1 2 2 4 6 : 37156
aida15 : 10 7 10 0 1 3 1 3 3 3 6 : 36288
aida16 : 28 7 3 2 1 3 3 3 3 3 6 : 36824 |
|
337
|
Thu May 20 14:56:33 2021 |
OH | Thursday 20th May 16:00 - 24:00 | 16:00 p+n threshold 0xc (120keV)
n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4
n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4
Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure.
17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector
18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now.
The deadtimes in FEE6 and 8 have risen to around 6.25%.
Looking at this mornings and yesterday evenings fill times this behavior isn't observed
18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago
18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2
This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file)
18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8
18:15 System wide checks ok except WR as usual
Base Current Difference
aida05 fault 0x500 : 0x56d : 109
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics increased at 18:02 still working on - attachment 2
FEE temp - attachment 3
Bias and leakage current - attachment 4
20:51 System wide checks ok - WR same as before
Statistics still high - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
22:52 Statistics still high - attachment 8
Temp ok - attachment 9
Bias and leakage current ok - attachment 10
System wide checks ok - WR difference still 109 |
|
336
|
Thu May 20 07:03:02 2021 |
CA | Thursday May 20th 08:00 - 16:00 shift | 08:00 CA takes over
08:04 DAQ continues ok - writing to file R14_989
09:30 all system wide checks ok *except* WR decoder status;
Base Current Difference
aida05 fault 0x500 : 0x56a : 106
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
09:34 temperatures ok - attachment 1
stats ok - attachment 2
bias/leakage currents ok - attachment 3
R14_1016 analysis - max deadtime ~4.3% - attachment 4
11:30 all system wide checks ok *except* WR decoder status;
Base Current Difference
aida05 fault 0x500 : 0x56b : 107
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
11:34 temperatures ok - attachment 5
stats ok - attachment 6
bias/leakage currents ok - attachment 7
R14_1052 analysis - max deadtime ~3.6% - attachment 8
13:05 beam optimisation currently taking place
DAQ continues ok and writing to file R14_1085
13:17 no beam
13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30.
13:31 temperatures ok - attachment 9
stats ok - attachment 10
bias/leakage currents ok - attachment 11
R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12
13:37 beam is back - rate spectra - attachment 13
DAQ ok - writing to file R14_1088
13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14
14:21 beam off - issue with UNILAC
DESPEC also report issues with their DAQ
AIDA continues to forward data to MBS ok - lower rates while beam off
14.30 1.8.H spectra - attachments 16-18
1.8.L spectra - attachments 19-22
aida01 pulser peak width 99 ch FWHM
Rate spectra - attachment 23
Grafana - DSSSD bias & leakage currents past 7 days - attachment 24
DSSSD bias & leakage currents - attachment 25
Merger/Tape Server/Merger stats - attachment 26
14.41 All histograms & stats zero'd
14.55 Lost Activity Monitors - attachments 27, 28 & 35
Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler
15:00 beam is back
15:05 AIDA writing to file R14_1108
ADC data items - attachment 36
15:27 system wide checks ok *except* WR decoder status
Base Current Difference
aida05 fault 0x500 : 0x56c : 108
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
15:30 temperatures ok - attachment 37
stats ok - attachment 38
bias/leakage currents ok - attachment 39
R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40 |
|
335
|
Thu May 20 01:00:30 2021 |
Muneerah | Thu May 20 00:00-08:00 | 02:00 System check
The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!
The rates on ucesb got back to work by around 01:12
Deat Time check ok R14_870, attachment 1
Statistics are ok attachment 2
Temperatures are ok attacment 3
Voltages are ok attachment 4
system wide check
All ok except
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbite
1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 8 8 5 0 6 1 2 2 2 7 : 37296
aida02 : 19 8 11 2 3 4 2 2 3 3 6 : 36412
aida03 : 14 6 3 4 1 3 1 3 3 3 6 : 36312
aida04 : 25 12 12 4 3 4 1 4 3 3 6 : 37316
aida05 : 4 8 3 4 1 2 2 4 3 3 6 : 36928
aida06 : 42 12 8 7 2 3 3 4 3 3 6 : 37736
aida07 : 24 6 7 3 3 3 1 3 2 2 7 : 37536
aida08 : 10 15 7 2 1 4 2 3 3 3 6 : 36752
aida09 : 29 13 6 6 4 3 3 3 3 3 6 : 37244
aida10 : 18 12 7 2 2 3 3 2 2 4 6 : 37464
aida11 : 20 8 0 5 5 4 2 3 3 3 6 : 36976
aida12 : 7 2 4 3 2 3 2 2 4 3 6 : 37068
aida13 : 2 5 5 4 2 3 2 3 3 3 6 : 36608
aida14 : 9 13 8 2 2 2 3 3 2 4 6 : 37836
aida15 : 24 12 12 1 0 3 2 3 3 3 6 : 36640
aida16 : 12 9 9 2 1 3 2 4 3 3 6 : 37128
6:45 System check
Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9
System wide check are all ok except:
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit 1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 11 3 5 0 6 1 2 3 2 7 : 38264
aida02 : 14 9 9 2 3 3 3 2 3 3 6 : 36496
aida03 : 13 7 3 5 2 3 2 3 3 3 6 : 36668
aida04 : 30 15 13 4 3 4 1 3 4 3 6 : 37888
aida05 : 17 9 3 3 1 2 2 4 3 3 6 : 36956
aida06 : 40 16 5 6 2 3 2 3 2 4 6 : 37936
aida07 : 3 2 3 2 3 4 2 2 2 2 7 : 37196
aida08 : 17 10 6 2 1 3 3 3 3 3 6 : 36852
aida09 : 29 11 7 6 3 3 2 4 3 3 6 : 37436
aida10 : 1 10 5 1 1 3 3 2 2 4 6 : 37252
aida11 : 30 9 0 5 4 4 2 3 3 3 6 : 36960
aida12 : 19 10 5 3 2 3 2 2 4 3 6 : 37196
aida13 : 22 9 6 3 2 3 3 3 3 3 6 : 36960
aida14 : 19 6 9 2 2 3 3 2 2 4 6 : 37452
aida15 : 6 8 11 2 1 3 2 3 3 3 6 : 36616
aida16 : 19 14 5 3 1 3 1 4 3 3 6 : 36908
07:33 system check
Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13
System wide check
all ok except
ADC all failed
White rabbit
1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 9 4 6 0 5 1 2 2 2 7 : 37152
aida02 : 19 6 13 1 3 3 4 2 3 3 6 : 36780
aida03 : 23 5 6 4 2 3 1 3 3 3 6 : 36452
aida04 : 24 11 14 3 3 4 2 3 3 3 6 : 37048
aida05 : 17 3 6 3 1 2 2 3 3 3 6 : 36444
aida06 : 36 15 4 9 2 4 2 4 3 3 6 : 37608
aida07 : 20 8 5 3 3 4 2 2 2 2 7 : 37376
aida08 : 21 11 10 2 1 4 2 2 2 4 6 : 37324
aida09 : 27 14 7 6 3 3 2 4 3 3 6 : 37452
aida10 : 25 9 5 2 2 3 3 2 2 4 6 : 37436
aida11 : 30 14 2 4 5 3 2 3 3 3 6 : 36936
aida12 : 22 3 5 3 2 3 2 3 3 3 6 : 36640
aida13 : 10 7 1 4 2 3 3 2 3 3 6 : 36336
aida14 : 13 9 12 2 3 3 2 3 2 4 6 : 37820
aida15 : 20 8 11 3 0 3 2 3 3 3 6 : 36640
aida16 : 23 6 6 2 0 2 1 3 4 3 6 : 37164
|
|
334
|
Wed May 19 20:02:27 2021 |
NH | AIDA Dead Time Analysis | Some plots of AIDA dead time (From PAUSE/RESUME data items) against time.
The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead.
Beam spills are indicated by the red bars (1 = on spill)
Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown
Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates .
The much quieter p+n strips show a similar pattern but much less frequently.
Deadtime does not always sync up between FEE64s it seems.
IN all case the pause/resume interval seems to only last for less than 100 ms. |
|
333
|
Wed May 19 18:50:01 2021 |
LS | Wednesday 19th May 20:00 to 24:00 | 20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)
20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)
21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)
System wide checks okay except white rabbit 1 fail:
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~2.8M items/s
Tape server ~10MB/s
Analysed R14_781 deadtimes are okay (Fig 9)
21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)
22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)
22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)
23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)
System wide checks okay except white rabbit 1 fail:
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~3M items/s
Tape server ~10MB/s
Analysed R14_819 deadtimes are okay (Fig 19)
23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21) |
|