ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 19 of 36

Full | Summary | Threaded | Hide attachments

703 Entries

Goto page Previous 1, 2, 3 ... 18, 19, 20 ... 34, 35, 36 Next

Selected entries:

	ID	Date	Author	Subject
	352	Sun May 30 08:57:53 2021	TD	Sunday 30 May
09.55 DAQ continues - file S496/R18_588 alpha background increased data rate due to hot HEC channels since c. 06.00 this am - ASIC check fixes problem All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b7 : 7 aida02 fault 0xfe72 : 0xfe79 : 7 aida03 fault 0x97f4 : 0x97fb : 7 aida04 fault 0x950 : 0x957 : 7 aida05 fault 0x8e93 : 0x8e98 : 5 aida06 fault 0x545a : 0x545f : 5 aida07 fault 0xc04d : 0xc052 : 5 aida08 fault 0xa0f7 : 0xa0fc : 5 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 6 9 5 1 3 2 2 2 2 4 6 : 37008 aida02 : 3 5 5 2 2 4 3 3 2 4 6 : 37956 aida03 : 22 12 7 5 3 3 2 3 3 3 6 : 36872 aida04 : 40 13 5 5 3 4 2 3 3 3 6 : 37048 aida05 : 15 5 5 4 2 2 2 2 2 4 6 : 37044 aida06 : 11 6 2 1 2 3 3 4 2 4 6 : 38300 aida07 : 17 5 4 3 1 3 3 2 2 4 6 : 37324 aida08 : 1 8 1 2 2 4 4 3 2 4 6 : 38164 aida09 : 10 8 6 1 2 4 3 3 1 4 6 : 36968 aida10 : 14 7 5 2 1 3 3 3 1 4 6 : 36800 aida11 : 7 9 3 1 2 3 3 3 1 4 6 : 36788 aida12 : 3 5 2 3 1 4 2 3 2 4 6 : 37620 aida13 : 12 4 4 2 1 3 2 2 2 4 6 : 37008 aida14 : 11 10 5 1 1 3 2 2 2 4 6 : 37036 aida15 : 11 5 1 2 1 3 3 4 2 4 6 : 38244 aida16 : 10 4 8 1 2 3 2 2 2 4 6 : 37096 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.24 All histograms, statistics and merger statistics zero'd 16.16 analysis of file S496/R18_541 - attachment 9 max deadtime 0.02% aida03 18.40 DAQ continues - file S496/R18_589 alpha background increased data rate due to hot HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-30_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-30_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-30_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-30_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 960.png

Attachment 8: 961.png

Attachment 9: R18_589
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 31985 ADC data format: 58630511 ( 25364.6 Hz) Other data format: 203166719 ( 87893.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 35 ( 0.0 Hz) RESUME: 35 ( 0.0 Hz) SYNC100: 40596 ( 17.6 Hz) WR48-63: 40596 ( 17.6 Hz) FEE64 disc: 1341777 ( 580.5 Hz) MBS info: 201743680 ( 87277.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2311.511 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 1879.226 2 0.365 0.000 3 0.313 0.000 4 0.000 1697.189 5 0.000 557.330 6 0.000 1686.560 7 0.000 647.171 8 0.000 0.000 9 0.000 0.000 10 0.382 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 157 0 0 0 0 0 0 0 0 0 0 0 1 3314609 1296 0 0 0 0 648 648 0 0 0 0 2 413 87031526 0 0 19 19 14856 14856 0 87001776 0 0 3 42258127 88383449 0 0 15 15 18954 18954 1341775 87003736 0 0 4 153 6936262 0 0 0 0 860 860 0 6934542 0 0 5 6312304 6939162 0 0 0 0 2310 2310 0 6934542 0 0 6 544 6936170 0 0 0 0 814 814 0 6934542 0 0 7 6741331 6938850 0 0 0 0 2154 2154 0 6934542 0 0 8 448 0 0 0 0 0 0 0 0 0 0 0 9 79 0 0 0 0 0 0 0 0 0 0 0 10 1189 2 0 0 1 1 0 0 0 0 0 0 11 111 2 0 0 0 0 0 0 2 0 0 0 12 767 0 0 0 0 0 0 0 0 0 0 0 13 99 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 180 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.504s ( 1015.271 blocks/s, 63.454 Mb/s)
	351	Sat May 29 13:13:45 2021	TD	Saturday 29 May
14.10 DAQ continues - file S496/R18_539 alpha background increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b7 : 7 aida02 fault 0xfe72 : 0xfe79 : 7 aida03 fault 0x97f4 : 0x97fb : 7 aida04 fault 0x950 : 0x957 : 7 aida05 fault 0x8e93 : 0x8e97 : 4 aida06 fault 0x545a : 0x545e : 4 aida07 fault 0xc04d : 0xc051 : 4 aida08 fault 0xa0f7 : 0xa0fb : 4 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 8 7 2 2 2 2 2 2 4 6 : 37036 aida02 : 14 3 5 2 2 4 3 3 2 4 6 : 37984 aida03 : 15 11 8 6 3 3 2 3 3 3 6 : 36884 aida04 : 43 13 5 5 3 4 2 3 3 3 6 : 37060 aida05 : 19 3 5 4 2 2 2 2 2 4 6 : 37044 aida06 : 17 4 1 3 2 3 3 4 2 4 6 : 38356 aida07 : 16 6 2 1 0 2 2 3 2 4 6 : 37296 aida08 : 21 8 1 1 3 4 4 3 2 4 6 : 38276 aida09 : 13 10 2 1 2 3 2 2 2 4 6 : 37060 aida10 : 13 9 4 1 1 3 3 3 1 4 6 : 36764 aida11 : 13 9 2 1 2 3 3 3 1 4 6 : 36796 aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724 aida13 : 7 6 6 2 1 3 2 2 2 4 6 : 37036 aida14 : 9 7 7 1 1 3 2 2 2 4 6 : 37036 aida15 : 14 5 2 2 1 3 3 4 2 4 6 : 38272 aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.24 All histograms, statistics and merger statistics zero'd 16.16 analysis of file S496/R18_541 - attachment 9 max deadtime 0.12% aida04 18.40 DAQ continues - file S496/R18_551 alpha background increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-29_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-29_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-29_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-29_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 950.png

Attachment 8: 951.png

Attachment 9: R18_541
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 46792076 ( 18702.9 Hz) Other data format: 215127929 ( 85987.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 89 ( 0.0 Hz) RESUME: 89 ( 0.0 Hz) SYNC100: 41318 ( 16.5 Hz) WR48-63: 41318 ( 16.5 Hz) FEE64 disc: 1013016 ( 404.9 Hz) MBS info: 214032099 ( 85549.1 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2501.862 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 1964.282 2 0.467 0.000 3 3.050 0.000 4 0.000 1832.921 5 0.000 844.521 6 0.000 1829.325 7 0.000 869.373 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 499 0 0 0 0 0 0 0 0 0 0 0 1 74891 24 0 0 0 0 12 12 0 0 0 0 2 1032 92085039 0 0 27 27 16251 16251 0 92052483 0 0 3 37271457 93008741 0 0 62 62 19135 19135 1013015 91957332 0 0 4 552 7507421 0 0 0 0 925 925 0 7505571 0 0 5 4051453 7509613 0 0 0 0 2021 2021 0 7505571 0 0 6 1401 7507471 0 0 0 0 950 950 0 7505571 0 0 7 5387708 7509619 0 0 0 0 2024 2024 0 7505571 0 0 8 460 0 0 0 0 0 0 0 0 0 0 0 9 183 0 0 0 0 0 0 0 0 0 0 0 10 537 0 0 0 0 0 0 0 0 0 0 0 11 388 0 0 0 0 0 0 0 0 0 0 0 12 674 0 0 0 0 0 0 0 0 0 0 0 13 244 1 0 0 0 0 0 0 1 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 597 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.855s ( 1004.537 blocks/s, 62.784 Mb/s)
	350	Fri May 28 10:29:07 2021	TD	Friday 28 May
11.30 DAQ continues - file S496/R18_472 alpha background All system wide checks OK except FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Base Current Difference aida01 fault 0x82b0 : 0x82b2 : 2 aida02 fault 0xfe72 : 0xfe74 : 2 aida03 fault 0x97f4 : 0x97f6 : 2 aida04 fault 0x950 : 0x952 : 2 aida05 fault 0x8e93 : 0x8e95 : 2 aida06 fault 0x545a : 0x545c : 2 aida07 fault 0xc04d : 0xc04f : 2 aida08 fault 0xa0f7 : 0xa0f9 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 4 7 1 3 2 2 2 2 4 6 : 36980 aida02 : 3 1 5 1 2 4 3 3 2 4 6 : 37892 aida03 : 8 8 6 6 3 3 2 3 3 3 6 : 36800 aida04 : 37 12 5 6 3 4 2 3 3 3 6 : 37060 aida05 : 11 5 6 4 2 2 2 2 2 4 6 : 37044 aida06 : 4 3 3 3 2 2 2 3 3 4 6 : 38456 aida07 : 1 12 3 2 0 3 3 2 2 4 6 : 37204 aida08 : 13 8 1 2 3 4 3 3 2 4 6 : 38020 aida09 : 7 7 5 1 1 3 2 2 2 4 6 : 36996 aida10 : 12 7 9 2 1 3 3 3 1 4 6 : 36856 aida11 : 2 9 3 1 2 3 3 3 1 4 6 : 36768 aida12 : 11 4 3 3 2 4 2 3 2 4 6 : 37724 aida13 : 10 8 6 2 2 3 2 2 2 4 6 : 37128 aida14 : 16 7 7 1 0 4 2 2 2 4 6 : 37128 aida15 : 12 6 2 2 1 3 3 4 2 4 6 : 38272 aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.30 analysis of file S496/R18_471 - attachment 9 max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-28_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 940.png

Attachment 8: 941.png

Attachment 9: R18_471
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 57407666 ( 20753.5 Hz) Other data format: 204512336 ( 73933.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 116 ( 0.0 Hz) RESUME: 116 ( 0.0 Hz) SYNC100: 42303 ( 15.3 Hz) WR48-63: 42303 ( 15.3 Hz) FEE64 disc: 1160567 ( 419.6 Hz) MBS info: 203266931 ( 73483.1 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 109 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2766.172 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2700.364 2 0.332 0.000 3 4.856 0.000 4 0.000 0.000 5 0.000 1685.309 6 0.000 0.000 7 0.000 1567.327 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 181 23 0 0 0 0 0 0 23 0 0 13 1 838470 386 0 0 0 0 166 166 54 0 0 0 2 162 101753181 0 0 19 19 17931 17931 15 101717266 0 5 3 42392398 102752755 0 0 97 97 21331 21331 1160234 101549665 0 12 4 211 21 0 0 0 0 0 0 21 0 0 10 5 6478076 2740 0 0 0 0 1337 1337 66 0 0 0 6 260 17 0 0 0 0 0 0 17 0 0 4 7 7695649 3163 0 0 0 0 1537 1537 89 0 0 28 8 132 14 0 0 0 0 1 1 12 0 0 10 9 368 5 0 0 0 0 0 0 5 0 0 4 10 173 18 0 0 0 0 0 0 18 0 0 5 11 303 5 0 0 0 0 0 0 5 0 0 5 12 186 1 0 0 0 0 0 0 1 0 0 7 13 585 2 0 0 0 0 0 0 2 0 0 2 14 0 0 0 0 0 0 0 0 0 0 0 0 15 512 5 0 0 0 0 0 0 5 0 0 4 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.742s ( 1008.122 blocks/s, 63.008 Mb/s)
	349	Thu May 27 12:25:53 2021	TD	Thursday 27 May
13.29 DAQ continues - file S496/R18_443 alpha background All system wide checks OK except FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Base Current Difference aida01 fault 0x82b0 : 0x82b2 : 2 aida02 fault 0xfe72 : 0xfe74 : 2 aida03 fault 0x97f4 : 0x97f6 : 2 aida04 fault 0x950 : 0x952 : 2 aida05 fault 0x8e93 : 0x8e95 : 2 aida06 fault 0x545a : 0x545c : 2 aida07 fault 0xc04d : 0xc04f : 2 aida08 fault 0xa0f7 : 0xa0f9 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 7 7 1 3 2 2 2 2 4 6 : 37008 aida02 : 3 1 5 1 3 4 3 3 2 4 6 : 37956 aida03 : 2 6 5 6 2 4 2 3 3 3 6 : 36808 aida04 : 24 12 5 5 4 3 2 3 3 3 6 : 36912 aida05 : 8 1 9 3 2 2 2 2 2 4 6 : 37016 aida06 : 3 6 2 2 2 2 2 3 3 4 6 : 38428 aida07 : 6 17 1 2 0 3 3 2 2 4 6 : 37232 aida08 : 4 2 1 2 3 4 3 3 2 4 6 : 37936 aida09 : 15 7 5 2 3 4 3 3 1 4 6 : 37060 aida10 : 7 6 9 2 0 3 3 3 1 4 6 : 36764 aida11 : 19 10 2 0 2 3 3 3 1 4 6 : 36796 aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724 aida13 : 17 9 2 0 0 4 2 2 2 4 6 : 37036 aida14 : 15 4 7 1 1 3 2 2 2 4 6 : 37036 aida15 : 9 6 1 2 1 3 3 4 2 4 6 : 38244 aida16 : 11 11 8 2 0 3 2 2 2 4 6 : 37060 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.30 analysis of file S496/R18_442 - attachment 9 max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-05-27_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-27_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-27_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-27_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 930.png

Attachment 8: 931.png

Attachment 9: R18_442
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 68693190 ( 21525.3 Hz) Other data format: 193226813 ( 60548.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 114 ( 0.0 Hz) RESUME: 114 ( 0.0 Hz) SYNC100: 43889 ( 13.8 Hz) WR48-63: 43889 ( 13.8 Hz) FEE64 disc: 1335474 ( 418.5 Hz) MBS info: 191803333 ( 60102.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 3 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 3191.275 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2989.890 2 0.511 643.077 3 8.760 0.000 4 0.000 0.000 5 0.000 1578.752 6 0.000 0.000 7 0.000 1340.150 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 155 0 0 0 0 0 0 0 0 0 0 0 1 527962 216 0 0 0 0 108 108 0 0 0 0 2 533 96091128 0 0 28 28 16832 16832 0 96057408 0 0 3 49735236 97127126 0 0 86 86 22778 22778 1335473 95745925 0 0 4 118 2 0 0 0 0 1 1 0 0 0 0 5 8268444 3874 0 0 0 0 1937 1937 0 0 0 0 6 632 0 0 0 0 0 0 0 0 0 0 0 7 10156762 4466 0 0 0 0 2233 2233 0 0 0 0 8 500 0 0 0 0 0 0 0 0 0 0 0 9 102 0 0 0 0 0 0 0 0 0 0 0 10 1498 0 0 0 0 0 0 0 0 0 0 0 11 105 0 0 0 0 0 0 0 0 0 0 0 12 886 0 0 0 0 0 0 0 0 0 0 0 13 95 1 0 0 0 0 0 0 1 0 0 2 14 0 0 0 0 0 0 0 0 0 0 0 0 15 162 0 0 0 0 0 0 0 0 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 32.027s ( 999.146 blocks/s, 62.447 Mb/s)
	348	Wed May 26 13:21:59 2021	TD	Wednesday 26 May
14.22 DAQ continues - file S496/R18_369 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b1 : 1 aida02 fault 0xfe72 : 0xfe73 : 1 aida03 fault 0x97f4 : 0x97f5 : 1 aida04 fault 0x950 : 0x951 : 1 aida05 fault 0x8e93 : 0x8e94 : 1 aida06 fault 0x545a : 0x545b : 1 aida07 fault 0xc04d : 0xc04e : 1 aida08 fault 0xa0f7 : 0xa0f8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 12 4 6 1 2 3 2 2 2 4 6 : 37072 aida02 : 12 2 4 1 3 4 3 3 2 4 6 : 37984 aida03 : 12 6 7 6 2 4 3 2 3 3 6 : 36624 aida04 : 36 11 4 6 4 3 2 3 3 3 6 : 36968 aida05 : 2 4 6 3 2 2 2 2 2 4 6 : 36968 aida06 : 8 5 3 2 2 2 2 3 3 4 6 : 38456 aida07 : 10 4 6 4 3 3 3 3 2 4 6 : 37992 aida08 : 21 8 3 2 3 3 3 3 2 4 6 : 37956 aida09 : 17 10 3 2 3 4 3 3 1 4 6 : 37060 aida10 : 3 6 10 2 1 3 3 3 1 4 6 : 36828 aida11 : 12 6 4 2 2 3 3 3 1 4 6 : 36832 aida12 : 2 9 5 1 3 3 2 3 2 4 6 : 37632 aida13 : 4 7 2 1 3 3 3 3 2 4 6 : 37832 aida14 : 15 6 6 1 0 4 2 2 2 4 6 : 37100 aida15 : 5 4 3 2 1 3 3 4 2 4 6 : 38244 aida16 : 17 6 6 2 1 3 2 2 2 4 6 : 37076 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.30 analysis of file S496/R18_340 - attachment 9 max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-05-26_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-26_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-26_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-26_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 920.png

Attachment 8: 921.png

Attachment 9: R18_368
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 60865731 ( 22299.7 Hz) Other data format: 201054280 ( 73661.2 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 152 ( 0.1 Hz) RESUME: 152 ( 0.1 Hz) SYNC100: 42166 ( 15.4 Hz) WR48-63: 42166 ( 15.4 Hz) FEE64 disc: 1309767 ( 479.9 Hz) MBS info: 199659877 ( 73150.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 107 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2729.447 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2401.079 2 0.453 0.000 3 8.570 0.000 4 0.000 0.000 5 0.000 1507.937 6 0.000 0.000 7 0.000 1499.170 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 1115 29 0 0 0 0 1 1 27 0 0 10 1 2766811 11645 0 0 0 0 574 574 10497 0 0 18 2 2194 100014163 0 0 27 27 17354 17354 0 99979401 0 1 3 44009553 100997602 0 0 125 125 21286 21286 1274310 99680470 0 5 4 902 6 0 0 0 0 0 0 0 6 0 0 5 6872755 14651 0 0 0 0 1451 1451 11749 0 0 10 6 1948 13 0 0 0 0 1 1 11 0 0 9 7 7202790 16106 0 0 0 0 1497 1497 13112 0 0 9 8 531 0 0 0 0 0 0 0 0 0 0 0 9 394 0 0 0 0 0 0 0 0 0 0 3 10 1118 1 0 0 0 0 0 0 1 0 0 0 11 900 26 0 0 0 0 0 0 26 0 0 12 12 1756 13 0 0 0 0 0 0 13 0 0 8 13 1346 11 0 0 0 0 1 1 9 0 0 7 14 0 0 0 0 0 0 0 0 0 0 0 0 15 1618 14 0 0 0 0 1 1 12 0 0 15 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.695s ( 1009.613 blocks/s, 63.101 Mb/s)
	347	Tue May 25 11:16:54 2021	TD	Tuesday 25 May
12.16 DAQ continues file S496/R18_324 alpha background Tape server c. 2.5Mb/s ASIC check Tape server c. 300kb/s - presumably hot HEC channels 12.20 All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b1 : 1 aida02 fault 0xfe72 : 0xfe73 : 1 aida03 fault 0x97f4 : 0x97f5 : 1 aida04 fault 0x950 : 0x951 : 1 aida05 fault 0x8e93 : 0x8e94 : 1 aida06 fault 0x545a : 0x545b : 1 aida07 fault 0xc04d : 0xc04e : 1 aida08 fault 0xa0f7 : 0xa0f8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 6 6 2 1 3 2 2 2 4 6 : 37016 aida02 : 3 6 5 3 3 4 3 2 1 3 7 : 38572 aida03 : 21 10 8 6 3 3 2 3 2 4 6 : 37924 aida04 : 29 15 4 5 3 4 2 3 3 3 6 : 37004 aida05 : 17 5 2 2 2 2 2 2 2 4 6 : 36940 aida06 : 1 6 0 1 2 2 2 3 3 4 6 : 38356 aida07 : 11 6 7 4 3 3 3 3 2 4 6 : 38028 aida08 : 13 8 3 2 2 3 3 2 1 3 7 : 38372 aida09 : 7 9 2 1 3 3 3 3 2 4 6 : 37860 aida10 : 9 8 4 2 0 3 3 3 1 4 6 : 36708 aida11 : 16 6 3 2 2 3 3 3 1 4 6 : 36832 aida12 : 21 10 7 4 2 4 2 2 1 3 7 : 38420 aida13 : 8 3 3 1 3 3 3 3 2 4 6 : 37832 aida14 : 14 3 6 1 0 4 2 2 2 4 6 : 37072 aida15 : 2 4 2 2 1 3 3 4 2 4 6 : 38216 aida16 : 16 5 5 3 0 3 2 2 2 4 6 : 37016 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1 Rate spectra - attachment 2 ADC data item stats - attachment 3 FEE64 temps OK - attachment 4 DSSSD bias & leakage current OK - attachment 5 Merger/Tape Server/Merger stats - attachment 6 no merger errors reported since previous restart 17.30 analysis of file S496/R18_340 max deadtime 0.36% aida04
Attachment 1: Screenshot_2021-05-25_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-25_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-05-25_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-05-25_Temperature_and_status_scan_aidas-gsi.png

Attachment 5: 910.png

Attachment 6: 911.png

Attachment 7: R18_340
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 63855540 ( 23748.1 Hz) Other data format: 198064462 ( 73660.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 109 ( 0.0 Hz) RESUME: 109 ( 0.0 Hz) SYNC100: 42015 ( 15.6 Hz) WR48-63: 42015 ( 15.6 Hz) FEE64 disc: 1291036 ( 480.1 Hz) MBS info: 196689178 ( 73149.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 35 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2688.871 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2304.866 2 0.344 0.000 3 9.625 0.000 4 0.000 0.000 5 0.000 1450.665 6 0.000 0.000 7 0.000 1243.447 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 303 7 0 0 0 0 1 1 5 0 0 5 1 3031185 1240 0 0 0 0 620 620 0 0 0 0 2 536 98548687 0 0 20 20 16872 16872 2 98514901 0 3 3 44821969 99507955 0 0 89 89 21248 21248 1291004 98174277 0 0 4 514 3 0 0 0 0 1 1 1 0 0 1 5 7529304 3056 0 0 0 0 1528 1528 0 0 0 0 6 1314 2 0 0 0 0 0 0 2 0 0 2 7 8467764 3488 0 0 0 0 1744 1744 0 0 0 0 8 226 4 0 0 0 0 0 0 4 0 0 4 9 144 4 0 0 0 0 0 0 4 0 0 4 10 524 1 0 0 0 0 0 0 1 0 0 0 11 432 2 0 0 0 0 0 0 2 0 0 2 12 719 0 0 0 0 0 0 0 0 0 0 0 13 154 7 0 0 0 0 1 1 5 0 0 7 14 0 0 0 0 0 0 0 0 0 0 0 0 15 452 6 0 0 0 0 0 0 6 0 0 7 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 39.492s ( 810.287 blocks/s, 50.643 Mb/s)
	346	Mon May 24 09:57:07 2021	TD	Monday 24 May
07.30 Hot HEC channels multiple FEE64s ASIC check fixes 10.56 DAQ continues OK file S496/R18_75 System wide checks OK * except* FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Base Current Difference aida01 fault 0x82b0 : 0x82b1 : 1 aida02 fault 0xfe72 : 0xfe73 : 1 aida03 fault 0x97f4 : 0x97f5 : 1 aida04 fault 0x950 : 0x951 : 1 aida05 fault 0x8e93 : 0x8e94 : 1 aida06 fault 0x545a : 0x545b : 1 aida07 fault 0xc04d : 0xc04e : 1 aida08 fault 0xa0f7 : 0xa0f8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 2 7 2 1 4 3 2 1 4 7 : 40452 aida02 : 13 3 3 1 3 4 2 2 2 4 7 : 41308 aida03 : 17 6 2 2 1 4 2 3 2 3 7 : 39700 aida04 : 8 6 1 0 1 3 1 4 2 3 7 : 39712 aida05 : 10 3 5 3 4 3 3 2 1 4 7 : 40560 aida06 : 4 4 2 0 2 3 2 3 3 4 7 : 42576 aida07 : 5 3 2 3 1 3 1 3 2 4 7 : 41324 aida08 : 11 5 1 1 2 3 3 2 2 4 7 : 41348 aida09 : 7 4 3 0 2 3 1 3 2 4 7 : 41324 aida10 : 11 4 2 2 2 3 2 4 1 4 7 : 41132 aida11 : 18 5 0 2 2 4 2 2 1 4 7 : 40240 aida12 : 2 3 0 1 2 3 2 2 2 4 7 : 41024 aida13 : 4 5 1 2 2 3 2 4 1 4 7 : 41096 aida14 : 9 3 1 2 1 4 2 2 2 4 7 : 41164 aida15 : 2 3 1 3 2 2 3 3 2 4 7 : 41744 aida16 : 5 5 2 0 3 3 2 4 1 4 7 : 41116 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5 1.8.H spectra - attachments 6-8 1.8.L spectra - attachments 9 & 10 Stat spectra - attachments 11 & 12 1.8.W spectra - 20us FSR - attachments 13 & 14 ADC data items - attachment 15 FEE64 temperatures OK - attachment 16 DSSSD bias & leakage currents OK - attachment 17 Merger/Tape Server/Merger statistics - attachment 18 no merger errors reported since previous restart analysis of file R18_74 max dead time 0.05% 10.17 ASIC check All histograms, statistics, merger statistics zero'd
Attachment 1: Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-24_Statistics_aidas-gsi(4).png

Attachment 3: Screenshot_2021-05-24_Statistics_aidas-gsi(3).png

Attachment 4: Screenshot_2021-05-24_Statistics_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-24_Statistics_aidas-gsi(1).png

Attachment 6: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png

Attachment 7: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png

Attachment 8: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png

Attachment 9: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png

Attachment 10: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png

Attachment 11: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png

Attachment 12: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png

Attachment 13: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png

Attachment 14: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png

Attachment 15: Screenshot_2021-05-24_Statistics_aidas-gsi.png

Attachment 16: Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png

Attachment 17: 900.png

Attachment 18: 901.png

Attachment 19: R18_74
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 42982076 ( 14466.2 Hz) Other data format: 218937925 ( 73686.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 59 ( 0.0 Hz) RESUME: 59 ( 0.0 Hz) SYNC100: 43069 ( 14.5 Hz) WR48-63: 43069 ( 14.5 Hz) FEE64 disc: 1163669 ( 391.6 Hz) MBS info: 217688000 ( 73265.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2971.209 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 0.000 2 0.425 0.000 3 1.488 0.000 4 0.000 0.000 5 0.000 2622.333 6 0.000 0.000 7 0.000 2340.476 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 64 0 0 0 0 0 0 0 0 0 0 0 1 2227 3 0 0 0 0 1 1 1 0 0 0 2 75 108903724 0 0 25 25 19964 19964 0 108863746 0 0 3 35357926 110031040 0 0 34 34 21525 21525 1163668 108824254 0 0 4 89 0 0 0 0 0 0 0 0 0 0 0 5 3145189 1316 0 0 0 0 658 658 0 0 0 0 6 64 0 0 0 0 0 0 0 0 0 0 0 7 4475895 1842 0 0 0 0 921 921 0 0 0 0 8 56 0 0 0 0 0 0 0 0 0 0 0 9 67 0 0 0 0 0 0 0 0 0 0 0 10 115 0 0 0 0 0 0 0 0 0 0 0 11 65 0 0 0 0 0 0 0 0 0 0 0 12 65 0 0 0 0 0 0 0 0 0 0 0 13 77 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 102 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 40.766s ( 784.975 blocks/s, 49.061 Mb/s)
	345	Sat May 22 23:40:32 2021	TD	Sunday 23 May
00.40 Unable to connect to FEE64s 1-7 DAQ power cycled OK ASIC settings 2021Apr29-13-16-00 slow comparator 0x64 all FEE64s BNC PB-5 OFF file S496/R16 alpha background 07.17 Some hot HEC channels ASIC check fixes issue System wide checks OK except Base Current Difference aida06 fault 0x5457 : 0x5457 : 0 aida06 : WR status 0x10 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x1cd : 461 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 16 9 4 3 1 4 3 2 1 4 7 : 40552 aida02 : 3 2 2 1 2 3 3 2 2 4 7 : 41308 aida03 : 14 5 0 1 1 3 2 4 2 3 7 : 40000 aida04 : 14 4 5 1 3 3 1 4 2 3 7 : 39944 aida05 : 12 4 1 3 2 3 3 2 2 4 7 : 41408 aida06 : 8 3 2 1 2 4 2 3 3 4 7 : 42744 aida07 : 10 4 3 2 1 2 2 3 2 4 7 : 41464 aida08 : 7 5 2 0 3 3 3 2 2 4 7 : 41380 aida09 : 10 5 1 1 1 2 2 3 2 4 7 : 41408 aida10 : 11 6 3 2 1 3 2 2 2 4 7 : 41100 aida11 : 12 8 3 2 2 2 3 2 1 4 7 : 40288 aida12 : 9 9 3 1 2 3 3 2 1 4 7 : 40380 aida13 : 10 4 0 1 1 2 1 3 2 4 7 : 41128 aida14 : 9 4 2 1 1 2 1 3 2 4 7 : 41156 aida15 : 5 4 0 3 3 3 3 3 2 4 7 : 41940 aida16 : 9 6 3 2 0 2 1 3 2 4 7 : 41156 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 00:29:48 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE64 temperatures OK - attachment 1 ADC data item stats - attachment 2 Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3 DSSSD bias & leakage currents OK - attachment 4 11.18 Analysis of file S496/R16_33 - attachment 5 [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 685M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.0T 867G 88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive 867/2 x 3279s = 16.5 days to filesystem full - OK 12.11 aida15 zero data/poll stats - re-loaded DAQ program OK aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK DAQ continues file S496/R18
Attachment 1: Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-23_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png

Attachment 4: 810.png

Attachment 5: R16_33
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 47385414 ( 14450.2 Hz) Other data format: 214534594 ( 65422.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 61 ( 0.0 Hz) RESUME: 61 ( 0.0 Hz) SYNC100: 44215 ( 13.5 Hz) WR48-63: 44215 ( 13.5 Hz) FEE64 disc: 1311957 ( 400.1 Hz) MBS info: 213134085 ( 64995.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 3279.230 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 0.000 2 0.386 0.000 3 1.000 0.000 4 0.000 0.000 5 0.000 2849.824 6 0.000 0.000 7 0.000 2608.689 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 71 0 0 0 0 0 0 0 0 0 0 0 1 1455 0 0 0 0 0 0 0 0 0 0 0 2 87 106617422 0 0 35 35 20186 20186 0 106576980 0 0 3 39184250 107913590 0 0 26 26 22238 22238 1311957 106557105 0 0 4 63 0 0 0 0 0 0 0 0 0 0 0 5 3345685 1512 0 0 0 0 756 756 0 0 0 0 6 60 0 0 0 0 0 0 0 0 0 0 0 7 4853274 2070 0 0 0 0 1035 1035 0 0 0 0 8 69 0 0 0 0 0 0 0 0 0 0 0 9 45 0 0 0 0 0 0 0 0 0 0 0 10 70 0 0 0 0 0 0 0 0 0 0 0 11 68 0 0 0 0 0 0 0 0 0 0 0 12 89 0 0 0 0 0 0 0 0 0 0 0 13 57 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 71 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 44.293s ( 722.462 blocks/s, 45.154 Mb/s)
	344	Sat May 22 13:40:56 2021	TD	Saturday 22 May
14.37 DAQ continues file S496/R15_14 Grafana - DSSSD bias & leakage current - attachment 1 1.8.L spectra - attachments 2 & 3 Stat spectra - attachments 4 & 5 FEE64 temperatures OK - attachment 6 stats - ADC data items, pause items OK - attachments 7 & 8 DSSSD bias & leakage current OK - attachment 9 Merger/Tape Server/Merger stats - attachment 10 analysis of file S496/R14_1920 - attachment 11 all FEE64 deadtime 0% as expected 17.30 aida01, 3, 5, 9 & 11 have hot HEC channels ASIC check fixes All histogram, statistics & merger statistics zero'd DAQ continues Note gamma-ray source tests have started with FATIMA and DEGAS 20.20 Analysis file R15_19 - attachment 12 [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 665M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.3T 530G 93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 90/2 x 3168s = 40h to filesystem full - OK 20.22 All system wide checks OK except Base Current Difference aida01 fault 0xf82c : 0xf82f : 3 aida02 fault 0xdda6 : 0xdda9 : 3 aida03 fault 0xb13d : 0xb140 : 3 aida04 fault 0xf4cd : 0xf4d0 : 3 aida05 fault 0x500 : 0x575 : 117 aida06 fault 0xc6e7 : 0xc6e9 : 2 aida07 fault 0x4920 : 0x4922 : 2 aida08 fault 0xbd9a : 0xbd9c : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 8 4 6 6 1 4 3 3 4 3 6 : 38048 aida02 : 11 8 10 3 4 4 3 4 3 3 6 : 37740 aida03 : 2 8 3 3 2 2 1 4 2 4 6 : 37720 aida04 : 26 15 17 4 4 2 1 4 2 4 6 : 38256 aida05 : 15 6 4 1 1 2 1 3 3 4 6 : 38156 aida06 : 7 7 16 6 3 4 2 2 3 4 6 : 38612 aida07 : 11 5 12 3 3 3 2 3 1 3 7 : 38836 aida08 : 13 6 7 2 1 2 2 2 3 4 6 : 37972 aida09 : 23 3 8 5 0 5 3 3 4 3 6 : 38164 aida10 : 12 5 7 1 2 2 4 3 2 4 6 : 37992 aida11 : 19 2 7 1 2 2 1 2 3 4 6 : 37740 aida12 : 21 5 3 3 3 3 1 2 3 4 6 : 37964 aida13 : 18 7 7 1 2 4 2 3 2 4 6 : 37776 aida14 : 8 3 13 7 2 4 2 3 2 4 6 : 37992 aida15 : 11 5 14 1 3 4 2 3 2 4 6 : 37908 aida16 : 17 6 6 2 2 3 1 2 3 4 6 : 37908 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19.30 Lost activity monitors - attachments 13-15 20.54 1.8.W spectra 20us FSR attachments 16 & 23 1.8.W spectra 200us FSR attachments 17 & 22 1.8.W spectra 2ms FSR attachments 18 & 21 1.8.W spectra 20ms FSR attachments 19 & 20
Attachment 1: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana(1).png

Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(11).png

Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(10).png

Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(9).png

Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(8).png

Attachment 6: Screenshot_2021-05-22_Statistics_aidas-gsi(9).png

Attachment 7: Screenshot_2021-05-22_Statistics_aidas-gsi(8).png

Attachment 8: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(3).png

Attachment 9: 805.png

Attachment 10: 806.png

Attachment 11: R15_13
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 102550190 ( 10129.0 Hz) Other data format: 159369822 ( 15741.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 69714 ( 6.9 Hz) WR48-63: 69714 ( 6.9 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 159230394 ( 15727.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 10124.450 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 0.000 2 0.000 67.951 3 0.000 0.000 4 0.000 0.000 5 0.000 9971.491 6 0.000 0.000 7 0.000 10069.837 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 213 0 0 0 0 0 0 0 0 0 0 0 1 1159 2 0 0 0 0 1 1 0 0 0 0 2 279 79660137 0 0 0 0 22470 22470 0 79615197 0 0 3 101219504 79708933 0 0 0 0 46868 46868 0 79615197 0 0 4 246 0 0 0 0 0 0 0 0 0 0 0 5 588847 366 0 0 0 0 183 183 0 0 0 0 6 280 0 0 0 0 0 0 0 0 0 0 0 7 737827 382 0 0 0 0 191 191 0 0 0 0 8 262 0 0 0 0 0 0 0 0 0 0 0 9 221 0 0 0 0 0 0 0 0 0 0 0 10 242 2 0 0 0 0 1 1 0 0 0 0 11 210 0 0 0 0 0 0 0 0 0 0 0 12 273 0 0 0 0 0 0 0 0 0 0 0 13 193 0 0 0 0 0 0 0 0 0 0 0 14 250 0 0 0 0 0 0 0 0 0 0 0 15 184 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 45.816s ( 698.440 blocks/s, 43.652 Mb/s)
Attachment 12: R15_19
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 55871232 ( 17633.8 Hz) Other data format: 206048770 ( 65032.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 56 ( 0.0 Hz) RESUME: 56 ( 0.0 Hz) SYNC100: 43803 ( 13.8 Hz) WR48-63: 43803 ( 13.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 205961052 ( 65004.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 3168.420 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2865.072 2 0.000 0.000 3 0.419 0.000 4 0.000 0.000 5 0.000 2166.184 6 0.000 0.000 7 0.000 1949.169 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 55 0 0 0 0 0 0 0 0 0 0 0 1 133932 56 0 0 0 0 28 28 0 0 0 0 2 84 103025372 0 0 0 0 18982 18982 0 102987408 0 0 3 42389344 103017790 0 0 56 56 22017 22017 0 102973644 0 0 4 56 0 0 0 0 0 0 0 0 0 0 0 5 6040914 2574 0 0 0 0 1287 1287 0 0 0 0 6 103 0 0 0 0 0 0 0 0 0 0 0 7 7306081 2976 0 0 0 0 1488 1488 0 0 0 0 8 78 0 0 0 0 0 0 0 0 0 0 0 9 86 0 0 0 0 0 0 0 0 0 0 0 10 68 0 0 0 0 0 0 0 0 0 0 0 11 75 0 0 0 0 0 0 0 0 0 0 0 12 116 2 0 0 0 0 1 1 0 0 0 0 13 80 0 0 0 0 0 0 0 0 0 0 0 14 80 0 0 0 0 0 0 0 0 0 0 0 15 80 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.406s ( 737.221 blocks/s, 46.076 Mb/s)
Attachment 13: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(4).png

Attachment 14: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(3).png

Attachment 15: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(2).png

Attachment 16: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(19).png

Attachment 17: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(18).png

Attachment 18: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(17).png

Attachment 19: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(16).png

Attachment 20: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(15).png

Attachment 21: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(14).png

Attachment 22: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(13).png

Attachment 23: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(12).png

	343	Fri May 21 23:19:23 2021	TD	Saturday 22 May 00:00-08:00
00.16 413Gb remaining /TapeData filesystem estimate 413/2 x 177s = 10.15h to filesystem full - OK 00.18 All histograms, statistics and merger statistics zero'd 00.57 All system wide checks OK except Base Current Difference aida05 fault 0x500 : 0x572 : 114 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 9 3 4 1 5 2 2 3 3 6 : 36356 aida02 : 4 8 7 2 4 3 3 3 2 3 6 : 35968 aida03 : 12 9 6 2 5 3 2 3 3 3 6 : 36824 aida04 : 11 5 9 2 2 2 2 4 3 3 6 : 37028 aida05 : 24 9 2 1 2 3 2 3 4 3 6 : 37608 aida06 : 25 6 3 4 2 4 3 3 3 3 6 : 37060 aida07 : 3 3 3 1 3 3 2 2 2 2 7 : 37044 aida08 : 20 4 2 3 1 3 2 2 3 3 6 : 36016 aida09 : 15 8 2 5 0 3 3 2 3 3 6 : 36284 aida10 : 16 6 4 1 2 3 3 2 2 4 6 : 37328 aida11 : 20 2 2 1 1 3 2 2 3 3 6 : 35936 aida12 : 1 2 1 2 3 4 1 2 4 3 6 : 36900 aida13 : 25 10 4 1 0 3 2 3 3 3 6 : 36500 aida14 : 20 6 9 2 2 5 1 2 2 4 6 : 37200 aida15 : 31 9 5 2 3 3 1 3 3 3 6 : 36500 aida16 : 24 4 2 3 1 4 2 3 3 3 6 : 36672 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 01.12 Lost activity monitor - attachment 1 1.8.H spectra - attachments 2 & 3 1.8.L spectra - attachments 4-6 aida01 pulser peak width 120 ch FWHM Rate specttra - attachments 7 & 8 01.06 Lost activity monitor - attachment 9 ucesb - attachment 10 Grafana - DSSSD bias & leakage currents past 7 days - attachment 11 statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17 FEE64 temperatures OK - attachment 18 DSSSD bias & leakage currents OK - attachment 19 Merger/Tape Server/Merger stats - attachment 20 no merger warning/error messages since last restart 01.26 Analysis file S496/R14_1798 - attachment 21 aida08 5.5% deadtime 01.17 Disk space remaining /TapeData filesystem [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 706M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.4T 395G 95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 395/2 x 177s = 9.7h to filesystem full - OK 01.30 despec shift reports no data reported by ucesb aida stats & tape server ok - no data being forwarded to MBS by data relay 01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811 04.35 System wide checks all OK except Base Current Difference aida05 fault 0x500 : 0x573 : 115 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 661M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.5T 337G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 337/2 x 177s = 8.3h to filesystem full - OK FEE64 temperatures OK - attachment 22 ADC data item statistics OK - attachment 24 DSSSD bias & leakage current OK - attachment 26 analysis of file S496/R14_1920 - attachment 28 aida04 deadtime 5.0% 07.10 Ssystem wide checks all OK except Base Current Difference aida05 fault 0x500 : 0x573 : 115 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 662M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.5T 287G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 287/2 x 178s = 7.1h to filesystem full - OK FEE64 temperatures OK - attachment 23 ADC data item statistics OK - attachment 25 DSSSD bias & leakage current OK - attachment 27 analysis of file S496/R14_1920 - attachment 29 aida04 deadtime 4.4% 07.35 Found MBS data monitor kill'd Restarted MBS data relay and data monitor OK 08.00 beam off - end of S496 08.28 Tape Server -> no storage mode DAQ continues 09.58 All slow comparators -> 0x64 BNC PB-5 OFF file S496/R15 alpha background Tape server 200kb/s, merger 260k data items/s
Attachment 1: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(1).png

Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(7).png

Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(6).png

Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(5).png

Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(2).png

Attachment 7: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(1).png

Attachment 8: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi.png

Attachment 9: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi.png

Attachment 10: Screenshot_2021-05-22_ucesb.png

Attachment 11: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana.png

Attachment 12: Screenshot_2021-05-22_Statistics_aidas-gsi(5).png

Attachment 13: Screenshot_2021-05-22_Statistics_aidas-gsi(4).png

Attachment 14: Screenshot_2021-05-22_Statistics_aidas-gsi(3).png

Attachment 15: Screenshot_2021-05-22_Statistics_aidas-gsi(2).png

Attachment 16: Screenshot_2021-05-22_Statistics_aidas-gsi(1).png

Attachment 17: Screenshot_2021-05-22_Statistics_aidas-gsi.png

Attachment 18: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi.png

Attachment 19: 800.png

Attachment 20: 801.png

Attachment 21: R14_1798
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254662088 ( 1433466.5 Hz) Other data format: 7257912 ( 40854.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 256 ( 1.4 Hz) RESUME: 257 ( 1.4 Hz) SYNC100: 32661 ( 183.8 Hz) WR48-63: 32661 ( 183.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7192077 ( 40483.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 365795 ( 2059.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 177.655 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.112 0.000 1 5.898 0.000 2 0.060 0.000 3 7.669 0.000 4 0.073 7.902 5 7.648 0.000 6 0.356 0.000 7 9.852 0.000 8 1.513 0.000 9 0.000 15.513 10 2.639 0.000 11 0.903 0.000 12 0.687 0.000 13 0.000 6.517 14 0.073 0.000 15 0.212 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5664289 2546 0 0 5 5 737 737 0 1062 0 19821 1 24002581 6376 0 0 39 39 3149 3149 0 0 0 40581 2 12140787 1878978 0 0 3 3 1758 1758 0 1875456 0 32794 3 29094115 1803103 0 0 54 54 3917 3917 0 1795161 0 46229 4 3034023 1008057 0 0 6 6 480 480 0 1007085 0 8526 5 29014047 561174 0 0 54 54 3636 3636 0 553794 0 24714 6 11191428 1180757 0 0 3 3 1528 1528 0 1177695 0 17521 7 34103710 790567 0 0 51 52 4320 4320 0 781824 0 29424 8 17182670 4348 0 0 9 9 2165 2165 0 0 0 41538 9 2132311 520 0 0 0 0 260 260 0 0 0 3304 10 25432181 6308 0 0 12 12 3142 3142 0 0 0 25046 11 16063496 3832 0 0 8 8 1908 1908 0 0 0 7738 12 11072472 2756 0 0 4 4 1374 1374 0 0 0 35884 13 2789016 664 0 0 0 0 332 332 0 0 0 2683 14 11165568 2934 0 0 4 4 1463 1463 0 0 0 24254 15 20579394 4992 0 0 4 4 2492 2492 0 0 0 5738 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 41.474s ( 771.565 blocks/s, 48.223 Mb/s)
Attachment 22: Screenshot_2021-05-22_Statistics_aidas-gsi(7).png

Attachment 23: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 24: Screenshot_2021-05-22_Statistics_aidas-gsi(6).png

Attachment 25: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 26: 802.png

Attachment 27: 804.png

Attachment 28: R14_1867
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254778017 ( 1461534.6 Hz) Other data format: 7141983 ( 40970.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 228 ( 1.3 Hz) RESUME: 228 ( 1.3 Hz) SYNC100: 32649 ( 187.3 Hz) WR48-63: 32649 ( 187.3 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7076229 ( 40592.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 358653 ( 2057.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 174.322 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.023 0.000 1 8.071 0.000 2 0.956 0.000 3 6.628 0.000 4 0.186 12.353 5 5.668 0.000 6 0.255 0.000 7 8.669 0.000 8 1.570 0.000 9 0.000 28.420 10 2.125 0.000 11 0.164 0.000 12 0.103 0.000 13 0.000 11.688 14 0.094 0.000 15 1.340 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5294176 2413 0 0 1 1 685 685 0 1041 0 18715 1 22878067 6136 0 0 44 44 3024 3024 0 0 0 38998 2 11959613 1840531 0 0 7 7 1681 1681 0 1837155 0 31909 3 28914032 1781740 0 0 51 51 3845 3845 0 1773948 0 44969 4 2970776 981532 0 0 7 7 454 454 0 980610 0 8409 5 29112496 551278 0 0 40 40 3688 3688 0 543822 0 24330 6 11438278 1149690 0 0 4 4 1643 1643 0 1146396 0 17361 7 34470364 802023 0 0 41 41 4342 4342 0 793257 0 30254 8 17110979 4312 0 0 6 6 2150 2150 0 0 0 40990 9 2129568 524 0 0 0 0 262 262 0 0 0 3068 10 25946722 6422 0 0 9 9 3202 3202 0 0 0 25071 11 16268948 3960 0 0 3 3 1977 1977 0 0 0 7119 12 11320269 2746 0 0 3 3 1370 1370 0 0 0 35684 13 2799924 716 0 0 0 0 358 358 0 0 0 2469 14 11389376 2772 0 0 2 2 1384 1384 0 0 0 23994 15 20774429 5188 0 0 10 10 2584 2584 0 0 0 5313 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.000s ( 744.186 blocks/s, 46.512 Mb/s)
Attachment 29: R14_1920
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254781729 ( 1428365.9 Hz) Other data format: 7138271 ( 40018.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 226 ( 1.3 Hz) RESUME: 227 ( 1.3 Hz) SYNC100: 32665 ( 183.1 Hz) WR48-63: 32665 ( 183.1 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7072488 ( 39650.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 352885 ( 1978.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 178.373 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.017 2.836 1 6.610 0.000 2 0.573 0.000 3 7.826 0.000 4 0.024 14.831 5 7.215 0.000 6 0.482 0.000 7 6.876 0.000 8 0.394 0.000 9 0.000 17.832 10 1.157 0.000 11 1.395 0.000 12 0.037 0.000 13 0.004 9.016 14 0.101 0.000 15 1.578 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5306415 2397 0 0 1 1 665 665 0 1065 0 18998 1 22746335 6008 0 0 36 36 2968 2968 0 0 0 37812 2 11931383 1875609 0 0 5 5 1756 1756 0 1872087 0 31542 3 28584441 1795513 0 0 57 57 3773 3773 0 1787853 0 43417 4 2954259 971818 0 0 3 3 494 494 0 970824 0 8522 5 28608195 543853 0 0 51 52 3567 3567 0 536616 0 24072 6 11355091 1133078 0 0 6 6 1573 1573 0 1129920 0 16979 7 34582740 783003 0 0 34 34 4406 4406 0 774123 0 29465 8 17425265 4332 0 0 7 7 2159 2159 0 0 0 40811 9 2130016 550 0 0 0 0 275 275 0 0 0 3176 10 26903024 6616 0 0 4 4 3304 3304 0 0 0 24219 11 16400506 4036 0 0 5 5 2013 2013 0 0 0 7163 12 11085724 2934 0 0 1 1 1466 1466 0 0 0 35361 13 2664218 650 0 0 1 1 324 324 0 0 0 2574 14 11252748 2738 0 0 4 4 1365 1365 0 0 0 23421 15 20851369 5136 0 0 11 11 2557 2557 0 0 0 5353 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.672s ( 732.737 blocks/s, 45.796 Mb/s)
	342	Fri May 21 15:03:48 2021	CA	Friday 21st May 16:00 - 00:00 shift
16:00 CA takes over DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4 DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4 PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 16:14 system wide checks all ok except Base Current Difference aida05 fault 0x500 : 0x570 : 112 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 16:17 temperatures ok - attachment 1 statistics ok - attachment 2 detector bias / leakage currents ok - attachment 3 analysis of R14_1625 - FEE7 deadtime at 4.5% 18:06 system wide checks all ok except Base Current Difference aida05 fault 0x500 : 0x571 : 113 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:07 temperatures ok - attachment 4 statistics ok - attachment 5 detector bias / leakage currents ok - attachment 6 analysis of R14_1662 - FEE7 deadtime at 4.3% 19:07 DAQ continues ok - writing to file R14_1682 20:07 system wide checks all ok except Base Current Difference aida05 fault 0x500 : 0x572 : 114 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 20:10 temperatures ok - attachment 7 statistics ok - attachment 8 detector bias / leakage currents ok - attachment 9 analysis of R14_1703 - max deadtime at 4.0% - attachment 10 20:35 rate spectra - attachment 11 HEC spectra - attachment 12 & 13 20:36 all histograms zeroed 20:37 DAQ continues ok - writing to file R14_1712 20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use. ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files 21:12 no beam - writing to file R14_1723 21:17 beam is back 22:07 system wide checks all ok - WR decoder fault same as entry at 20:07 22:08 temperatures ok - attachment 14 statistics ok - attachment 15 detector bias / leakage currents ok - attachment 16 analysis of R14_1740 - max deadtime at 4.2% - attachment 17
Attachment 1: Screenshot_from_2021-05-21_15-14-59.png

Attachment 2: Screenshot_from_2021-05-21_15-15-38.png

Attachment 3: Screenshot_from_2021-05-21_15-15-54.png

Attachment 4: Screenshot_from_2021-05-21_17-06-56.png

Attachment 5: Screenshot_from_2021-05-21_17-07-18.png

Attachment 6: Screenshot_from_2021-05-21_17-07-47.png

Attachment 7: Screenshot_from_2021-05-21_19-09-19.png

Attachment 8: Screenshot_from_2021-05-21_19-09-59.png

Attachment 9: Screenshot_from_2021-05-21_19-10-15.png

Attachment 10: Screenshot_from_2021-05-21_19-14-45.png

Attachment 11: Screenshot_from_2021-05-21_19-25-51.png

Attachment 12: Screenshot_from_2021-05-21_19-30-14.png

Attachment 13: Screenshot_from_2021-05-21_19-32-13.png

Attachment 14: Screenshot_from_2021-05-21_21-08-28.png

Attachment 15: Screenshot_from_2021-05-21_21-08-51.png

Attachment 16: Screenshot_from_2021-05-21_21-09-06.png

Attachment 17: Screenshot_from_2021-05-21_21-12-03.png

	341	Fri May 21 07:03:50 2021	OH, TD	Friday 21 May 08:00-16:00
08:00 OH and TD take over Current settings p+n sides at 0xc DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4 DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4 PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 08:29 System wide checks ok - WR difference at 111 Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1 Temperature ok - attachment 2 Bias and leakage current ok - attachment 3 09:58 System wide checks ok - WR difference at 111 Statistics ok - attachment 4 Temperature ok - attachment 5 Bias and leakage current ok - attachment 6 Max deadtime this morning has been around 5.5% on FEE8 12:32 System wide checks ok - WR difference at 111 Statistics ok - attachment 7 Temperature ok - attachment 9 Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png

Attachment 2: 210521_0829_Temp.png

Attachment 3: 210521_0830_Bias.png

Attachment 4: 210521_0957.png

Attachment 5: 210521_0957_Temp.png

Attachment 6: 210521_0958_Bias.png

Attachment 7: 210521_1232_Stats.png

Attachment 8: 210521_1233_Bias.png

Attachment 9: 210521_1233_Temp.png

	340	Fri May 21 06:09:28 2021	ML	system wide checks
System wide checks at 7am CET All seems fine, nothing significant to report Slight increase of dead time noted in FEE 3, 5 and 7. ----- Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ----- ADC calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module ----- White Rabbit: Base Current Difference aida05 fault 0x500 : 0x56e : 110 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR ----- FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last ----- Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 23 5 6 5 0 4 1 3 3 3 6 : 36484 aida02 : 10 3 3 2 3 2 2 2 3 3 6 : 35952 aida03 : 2 5 1 4 5 2 2 3 3 3 6 : 36608 aida04 : 1 3 4 0 4 3 2 3 3 3 6 : 36572 aida05 : 19 9 3 2 2 3 1 3 4 3 6 : 37380 aida06 : 13 11 1 5 3 4 4 2 3 3 6 : 36860 aida07 : 12 2 4 3 3 2 1 3 2 2 7 : 37280 aida08 : 25 5 7 1 1 3 3 3 3 3 6 : 36828 aida09 : 22 5 6 4 1 3 2 3 3 3 6 : 36640 aida10 : 23 8 6 3 2 3 2 2 2 4 6 : 37212 aida11 : 24 14 7 3 3 2 3 3 3 3 6 : 36960 aida12 : 16 10 5 3 1 2 1 2 4 3 6 : 36736 aida13 : 23 6 4 2 2 3 2 3 3 3 6 : 36620 aida14 : 22 7 8 4 3 4 1 2 2 4 6 : 37200 aida15 : 10 12 9 2 1 3 1 3 3 3 6 : 36376 aida16 : 18 5 6 2 0 3 2 4 3 3 6 : 37008
Attachment 1: 2021-05-21_7h00_LeakCurrent.png

Attachment 2: 2021-05-21_7h00_Stats_GoodEvents.png

Attachment 3: 2021-05-21_7h00_Stats_ADCitems.png

Attachment 4: 2021-05-21_7h00_stats_Correlation.png

Attachment 5: 2021-05-21_7h00_Stats_PAUSE.png

Attachment 6: 2021-05-21_7h00_Stats_RESUME.png

Attachment 7: 2021-05-21_7h00_temperature.png

Attachment 8: 2021-05-21_7h00_ucesb.png

	339	Fri May 21 02:17:47 2021	ML	system wide checks
Wide check at 3am - All fine. Same output as at the beginning of the shift (see my previous entry) and attachments.
Attachment 1: 2021-05-21_3h05_Leakcurrent.png

Attachment 2: 2021-05-21_3h05_stats_adcItem.png

Attachment 3: 2021-05-21_3h05_stats_Correlation.png

Attachment 4: 2021-05-21_3h05_stats_GoodEvents.png

Attachment 5: 2021-05-21_3h05_stats_PAUSE.png

Attachment 6: 2021-05-21_3h05_stats_RESUME.png

Attachment 7: 2021-05-21_3h05_temperature.png

Attachment 8: 2021-05-21_3h05_ucesb.png

Attachment 9: 2021-05-21_3h05_aidaGrafana.png

	338	Thu May 20 23:26:11 2021	Marc Labiche (ML)	system wide checks
System Wide check at begining of the night shift: ----- Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ----- ADC calibration: FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module ----- White rabbit status: Base Current Difference aida05 fault 0x500 : 0x56d : 109 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR ----- FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last ----- Memory info from FEE cards Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 7 7 6 1 5 2 2 3 3 6 : 36472 aida02 : 19 5 4 4 4 3 3 3 2 3 6 : 36020 aida03 : 22 10 2 2 5 2 2 3 3 3 6 : 36680 aida04 : 12 7 1 0 4 3 1 4 3 3 6 : 36856 aida05 : 16 7 5 3 1 3 2 3 3 3 6 : 36584 aida06 : 20 7 5 6 2 4 4 2 3 3 6 : 36888 aida07 : 13 7 5 2 3 2 1 3 2 2 7 : 37308 aida08 : 18 7 5 3 0 2 3 3 3 3 6 : 36656 aida09 : 21 6 4 4 1 4 2 3 3 3 6 : 36740 aida10 : 21 10 2 3 2 3 2 2 2 4 6 : 37156 aida11 : 23 2 4 4 3 2 2 3 2 4 6 : 37612 aida12 : 24 1 4 3 2 2 1 2 4 3 6 : 36744 aida13 : 26 9 4 2 1 4 3 2 3 3 6 : 36464 aida14 : 19 9 9 2 3 4 1 2 2 4 6 : 37156 aida15 : 10 7 10 0 1 3 1 3 3 3 6 : 36288 aida16 : 28 7 3 2 1 3 3 3 3 3 6 : 36824
Attachment 1: 2021-05-21_0h20_ucesb.png

Attachment 2: 2021-05-21_0h10_Leakcurrent.png

Attachment 3: 2021-05-21_0h10_Stat_adcItem.png

Attachment 4: 2021-05-21_0h10_Stats_Correlation.png

Attachment 5: 2021-05-21_0h10_Stats_GoodEvents.png

Attachment 6: 2021-05-21_0h10_Stats_PAUSE.png

Attachment 7: 2021-05-21_0h10_stats_RESUME.png

Attachment 8: 2021-05-21_0h10_temperature.png

Attachment 9: 2021-05-21_0h30_AIDAgrafana.png

	337	Thu May 20 14:56:33 2021	OH	Thursday 20th May 16:00 - 24:00
16:00 p+n threshold 0xc (120keV) n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4 n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4 Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure. 17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector 18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now. The deadtimes in FEE6 and 8 have risen to around 6.25%. Looking at this mornings and yesterday evenings fill times this behavior isn't observed 18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago 18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2 This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file) 18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8 18:15 System wide checks ok except WR as usual Base Current Difference aida05 fault 0x500 : 0x56d : 109 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics increased at 18:02 still working on - attachment 2 FEE temp - attachment 3 Bias and leakage current - attachment 4 20:51 System wide checks ok - WR same as before Statistics still high - attachment 5 Temperatures ok - attachment 6 Bias and leakage currents ok - attachment 7 22:52 Statistics still high - attachment 8 Temp ok - attachment 9 Bias and leakage current ok - attachment 10 System wide checks ok - WR difference still 109
Attachment 1: 210520_18002_MBSRate.png

Attachment 2: 210520_1916_Stats.png

Attachment 3: 210520_1916_Temp.png

Attachment 4: 210520_1917_Bias.png

Attachment 5: 21050_2052_Stats.png

Attachment 6: 210520_2052_Temp.png

Attachment 7: 210520_2053_Bias.png

Attachment 8: 210520_2251_Stats.png

Attachment 9: 210520_2251_Temp.png

Attachment 10: 210520_2254_Bias.png

	336	Thu May 20 07:03:02 2021	CA	Thursday May 20th 08:00 - 16:00 shift
08:00 CA takes over 08:04 DAQ continues ok - writing to file R14_989 09:30 all system wide checks ok except WR decoder status; Base Current Difference aida05 fault 0x500 : 0x56a : 106 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 09:34 temperatures ok - attachment 1 stats ok - attachment 2 bias/leakage currents ok - attachment 3 R14_1016 analysis - max deadtime ~4.3% - attachment 4 11:30 all system wide checks ok except WR decoder status; Base Current Difference aida05 fault 0x500 : 0x56b : 107 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 11:34 temperatures ok - attachment 5 stats ok - attachment 6 bias/leakage currents ok - attachment 7 R14_1052 analysis - max deadtime ~3.6% - attachment 8 13:05 beam optimisation currently taking place DAQ continues ok and writing to file R14_1085 13:17 no beam 13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30. 13:31 temperatures ok - attachment 9 stats ok - attachment 10 bias/leakage currents ok - attachment 11 R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12 13:37 beam is back - rate spectra - attachment 13 DAQ ok - writing to file R14_1088 13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14 14:21 beam off - issue with UNILAC DESPEC also report issues with their DAQ AIDA continues to forward data to MBS ok - lower rates while beam off 14.30 1.8.H spectra - attachments 16-18 1.8.L spectra - attachments 19-22 aida01 pulser peak width 99 ch FWHM Rate spectra - attachment 23 Grafana - DSSSD bias & leakage currents past 7 days - attachment 24 DSSSD bias & leakage currents - attachment 25 Merger/Tape Server/Merger stats - attachment 26 14.41 All histograms & stats zero'd 14.55 Lost Activity Monitors - attachments 27, 28 & 35 Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler 15:00 beam is back 15:05 AIDA writing to file R14_1108 ADC data items - attachment 36 15:27 system wide checks ok except WR decoder status Base Current Difference aida05 fault 0x500 : 0x56c : 108 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 15:30 temperatures ok - attachment 37 stats ok - attachment 38 bias/leakage currents ok - attachment 39 R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40
Attachment 1: Screenshot_from_2021-05-20_08-33-49.png

Attachment 2: Screenshot_from_2021-05-20_08-34-10.png

Attachment 3: Screenshot_from_2021-05-20_08-34-44.png

Attachment 4: Screenshot_from_2021-05-20_08-37-13.png

Attachment 5: Screenshot_from_2021-05-20_10-32-22.png

Attachment 6: Screenshot_from_2021-05-20_10-32-47.png

Attachment 7: Screenshot_from_2021-05-20_10-33-04.png

Attachment 8: Screenshot_from_2021-05-20_10-36-31.png

Attachment 9: Screenshot_from_2021-05-20_12-29-26.png

Attachment 10: Screenshot_from_2021-05-20_12-29-49.png

Attachment 11: Screenshot_from_2021-05-20_12-30-20.png

Attachment 12: Screenshot_from_2021-05-20_12-33-48.png

Attachment 13: Screenshot_from_2021-05-20_12-38-42.png

Attachment 14: Screenshot_from_2021-05-20_12-38-42.png

Attachment 15: Screenshot_from_2021-05-20_12-43-18.png

Attachment 16: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(7).png

Attachment 17: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(6).png

Attachment 18: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(5).png

Attachment 19: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(4).png

Attachment 20: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(3).png

Attachment 21: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(2).png

Attachment 22: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(1).png

Attachment 23: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi.png

Attachment 24: Screenshot_2021-05-20_AIDA_Alerting_-_Grafana.png

Attachment 25: 710.png

Attachment 26: 711.png

Attachment 27: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(2).png

Attachment 28: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(1).png

Attachment 29: Screenshot_2021-05-20_Statistics_aidas-gsi(5).png

Attachment 30: Screenshot_2021-05-20_Statistics_aidas-gsi(4).png

Attachment 31: Screenshot_2021-05-20_Statistics_aidas-gsi(3).png

Attachment 32: Screenshot_2021-05-20_Statistics_aidas-gsi(2).png

Attachment 33: Screenshot_2021-05-20_Statistics_aidas-gsi(1).png

Attachment 34: Screenshot_2021-05-20_Statistics_aidas-gsi.png

Attachment 35: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi.png

Attachment 36: Screenshot_from_2021-05-20_14-07-22.png

Attachment 37: Screenshot_from_2021-05-20_14-28-33.png

Attachment 38: Screenshot_from_2021-05-20_14-29-04.png

Attachment 39: Screenshot_from_2021-05-20_14-29-18.png

Attachment 40: Screenshot_from_2021-05-20_14-31-50.png

	335	Thu May 20 01:00:30 2021	Muneerah	Thu May 20 00:00-08:00
02:00 System check The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA! The rates on ucesb got back to work by around 01:12 Deat Time check ok R14_870, attachment 1 Statistics are ok attachment 2 Temperatures are ok attacment 3 Voltages are ok attachment 4 system wide check All ok except ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbite 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 8 8 5 0 6 1 2 2 2 7 : 37296 aida02 : 19 8 11 2 3 4 2 2 3 3 6 : 36412 aida03 : 14 6 3 4 1 3 1 3 3 3 6 : 36312 aida04 : 25 12 12 4 3 4 1 4 3 3 6 : 37316 aida05 : 4 8 3 4 1 2 2 4 3 3 6 : 36928 aida06 : 42 12 8 7 2 3 3 4 3 3 6 : 37736 aida07 : 24 6 7 3 3 3 1 3 2 2 7 : 37536 aida08 : 10 15 7 2 1 4 2 3 3 3 6 : 36752 aida09 : 29 13 6 6 4 3 3 3 3 3 6 : 37244 aida10 : 18 12 7 2 2 3 3 2 2 4 6 : 37464 aida11 : 20 8 0 5 5 4 2 3 3 3 6 : 36976 aida12 : 7 2 4 3 2 3 2 2 4 3 6 : 37068 aida13 : 2 5 5 4 2 3 2 3 3 3 6 : 36608 aida14 : 9 13 8 2 2 2 3 3 2 4 6 : 37836 aida15 : 24 12 12 1 0 3 2 3 3 3 6 : 36640 aida16 : 12 9 9 2 1 3 2 4 3 3 6 : 37128 6:45 System check Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9 System wide check are all ok except: ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 11 3 5 0 6 1 2 3 2 7 : 38264 aida02 : 14 9 9 2 3 3 3 2 3 3 6 : 36496 aida03 : 13 7 3 5 2 3 2 3 3 3 6 : 36668 aida04 : 30 15 13 4 3 4 1 3 4 3 6 : 37888 aida05 : 17 9 3 3 1 2 2 4 3 3 6 : 36956 aida06 : 40 16 5 6 2 3 2 3 2 4 6 : 37936 aida07 : 3 2 3 2 3 4 2 2 2 2 7 : 37196 aida08 : 17 10 6 2 1 3 3 3 3 3 6 : 36852 aida09 : 29 11 7 6 3 3 2 4 3 3 6 : 37436 aida10 : 1 10 5 1 1 3 3 2 2 4 6 : 37252 aida11 : 30 9 0 5 4 4 2 3 3 3 6 : 36960 aida12 : 19 10 5 3 2 3 2 2 4 3 6 : 37196 aida13 : 22 9 6 3 2 3 3 3 3 3 6 : 36960 aida14 : 19 6 9 2 2 3 3 2 2 4 6 : 37452 aida15 : 6 8 11 2 1 3 2 3 3 3 6 : 36616 aida16 : 19 14 5 3 1 3 1 4 3 3 6 : 36908 07:33 system check Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13 System wide check all ok except ADC all failed White rabbit 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 9 4 6 0 5 1 2 2 2 7 : 37152 aida02 : 19 6 13 1 3 3 4 2 3 3 6 : 36780 aida03 : 23 5 6 4 2 3 1 3 3 3 6 : 36452 aida04 : 24 11 14 3 3 4 2 3 3 3 6 : 37048 aida05 : 17 3 6 3 1 2 2 3 3 3 6 : 36444 aida06 : 36 15 4 9 2 4 2 4 3 3 6 : 37608 aida07 : 20 8 5 3 3 4 2 2 2 2 7 : 37376 aida08 : 21 11 10 2 1 4 2 2 2 4 6 : 37324 aida09 : 27 14 7 6 3 3 2 4 3 3 6 : 37452 aida10 : 25 9 5 2 2 3 3 2 2 4 6 : 37436 aida11 : 30 14 2 4 5 3 2 3 3 3 6 : 36936 aida12 : 22 3 5 3 2 3 2 3 3 3 6 : 36640 aida13 : 10 7 1 4 2 3 3 2 3 3 6 : 36336 aida14 : 13 9 12 2 3 3 2 3 2 4 6 : 37820 aida15 : 20 8 11 3 0 3 2 3 3 3 6 : 36640 aida16 : 23 6 6 2 0 2 1 3 4 3 6 : 37164
Attachment 1: 210520_0156_DT.png

Attachment 2: 210520_0157_Stats.png

Attachment 3: 210520_0157_Temps.png

Attachment 4: 210520_0157_Voltages.png

Attachment 5: 210520_0537_Temps.png

Attachment 6: 210520_0539_VOltages.png

Attachment 7: 210520_0540_Histo.png

Attachment 8: 210520_0541_Rates.png

Attachment 9: 210520_0542_DT.png

Attachment 10: 210520_0730_DT.png

Attachment 11: 210520_0730_Voltages.png

Attachment 12: 210520_0731_Stats.png

Attachment 13: 210520_0731_Tems.png

	334	Wed May 19 20:02:27 2021	NH	AIDA Dead Time Analysis
Some plots of AIDA dead time (From PAUSE/RESUME data items) against time. The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead. Beam spills are indicated by the red bars (1 = on spill) Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates . The much quieter p+n strips show a similar pattern but much less frequently. Deadtime does not always sync up between FEE64s it seems. IN all case the pause/resume interval seems to only last for less than 100 ms.
Attachment 1: dt_fee2.png

Attachment 2: dt_fee4.png

Attachment 3: dt_fee6.png

Attachment 4: dt_fee8.png

Attachment 5: dt_fee9.png

Attachment 6: dt_fee9_lr.png

	333	Wed May 19 18:50:01 2021	LS	Wednesday 19th May 20:00 to 24:00
20.00 Stats okay (Fig 1) Analysed R14_762 deadtimes are okay (Fig 2) 20.30 Stats okay (Fig 3) Analysed R14_771 deadtimes are okay (Fig 4) 21.00 Stats okay (Fig 5) FEE temps okay (Fig 6) Bias (Fig 7) Rate spectra (Fig 8) System wide checks okay except white rabbit 1 fail: Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~2.8M items/s Tape server ~10MB/s Analysed R14_781 deadtimes are okay (Fig 9) 21.30 Stats okay (Fig 10) Analysed R14_789 deadtimes are okay (Fig 11) 22.00 Stats okay (Fig 12) Analysed R14_799 deadtimes are okay (Fig 13) 22.30 Stats okay (Fig 14) Analysed R14_807 deadtimes are okay (Fig 15) 23.00 Stats okay (Fig 16) FEE temps okay (Fig 17) Bias (Fig 18) System wide checks okay except white rabbit 1 fail: Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~3M items/s Tape server ~10MB/s Analysed R14_819 deadtimes are okay (Fig 19) 23.30 Stats okay (Fig 20) Analysed R14_827 deadtimes are okay (Fig 21)
Attachment 1: 210519_Stats_2000.png

Attachment 2: Deadtimes_R14_762.png

Attachment 3: 210519_Stats_2030.png

Attachment 4: Deadtimes_R14_771.png

Attachment 5: 210519_Stats_2100.png

Attachment 6: 210519_FEEtemps_2100.png

Attachment 7: 210519_BIAS_2100.png

Attachment 8: 210519_Spectra_2100.png

Attachment 9: deadtimes_R14_781.png

Attachment 10: 210519_Stats_2130.png

Attachment 11: deadtimes_R14_789.png

Attachment 12: 210519_Stats_2200.png

Attachment 13: deadtimes_R14_799.png

Attachment 14: 210519_Stats_2230.png

Attachment 15: deatimes_R14_807.png

Attachment 16: 210519_Stats_2300.png

Attachment 17: 210519_FEEtemps_2300.png

Attachment 18: 210519_bias_2300.png

Attachment 19: deatimes_R14_819.png

Attachment 20: 210519_Stats_2330.png

Attachment 21: deadtimes_R14_827.png

Goto page Previous 1, 2, 3 ... 18, 19, 20 ... 34, 35, 36 Next

ELOG V3.1.4-unknown