ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 18 of 36

New entries since:

Thu Jan 1 01:00:00 1970

Full | Summary | Threaded | Hide attachments

703 Entries

Goto page Previous 1, 2, 3 ... 17, 18, 19 ... 34, 35, 36 Next

ID	Date	Author	Subject
373	Tue Jun 22 13:01:19 2021	NH	AIDA Data on lustre
The following runs are now copied to lustre If necessary they can be removed from AIDA HDDs for space. There are also spare HDDs that can (and should) be swapped for the current ones /lustre/gamma/DESPEC_S452_FILES/AIDA S452 R1 R2 R3 R4 R5 R6 R7 R9 R11 R13 R14 R15 R16 R17 R20 R21 R22 R23 R25 R27 R28 R29 R30 R33 R34 R35 R36 R38 R39 R40 R42 R43 R46 R50 R56 R57 R58 /lustre/gamma/DESPEC_S460_FILES/AIDA S460 R1 R2 R4 R5 R7 R9 R11 R13 R14 R16 R20 R21 R23 R24 R25 R26 R27 R30 R31 R32 R33 R34 R35 R36 R38 R40 R45 R46 R48 R50 R51 R52 /lustre/gamma/DESPEC_S496_FILES/AIDA S496 R1 R2 R3 R4 R5 R6 R7 R9 R11 R13 R14 R15 R16 R17 R20 R21 R22 R23 R25 R27 R28 R29 R30 R33 R34 R35 R36 R38 R39 R40 R42 R43 R46 R50 R56 R57 R58
372	Mon Jun 21 10:21:50 2021	NH	Monday 21 June
11.21 - DAQ continues OK (no storage mode) 11.21 - System checks ok except Base Current Difference aida01 fault 0x8944 : 0x8963 : 31 aida02 fault 0x9b36 : 0x9b55 : 31 aida03 fault 0xf26b : 0xf28a : 31 aida04 fault 0x57a0 : 0x57bf : 31 aida05 fault 0x4d3c : 0x4d95 : 89 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x3 : 3 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x9560 : 38240 aida13 fault 0x0 : 0x15b9 : 5561 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 33 12 14 2 4 4 2 2 2 4 6 : 37636 aida02 : 40 7 15 1 3 4 3 2 2 4 6 : 37800 aida03 : 39 7 9 6 5 3 1 3 2 4 6 : 37860 aida04 : 27 7 5 3 3 4 2 3 2 4 6 : 37908 aida05 : 33 5 5 3 6 4 3 3 2 4 6 : 38364 aida06 : 45 12 3 4 4 2 2 3 4 3 6 : 37828 aida07 : 31 11 5 4 4 2 3 2 2 4 6 : 37540 aida08 : 31 6 8 5 3 4 3 2 2 4 6 : 37772 aida09 : 45 4 9 3 2 3 3 2 2 4 6 : 37572 aida10 : 37 11 10 2 3 4 3 3 3 3 6 : 37260 aida11 : 40 9 11 3 3 4 3 3 1 4 6 : 37304 aida12 : 44 11 6 5 3 4 3 3 1 4 6 : 37320 aida13 : 41 11 9 3 5 3 3 3 1 4 6 : 37292 aida14 : 31 12 4 6 5 3 3 3 3 3 6 : 37276 aida15 : 17 9 7 1 2 2 3 2 1 4 7 : 40348 aida16 : 41 12 5 4 5 3 3 3 1 4 6 : 37268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 11.23 - figures Grafana temp & leakage - attachment 1 Weather cooler this week in Germany, expect S4 to cool down eventually Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 aida05 looks very noisy (?) ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 Looks some new time seq error since Thursday Experiment is runnign in S4 so no access possible this week Data is being archived to lustre (GSI PB file storage)
Attachment 1: firefox_2021-06-21_11-27-00.png

Attachment 2: AnyDesk_2021-06-21_11-24-28.png

Attachment 3: AnyDesk_2021-06-21_11-25-44.png

Attachment 4: AnyDesk_2021-06-21_11-26-07.png

Attachment 5: AnyDesk_2021-06-21_11-24-15.png

Attachment 6: AnyDesk_2021-06-21_11-24-43.png

Attachment 7: AnyDesk_2021-06-21_11-25-04.png

Attachment 8: AnyDesk_2021-06-21_11-25-15.png

371	Thu Jun 17 11:14:04 2021	TD	Thursday 17 June
12.15 DAQ continues OK - no storage mode 12.15 System checks OK except Base Current Difference aida01 fault 0x8944 : 0x8963 : 31 aida02 fault 0x9b36 : 0x9b55 : 31 aida03 fault 0xf26b : 0xf28a : 31 aida04 fault 0x57a0 : 0x57bf : 31 aida05 fault 0x4d3c : 0x4d55 : 25 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x3e03 : 15875 aida13 fault 0x0 : 0x32d6 : 13014 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 31 11 9 4 4 4 2 2 2 4 6 : 37604 aida02 : 30 10 10 3 3 4 3 2 2 4 6 : 37768 aida03 : 35 11 8 6 5 3 2 2 3 3 6 : 36580 aida04 : 33 9 4 4 3 4 2 3 3 3 6 : 36940 aida05 : 41 9 7 3 4 4 2 2 2 4 6 : 37564 aida06 : 38 6 5 3 2 3 2 3 4 3 6 : 37752 aida07 : 25 12 6 5 4 2 3 2 2 4 6 : 37572 aida08 : 32 7 10 5 4 3 3 2 2 4 6 : 37752 aida09 : 41 8 6 4 2 3 3 2 2 4 6 : 37572 aida10 : 31 10 11 5 5 3 3 3 3 3 6 : 37340 aida11 : 29 7 12 3 4 4 3 3 1 4 6 : 37324 aida12 : 42 8 7 5 3 4 3 3 1 4 6 : 37304 aida13 : 43 10 4 5 5 3 3 3 1 4 6 : 37276 aida14 : 40 15 4 5 4 4 3 3 3 3 6 : 37368 aida15 : 13 5 8 2 1 2 3 2 1 4 7 : 40284 aida16 : 39 9 6 6 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 12.15 Hot HEC channels - ASIC check Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-17_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1008.png

Attachment 8: 1009.png

370	Wed Jun 16 18:05:03 2021	TD	Wednesday 16 June
19.04 All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8960 : 28 aida02 fault 0x9b36 : 0x9b52 : 28 aida03 fault 0xf26b : 0xf287 : 28 aida04 fault 0x57a0 : 0x57bc : 28 aida05 fault 0x4d3c : 0x4d55 : 25 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x3e03 : 15875 aida13 fault 0x0 : 0x32d4 : 13012 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 53 10 4 4 4 4 2 2 3 3 6 : 36580 aida02 : 28 7 10 4 3 4 3 2 2 4 6 : 37768 aida03 : 41 10 7 6 5 3 1 3 3 3 6 : 36836 aida04 : 45 7 2 4 3 4 2 3 3 3 6 : 36940 aida05 : 35 8 8 4 4 4 2 2 2 4 6 : 37580 aida06 : 45 6 6 3 3 3 2 3 4 3 6 : 37860 aida07 : 37 10 6 4 5 2 3 2 2 4 6 : 37636 aida08 : 32 5 10 4 4 4 3 2 2 4 6 : 37832 aida09 : 24 5 8 7 1 3 3 2 2 4 6 : 37544 aida10 : 29 9 12 5 4 3 3 3 3 3 6 : 37276 aida11 : 33 7 7 6 3 4 3 3 1 4 6 : 37292 aida12 : 36 11 6 6 3 4 3 3 1 4 6 : 37320 aida13 : 37 7 6 6 5 3 3 3 1 4 6 : 37292 aida14 : 25 9 7 6 5 3 3 3 3 3 6 : 37276 aida15 : 3 3 4 3 0 3 3 2 1 4 7 : 40260 aida16 : 35 7 6 6 4 4 3 3 1 4 6 : 37348 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19.07 Hot HEC channels - ASIC check Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 significant increases of leakage current to c. 12uA - temperature c. 30 deg C Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-16_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 1005.png

Attachment 7: 1006.png

369	Tue Jun 15 12:35:48 2021	TD	Tuesday 15 June
13.35 DAQ continues OK - no storage mode All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 37 10 8 5 4 4 2 2 2 4 6 : 37636 aida02 : 37 10 10 3 4 4 3 2 2 4 6 : 37860 aida03 : 41 8 7 7 5 3 1 3 2 4 6 : 37876 aida04 : 33 9 4 4 2 5 2 3 3 3 6 : 37004 aida05 : 37 9 9 3 5 4 2 2 2 4 6 : 37644 aida06 : 39 9 6 5 4 2 2 3 4 3 6 : 37860 aida07 : 29 12 6 5 4 2 3 2 2 4 6 : 37588 aida08 : 25 7 9 5 4 3 3 2 2 4 6 : 37708 aida09 : 34 5 9 6 2 3 3 2 2 4 6 : 37632 aida10 : 34 12 11 5 4 3 3 3 3 3 6 : 37304 aida11 : 38 8 7 6 3 4 3 3 1 4 6 : 37320 aida12 : 34 10 8 6 3 4 3 3 1 4 6 : 37336 aida13 : 33 11 5 6 4 4 3 3 1 4 6 : 37356 aida14 : 31 10 5 6 5 3 3 3 3 3 6 : 37276 aida15 : 14 6 7 4 1 2 3 2 1 4 7 : 40344 aida16 : 33 8 6 7 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 13.40 All histograms, stats & merger stats zero'd
Attachment 1: Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-15_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1002.png

Attachment 8: 1003.png

368	Tue Jun 15 12:33:46 2021	TD	Monday 14 June

367	Sun Jun 13 17:29:04 2021	TD	Sunday 13 June
18.15 DAQ continues file S496/R32_946 Hot HEC channels multiple FEE64s from c. 06.00 this morning ASIC check fixes Out of disk space on /TapeData filesystem - switch to no storage mode System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 26 16 4 6 4 4 2 2 2 4 6 : 37608 aida02 : 29 11 9 3 3 4 3 2 2 4 6 : 37756 aida03 : 32 9 6 8 5 3 1 3 3 3 6 : 36840 aida04 : 32 8 3 4 3 4 2 3 3 3 6 : 36912 aida05 : 36 8 6 4 5 4 2 2 2 4 6 : 37616 aida06 : 25 13 4 3 2 3 2 3 4 3 6 : 37740 aida07 : 28 13 4 5 5 2 3 2 2 4 6 : 37624 aida08 : 29 5 9 4 3 4 3 2 2 4 6 : 37740 aida09 : 41 8 4 7 1 3 3 2 2 4 6 : 37572 aida10 : 45 11 7 5 5 3 3 3 3 3 6 : 37340 aida11 : 33 13 5 6 3 4 3 3 1 4 6 : 37308 aida12 : 39 8 6 6 4 4 3 3 1 4 6 : 37372 aida13 : 28 14 3 6 4 4 3 3 1 4 6 : 37328 aida14 : 30 11 3 6 4 4 3 3 3 3 6 : 37312 aida15 : 3 3 4 3 0 3 3 2 1 4 7 : 40260 aida16 : 28 9 5 6 5 3 3 3 1 4 6 : 37256 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png

Attachment 2: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png

Attachment 5: Screenshot_2021-06-13_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1000.png

Attachment 8: 1001.png

366	Sat Jun 12 14:53:56 2021	TD	aida15 - system console output - 15.33 Sat 12 June reboot
04/11:03:40\|get_ASICBlk : Blk=1 : bytes = 196096 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500240 : 1 => 0F07C000 04/11:03:41\|Last Data : Databuffer 273286648 : 0 => 80500240 : 1 => 0AFF8000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:42\|Machine check in kernel mode. 12/15:30:08\|I-Cache Parity Error 12/15:30:08\|Oops: Machine check, sig: 7 [#1] 12/15:30:08\|PREEMPT Xilinx Virtex440 12/15:30:08\|Modules linked in: aidamem xdriver xh_spidev_register 12/15:30:08\|NIP: c0055c54 LR: c00558fc CTR: c000ae5c 12/15:30:08\|REGS: c6ff9f10 TRAP: 0214 Not tainted (2.6.31) 12/15:30:08\|MSR: 00021000 <ME,CE> CR: 82000028 XER: 20000000 12/15:30:08\|TASK = c6ac44e0[375] 'AidaExecV9' THREAD: c6338000 12/15:30:08\|GPR00: 00000000 c6339cd0 c6ac44e0 ffffffff ffe79600 ffffffff ffe79600 ffffffff 12/15:30:08\|GPR08: 00000000 0000001f ffffffe0 00000000 00000000 1005520c 1dcd6500 00000000 12/15:30:08\|GPR16: 00000000 1009df98 100f0000 c0390000 000007a1 7fe9e580 ffffffff ffffffff 12/15:30:08\|GPR24: c0390000 00000001 00000000 001366ca c036f0d0 ffffffe1 ffffffff ffec9936 12/15:30:08\|NIP [c0055c54] update_wall_time+0x40c/0xd3c 12/15:30:09\|LR [c00558fc] update_wall_time+0xb4/0xd3c 12/15:30:09\|Call Trace: 12/15:30:09\|[c6339cd0] [c0055f44] update_wall_time+0x6fc/0xd3c (unreliable) 12/15:30:09\|[c6339d90] [c0040b84] do_timer+0x38/0x4c 12/15:30:09\|[c6339da0] [c005a088] tick_periodic+0xbc/0xe8 12/15:30:09\|[c6339db0] [c005a0d4] tick_handle_periodic+0x20/0x120 12/15:30:09\|[c6339df0] [c000af70] timer_interrupt+0xa4/0x10c 12/15:30:09\|[c6339e10] [c000e9c4] ret_from_except+0x0/0x18 12/15:30:09\|[c6339ed0] [d10751dc] aidamem_read+0x28/0x108 [aidamem] 12/15:30:09\|[c6339ef0] [c0097e80] vfs_read+0xb4/0x188 12/15:30:09\|[c6339f10] [c00982f8] sys_read+0x4c/0x90 12/15:30:09\|[c6339f40] [c000e364] ret_from_syscall+0x0/0x3c 12/15:30:09\|Instruction dump: 12/15:30:09\|39600000 354bffe0 418008b0 7cc35030 38800000 2f8a0000 419c08bc 7ffe5030 12/15:30:09\|3aa00000 93c10048 92a1004c 83c10048 <83e1004c> 7d095830 4800056c 419e06bc 12/15:30:09\|Kernel panic - not syncing: Fatal exception in interrupt 12/15:30:09\|Call Trace: 12/15:30:09\|Rebooting in 180 seconds.. ISOL Version 1.00 Date 9th January 2017 Flash base address=FC000000 Set Flash to ASync Mode XST_SUCCESS\| Finished copying zImage to RAM 12/15:33:10\| Found 0 errors checking kernel image 12/15:33:11\|VHDL version number 0X03350706 Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012 Starting LMK 3200 setup 12/15:33:11\| Setting LMK03200 to standard clock settings -- External Clock 23Nov15 .... SPI Base Address=0x81400000 clk_control_reg=0x4 12/15:33:11\|Next step is SPIconfig Control 32(0x81400000)=0x180 SlaveSel(0x81400000)=0x3 Ctrl(0x81400000)=0xE6 Ctrl(0x81400000)=0x86 12/15:33:11\|SPIconfig done now to set up the LMK3200 registers 12/15:33:12\|LMK #0 : regInit[0]=0x80000000 12/15:33:12\|LMK #0 : regInit[1]=0x10070600 12/15:33:12\|LMK #0 : regInit[2]=0x60601 12/15:33:12\|LMK #0 : regInit[3]=0x60602 12/15:33:12\|LMK #0 : regInit[4]=0x60603 12/15:33:12\|LMK #0 : regInit[5]=0x70624 12/15:33:12\|LMK #0 : regInit[6]=0x70605 12/15:33:12\|LMK #0 : regInit[7]=0x70606 12/15:33:12\|LMK #0 : regInit[8]=0x70627 12/15:33:12\|LMK #0 : regInit[9]=0x10000908 12/15:33:12\|LMK #0 : regInit[10]=0xA0022A09 12/15:33:12\|LMK #0 : regInit[11]=0x82800B 12/15:33:12\|LMK #0 : regInit[12]=0x28C800D 12/15:33:12\|LMK #0 : regInit[13]=0x830020E 12/15:33:12\|LMK #0 : regInit[14]=0xC800180F Calibrate completed at 941 counts Setting Clock Control =0x0000000B, to set GOE and sync bit Ctrl @ SPIstop (0x81400000)=0x186 Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004 12/15:33:12\| Finished Clock setup LMK03200 completed LMK 3200 setup Loaded all four ASICs with default settings Setting the ADCs into calibration mode 12/15:33:12\| Control 32(0x81400400)=0x180 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86 Init : Config of AD9252 SPI ok 12/15:33:13\| Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised ADCs calibrated 12/15:33:13\| Control 32(0x81400400)=0x186 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86Config of AD9252 SPI ok 12/15:33:13\| Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot... 12/15:33:13\| zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0) Allocating 0x3b78cc bytes for kernel ... gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes 12/15:33:16\| Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M Finalizing device tree... flat tree at 0xbd1300 Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 12/15:33:23\|Using Xilinx Virtex440 machine description 12/15:33:23\|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011 12/15:33:23\|Zone PFN ranges: 12/15:33:23\| DMA 0x00000000 -> 0x00007000 12/15:33:23\| Normal 0x00007000 -> 0x00007000 12/15:33:23\|Movable zone start PFN for each node 12/15:33:23\|early_node_map[1] active PFN ranges 12/15:33:23\| 0: 0x00000000 -> 0x00007000 12/15:33:23\|MMU: Allocated 1088 bytes of context maps for 255 contexts 12/15:33:23\|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448 12/15:33:24\|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M 12/15:33:24\|PID hash table entries: 512 (order: 9, 2048 bytes) 12/15:33:24\|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) 12/15:33:24\|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) 12/15:33:24\|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init) 12/15:33:24\|Kernel virtual memory layout: 12/15:33:24\| * 0xffffe000..0xfffff000 : fixmap 12/15:33:24\| * 0xfde00000..0xfe000000 : consistent mem 12/15:33:24\| * 0xfde00000..0xfde00000 : early ioremap 12/15:33:24\| * 0xd1000000..0xfde00000 : vmalloc & ioremap 12/15:33:24\|NR_IRQS:512 12/15:33:24\|clocksource: timebase mult[a00000] shift[22] registered 12/15:33:24\|Console: colour dummy device 80x25 12/15:33:24\|Mount-cache hash table entries: 512 12/15:33:24\|NET: Registered protocol family 16 12/15:33:24\|PCI: Probing PCI hardware 12/15:33:24\|bio: create slab <bio-0> at 0 12/15:33:24\|NET: Registered protocol family 2 12/15:33:24\|IP route cache hash table entries: 1024 (order: 0, 4096 bytes) 12/15:33:24\|TCP established hash table entries: 4096 (order: 3, 32768 bytes) 12/15:33:25\|TCP bind hash table entries: 4096 (order: 2, 16384 bytes) 12/15:33:25\|TCP: Hash tables configured (established 4096 bind 4096) 12/15:33:25\|TCP reno registered 12/15:33:25\|NET: Registered protocol family 1 12/15:33:25\|ROMFS MTD (C) 2007 Red Hat, Inc. 12/15:33:25\|msgmni has been set to 214 12/15:33:25\|io scheduler noop registered 12/15:33:25\|io scheduler anticipatory registered 12/15:33:25\|io scheduler deadline registered 12/15:33:25\|io scheduler cfq registered (default) 12/15:33:25\|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 12/15:33:25\|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550 12/15:33:25\|console [ttyS0] enabled 12/15:33:25\|brd: module loaded 12/15:33:25\|loop: module loaded 12/15:33:25\|Device Tree Probing 'ethernet' 12/15:33:25\|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 12/15:33:25\|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode. 12/15:33:25\|XLlTemac: DCR address: 0x80 12/15:33:25\|XLlTemac: buffer descriptor size: 32768 (0x8000) 12/15:33:25\|XLlTemac: Allocating DMA descriptors with kmalloc 12/15:33:25\|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000 12/15:33:26\|XTemac: PHY detected at address 7. 12/15:33:26\|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17 12/15:33:26\|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\|Using buffer write method 12/15:33:26\|cfi_cmdset_0001: Erase suspend on write enabled 12/15:33:26\|cmdlinepart partition parsing not available 12/15:33:26\|RedBoot partition parsing not available
365	Sat Jun 12 13:56:00 2021	TD	Saturday 12 June
14.11 DAQ continues file S496/R32_697 High data rate from c. 06.00 this morning ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8948 : 4 aida02 fault 0x9b36 : 0x9b3a : 4 aida03 fault 0xf26b : 0xf26f : 4 aida04 fault 0x57a0 : 0x57a4 : 4 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x21be : 8638 aida13 fault 0x0 : 0x2294 : 8852 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 33 10 9 2 5 4 2 2 2 4 6 : 37604 aida02 : 46 12 7 2 4 4 3 2 2 4 6 : 37832 aida03 : 34 10 9 5 4 3 1 3 3 3 6 : 36744 aida04 : 35 12 10 1 3 4 2 3 3 3 6 : 36972 aida05 : 37 9 8 3 4 4 2 2 2 4 6 : 37564 aida06 : 34 10 5 5 4 2 2 3 2 4 6 : 37832 aida07 : 31 11 1 7 4 2 3 2 2 4 6 : 37572 aida08 : 40 7 4 5 3 4 3 2 2 4 6 : 37752 aida09 : 39 9 8 4 1 3 3 2 2 4 6 : 37540 aida10 : 43 12 8 4 4 3 3 3 1 4 6 : 37260 aida11 : 34 14 10 5 3 4 3 3 1 4 6 : 37368 aida12 : 46 12 3 4 4 4 3 3 1 4 6 : 37320 aida13 : 41 11 5 4 5 3 3 3 1 4 6 : 37260 aida14 : 39 10 2 6 4 4 3 3 1 4 6 : 37324 aida15 : 19 12 4 3 2 2 3 2 2 4 6 : 37324 aida16 : 45 10 6 4 5 3 3 3 1 4 6 : 37284 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 15.05 analysis of file S496/R32_698 - attachment 9 max dead time 0.05% aida04 15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365 successful DAQ stop aida15 DAQ reset, setup, go successful DAQ go
Attachment 1: Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-12_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 998.png

Attachment 8: 999.png

Attachment 9: R32_698
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 44970784 ( 15685.0 Hz) Other data format: 216949222 ( 75668.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 62 ( 0.0 Hz) RESUME: 62 ( 0.0 Hz) SYNC100: 42680 ( 14.9 Hz) WR48-63: 42680 ( 14.9 Hz) FEE64 disc: 985985 ( 343.9 Hz) MBS info: 215877753 ( 75294.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2867.115 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2299.418 2 0.673 0.000 3 1.368 0.000 4 0.000 0.000 5 0.000 2495.101 6 0.000 0.000 7 0.000 2566.695 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 467 2 0 0 0 0 1 1 0 0 0 0 1 72420 30 0 0 0 0 15 15 0 0 0 0 2 935 107990737 0 0 33 33 19385 19385 0 107951901 0 0 3 38882631 108955999 0 0 29 29 22052 22052 985985 107925852 0 0 4 496 0 0 0 0 0 0 0 0 0 0 0 5 3224207 1336 0 0 0 0 668 668 0 0 0 0 6 1520 0 0 0 0 0 0 0 0 0 0 0 7 2784837 1118 0 0 0 0 559 559 0 0 0 0 8 473 0 0 0 0 0 0 0 0 0 0 0 9 173 0 0 0 0 0 0 0 0 0 0 0 10 554 0 0 0 0 0 0 0 0 0 0 0 11 317 0 0 0 0 0 0 0 0 0 0 0 12 752 0 0 0 0 0 0 0 0 0 0 0 13 204 0 0 0 0 0 0 0 0 0 0 0 14 302 0 0 0 0 0 0 0 0 0 0 0 15 496 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.945s ( 1068.615 blocks/s, 66.788 Mb/s)
363	Thu Jun 10 11:32:27 2021	TD	Thursday 10 June
12.26 DAQ continues file S496/R32_428 High data rate from c. 06.00 this morning ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 11 6 3 4 4 2 2 2 4 6 : 37540 aida02 : 30 8 7 3 4 4 3 2 2 4 6 : 37768 aida03 : 38 8 13 3 4 3 1 3 3 3 6 : 36744 aida04 : 37 9 7 2 3 4 2 3 3 3 6 : 36940 aida05 : 29 5 9 3 3 3 1 3 2 4 6 : 37580 aida06 : 38 6 2 3 3 3 2 3 2 4 6 : 37768 aida07 : 31 9 2 7 4 2 3 2 2 4 6 : 37572 aida08 : 34 6 3 5 4 4 3 2 2 4 6 : 37768 aida09 : 29 2 12 5 1 3 3 2 2 4 6 : 37540 aida10 : 24 8 15 3 5 3 3 3 1 4 6 : 37296 aida11 : 35 4 15 3 3 4 3 3 1 4 6 : 37308 aida12 : 36 5 3 5 6 4 3 3 1 4 6 : 37384 aida13 : 31 10 4 3 6 4 3 3 1 4 6 : 37356 aida14 : 37 7 1 4 5 4 3 3 1 4 6 : 37276 aida15 : 34 10 6 3 4 1 2 4 2 4 6 : 38168 aida16 : 31 9 3 3 5 4 3 3 1 4 6 : 37268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 202 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 15.07 analysis of file S496/R32_431 - attachment 9 max dead time aida04 0.33%
Attachment 1: Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-10_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 996.png

Attachment 8: 997.png

Attachment 9: R32_431
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 54180182 ( 19758.0 Hz) Other data format: 207739819 ( 75756.7 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 105 ( 0.0 Hz) RESUME: 105 ( 0.0 Hz) SYNC100: 42213 ( 15.4 Hz) WR48-63: 42213 ( 15.4 Hz) FEE64 disc: 1515114 ( 552.5 Hz) MBS info: 206140069 ( 75173.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 6 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2742.196 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2662.854 2 0.384 0.000 3 9.009 0.000 4 0.000 0.000 5 0.000 1805.119 6 0.000 0.000 7 0.000 1797.783 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 559 0 0 0 0 0 0 0 0 0 0 0 1 413404 177 0 0 0 0 84 84 9 0 0 0 2 1153 103268358 0 0 19 19 17963 17963 0 103232394 0 0 3 42054837 104466440 0 0 86 86 21757 21757 1515079 102907675 0 0 4 515 0 0 0 0 0 0 0 0 0 0 0 5 5561752 2359 0 0 0 0 1175 1175 9 0 0 0 6 1738 2 0 0 0 0 1 1 0 0 0 0 7 6142236 2472 0 0 0 0 1232 1232 8 0 0 0 8 565 1 0 0 0 0 0 0 1 0 0 2 9 190 0 0 0 0 0 0 0 0 0 0 0 10 694 0 0 0 0 0 0 0 0 0 0 0 11 417 6 0 0 0 0 0 0 6 0 0 4 12 967 4 0 0 0 0 1 1 2 0 0 0 13 244 0 0 0 0 0 0 0 0 0 0 0 14 285 0 0 0 0 0 0 0 0 0 0 0 15 626 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.160s ( 1097.388 blocks/s, 68.587 Mb/s)
362	Wed Jun 9 19:17:21 2021	TD	FEE64 system console logs

Attachment 1: logs-090621.tar.gz
361	Wed Jun 9 11:27:47 2021	TD	Wednesday 9 June
12.30 DAQ continues OK file S496/R32_373 System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 13 5 12 7 2 4 2 2 1 4 7 : 40572 aida02 : 11 6 8 6 1 3 3 2 1 4 7 : 40540 aida03 : 26 5 11 5 1 2 2 3 2 3 7 : 39712 aida04 : 18 7 8 3 2 3 2 3 2 3 7 : 39776 aida05 : 11 5 9 4 1 2 2 3 1 4 7 : 40612 aida06 : 17 3 6 5 5 2 1 3 1 4 7 : 40604 aida07 : 9 3 4 9 1 3 3 2 1 4 7 : 40540 aida08 : 11 8 5 5 3 3 3 2 1 4 7 : 40604 aida09 : 13 5 8 5 2 3 3 2 1 4 7 : 40572 aida10 : 13 8 13 5 1 2 2 2 1 4 7 : 40228 aida11 : 11 5 14 6 3 3 3 3 2 3 7 : 40244 aida12 : 10 9 5 5 3 3 2 2 1 4 7 : 40352 aida13 : 15 7 7 4 3 4 3 3 2 3 7 : 40228 aida14 : 9 6 7 6 3 4 3 3 2 3 7 : 40260 aida15 : 14 3 8 5 1 2 2 4 1 4 7 : 41136 aida16 : 15 4 9 4 4 4 3 3 2 3 7 : 40300 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 1.8.L specttra - attachments 1 & 2 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3 Lost activity monitor - attachment 4 1.8.W spectra - 20us FSR - attachments 5 & 6 ADC data items - attachment 7 FEE64 temperatures OK - attachment 8 DSSSD bias & leakage currents OK - attachment 9 Merger/Tape Server/Merger statistics - attachment 10 no merger errors reported since previous restart 16.28 analysis of file S496/R32_372 - attachment 11 max deadtime 0.07% aida04
Attachment 1: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png

Attachment 2: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png

Attachment 3: Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png

Attachment 5: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png

Attachment 6: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png

Attachment 7: Screenshot_2021-06-09_Statistics_aidas-gsi.png

Attachment 8: Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png

Attachment 9: 991.png

Attachment 10: 992.png

Attachment 11: R32_372
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 65571173 ( 25420.1 Hz) Other data format: 196348830 ( 76118.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 52 ( 0.0 Hz) RESUME: 52 ( 0.0 Hz) SYNC100: 41608 ( 16.1 Hz) WR48-63: 41608 ( 16.1 Hz) FEE64 disc: 2050316 ( 794.8 Hz) MBS info: 194215194 ( 75291.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 284 ( 0.1 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2579.504 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.015 1853.986 2 0.548 0.000 3 1.886 0.000 4 0.000 0.000 5 0.000 1457.725 6 0.000 0.000 7 0.000 1391.289 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 1323.904 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 836.274 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 6664 532 0 0 0 0 1 1 530 0 0 3 1 4648488 85466 0 0 1 1 947 947 83570 0 0 92 2 5675 97165745 0 0 27 27 16487 16487 7 97132710 0 2 3 47009038 98898439 0 0 24 24 21346 21346 1773239 97082460 0 42 4 3608 4 0 0 0 0 0 0 4 0 0 1 5 6949043 95991 0 0 0 0 1383 1383 93225 0 0 60 6 5001 30 0 0 0 0 1 1 4 24 0 0 7 6897601 102341 0 0 0 0 1436 1436 99469 0 0 74 8 1797 1 0 0 0 0 0 0 1 0 0 1 9 2911 4 0 0 0 0 1 1 2 0 0 3 10 8722 137 0 0 0 0 1 1 135 0 0 0 11 5985 36 0 0 0 0 2 2 32 0 0 4 12 7238 34 0 0 0 0 1 1 32 0 0 0 13 7345 52 0 0 0 0 0 0 52 0 0 0 14 4898 4 0 0 0 0 0 0 4 0 0 1 15 7159 14 0 0 0 0 2 2 10 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 30.055s ( 1064.726 blocks/s, 66.545 Mb/s)
360	Tue Jun 8 10:51:55 2021	TD	Tuesday 8 June
11.52 DAQ continues file S496/R32_340 High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning 11.53 ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 3 12 8 3 3 2 2 1 4 7 : 40508 aida02 : 5 5 10 6 1 3 3 2 1 4 7 : 40540 aida03 : 25 7 12 5 1 2 2 3 2 3 7 : 39740 aida04 : 19 6 11 5 1 3 2 3 2 3 7 : 39820 aida05 : 9 6 7 5 0 2 2 3 1 4 7 : 40548 aida06 : 10 8 7 4 3 3 1 3 1 4 7 : 40600 aida07 : 9 3 4 8 1 3 3 2 1 4 7 : 40508 aida08 : 12 5 8 5 3 3 3 2 1 4 7 : 40632 aida09 : 15 4 8 5 1 3 3 2 1 4 7 : 40508 aida10 : 23 5 12 5 1 2 2 2 1 4 7 : 40228 aida11 : 15 9 11 6 2 4 3 3 2 3 7 : 40308 aida12 : 14 9 4 5 3 3 2 2 1 4 7 : 40352 aida13 : 15 9 6 3 4 4 3 3 2 3 7 : 40260 aida14 : 11 5 7 6 4 4 3 3 2 3 7 : 40324 aida15 : 16 8 7 4 1 2 2 4 1 4 7 : 41136 aida16 : 15 8 7 4 3 4 3 3 2 3 7 : 40236 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 16.28 analysis of file S496/R32_339 - attachment 9 max deadtime 0.24% aida04
Attachment 1: Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-08_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 990.png

Attachment 8: 991.png

Attachment 9: R32_339
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 52833088 ( 19139.6 Hz) Other data format: 209086916 ( 75745.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 101 ( 0.0 Hz) RESUME: 101 ( 0.0 Hz) SYNC100: 42281 ( 15.3 Hz) WR48-63: 42281 ( 15.3 Hz) FEE64 disc: 1254898 ( 454.6 Hz) MBS info: 207747254 ( 75259.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2760.405 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2672.516 2 0.610 0.000 3 6.711 0.000 4 0.000 0.000 5 0.000 2111.274 6 0.000 0.000 7 0.000 1910.870 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 589 0 0 0 0 0 0 0 0 0 0 0 1 221336 98 0 0 0 0 49 49 0 0 0 0 2 1118 104025022 0 0 30 30 18193 18193 0 103988576 0 0 3 42352603 105057740 0 0 71 71 22011 22011 1254898 103758678 0 0 4 548 0 0 0 0 0 0 0 0 0 0 0 5 4562177 1856 0 0 0 0 928 928 0 0 0 0 6 1810 0 0 0 0 0 0 0 0 0 0 0 7 5688898 2196 0 0 0 0 1098 1098 0 0 0 0 8 593 0 0 0 0 0 0 0 0 0 0 0 9 215 2 0 0 0 0 1 1 0 0 0 0 10 675 2 0 0 0 0 1 1 0 0 0 0 11 387 0 0 0 0 0 0 0 0 0 0 0 12 964 0 0 0 0 0 0 0 0 0 0 0 13 267 0 0 0 0 0 0 0 0 0 0 0 14 369 0 0 0 0 0 0 0 0 0 0 0 15 539 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.656s ( 1116.685 blocks/s, 69.793 Mb/s)
359	Mon Jun 7 10:33:48 2021	TD	Monday 7 June
11.30 DAQ continues OK - file S496/R32_144 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 5 11 6 3 4 2 2 1 4 7 : 40572 aida02 : 1 3 6 7 2 3 3 2 1 4 7 : 40540 aida03 : 21 7 7 8 2 4 1 3 2 3 7 : 39804 aida04 : 13 7 5 6 1 3 2 3 2 3 7 : 39740 aida05 : 2 8 6 5 1 2 2 3 1 4 7 : 40584 aida06 : 10 4 5 5 4 3 1 3 1 4 7 : 40632 aida07 : 9 3 4 7 3 3 3 2 1 4 7 : 40604 aida08 : 11 2 6 4 4 3 3 2 1 4 7 : 40604 aida09 : 10 2 5 4 2 2 2 3 1 4 7 : 40584 aida10 : 14 5 8 6 2 1 1 3 1 4 7 : 40352 aida11 : 8 6 9 5 4 4 3 3 2 3 7 : 40320 aida12 : 10 5 3 7 3 3 2 2 1 4 7 : 40352 aida13 : 7 5 4 6 5 4 3 3 2 3 7 : 40324 aida14 : 7 5 4 8 4 4 3 3 2 3 7 : 40324 aida15 : 8 4 7 4 2 2 2 4 1 4 7 : 41136 aida16 : 13 5 7 6 3 4 3 3 2 3 7 : 40268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 11.35 analysis of file S496/R32_143 - attachment 9 max deadtime 0.29% aida03
Attachment 1: Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-07_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 986.png

Attachment 8: 987.png

Attachment 9: R32_143
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 57707988 ( 21988.9 Hz) Other data format: 204212013 ( 77812.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 85 ( 0.0 Hz) RESUME: 85 ( 0.0 Hz) SYNC100: 41772 ( 15.9 Hz) WR48-63: 41772 ( 15.9 Hz) FEE64 disc: 1109863 ( 422.9 Hz) MBS info: 203018436 ( 77357.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2624.418 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2432.488 2 0.665 0.000 3 7.525 0.000 4 0.000 0.000 5 0.000 1464.970 6 0.000 0.000 7 0.000 1288.051 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 112 0 0 0 0 0 0 0 0 0 0 0 1 1940572 788 0 0 0 0 394 394 0 0 0 0 2 442 101676817 0 0 30 30 17294 17294 0 101642169 0 0 3 40857770 102528388 0 0 55 55 21074 21074 1109863 101376267 0 0 4 129 0 0 0 0 0 0 0 0 0 0 0 5 7006930 2878 0 0 0 0 1439 1439 0 0 0 0 6 548 0 0 0 0 0 0 0 0 0 0 0 7 7898329 3142 0 0 0 0 1571 1571 0 0 0 0 8 285 0 0 0 0 0 0 0 0 0 0 0 9 71 0 0 0 0 0 0 0 0 0 0 0 10 1135 0 0 0 0 0 0 0 0 0 0 0 11 122 0 0 0 0 0 0 0 0 0 0 0 12 679 0 0 0 0 0 0 0 0 0 0 0 13 92 0 0 0 0 0 0 0 0 0 0 0 14 666 0 0 0 0 0 0 0 0 0 0 0 15 106 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.551s ( 1082.882 blocks/s, 67.680 Mb/s)
358	Sun Jun 6 14:15:05 2021	TD	Sunday 6 June
14.15 DAQ continues file S496/R32_117 Hot HEC channels multiple FEE64s from c. 06.00 this morning ASIC check fixes System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 12 12 3 3 4 2 2 1 4 7 : 40572 aida02 : 7 8 10 5 2 3 3 2 1 4 7 : 40604 aida03 : 27 10 4 8 1 4 1 3 2 3 7 : 39740 aida04 : 18 7 4 5 1 3 2 3 2 3 7 : 39712 aida05 : 15 13 8 4 1 3 1 3 1 4 7 : 40548 aida06 : 14 8 8 6 3 2 1 3 1 4 7 : 40568 aida07 : 11 4 5 6 2 3 3 2 1 4 7 : 40540 aida08 : 15 6 5 3 4 3 3 2 1 4 7 : 40604 aida09 : 18 4 4 3 3 2 2 3 1 4 7 : 40648 aida10 : 12 6 8 6 1 1 1 3 1 4 7 : 40288 aida11 : 16 8 6 6 4 4 3 3 2 3 7 : 40352 aida12 : 10 7 4 8 3 4 3 3 2 3 7 : 40288 aida13 : 15 7 5 4 5 4 3 3 2 3 7 : 40324 aida14 : 9 4 6 7 3 4 3 3 2 3 7 : 40260 aida15 : 12 4 8 3 2 2 2 4 1 4 7 : 41136 aida16 : 11 8 6 6 3 4 3 3 2 3 7 : 40268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 16.28 analysis of file S496/R32_118 - attachment 9 max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-06_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 984.png

Attachment 8: 985.png

Attachment 9: R32_118
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 48860593 ( 17844.1 Hz) Other data format: 213059412 ( 77810.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 101 ( 0.0 Hz) RESUME: 101 ( 0.0 Hz) SYNC100: 42200 ( 15.4 Hz) WR48-63: 42200 ( 15.4 Hz) FEE64 disc: 1112345 ( 406.2 Hz) MBS info: 211862465 ( 77373.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2738.188 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2613.952 2 0.575 0.000 3 7.745 0.000 4 0.000 0.000 5 0.000 2154.815 6 0.000 0.000 7 0.000 1921.162 8 0.000 777.447 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 155 0 0 0 0 0 0 0 0 0 0 0 1 66625 28 0 0 0 0 14 14 0 0 0 0 2 415 106108157 0 0 26 26 18899 18899 0 106070307 0 0 3 39298810 106947309 0 0 75 75 21328 21328 1112345 105792158 0 0 4 171 0 0 0 0 0 0 0 0 0 0 0 5 4255149 1684 0 0 0 0 842 842 0 0 0 0 6 596 0 0 0 0 0 0 0 0 0 0 0 7 5234928 2226 0 0 0 0 1113 1113 0 0 0 0 8 493 4 0 0 0 0 2 2 0 0 0 0 9 92 0 0 0 0 0 0 0 0 0 0 0 10 1212 2 0 0 0 0 1 1 0 0 0 0 11 160 0 0 0 0 0 0 0 0 0 0 0 12 781 0 0 0 0 0 0 0 0 0 0 0 13 89 0 0 0 0 0 0 0 0 0 0 0 14 724 2 0 0 0 0 1 1 0 0 0 0 15 193 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.738s ( 1113.497 blocks/s, 69.594 Mb/s)
357	Sat Jun 5 11:20:15 2021	TD	Saturday 5 June
12.25 DAQ continues OK - file S496/R32_48 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 10 6 1 2 4 4 2 2 1 4 7 : 40360 aida02 : 4 3 3 3 3 4 2 2 1 4 7 : 40312 aida03 : 13 5 4 4 2 4 1 3 2 3 7 : 39580 aida04 : 16 5 3 3 2 4 1 3 2 3 7 : 39544 aida05 : 8 6 1 2 3 3 1 3 1 4 7 : 40416 aida06 : 4 3 2 1 3 3 1 3 1 4 7 : 40360 aida07 : 6 6 2 4 4 2 3 2 1 4 7 : 40424 aida08 : 6 4 1 1 4 3 3 2 1 4 7 : 40424 aida09 : 16 1 3 1 0 2 2 3 1 4 7 : 40344 aida10 : 2 5 1 4 1 3 2 4 2 3 7 : 40064 aida11 : 14 3 2 1 4 4 3 3 2 3 7 : 40080 aida12 : 11 2 2 4 4 4 3 3 2 3 7 : 40156 aida13 : 11 3 2 2 3 4 3 3 2 3 7 : 40036 aida14 : 6 5 0 4 3 4 3 3 2 3 7 : 40064 aida15 : 11 6 3 1 3 1 2 4 1 4 7 : 40940 aida16 : 12 5 0 4 4 4 3 3 2 3 7 : 40152 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 1.8.W spectra - 20us FSR - attachments 2 & 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart analysis of file S496/R32_47 - attachment 8 max deadtime 0.35% aida04 13.56 Lost activity monitor - see attachment 9
Attachment 1: Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-06-05_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 982.png

Attachment 7: 983.png

Attachment 8: R32_47
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 47883982 ( 17391.8 Hz) Other data format: 214036019 ( 77739.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 184 ( 0.1 Hz) RESUME: 184 ( 0.1 Hz) SYNC100: 42255 ( 15.3 Hz) WR48-63: 42255 ( 15.3 Hz) FEE64 disc: 1053753 ( 382.7 Hz) MBS info: 212897388 ( 77325.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2753.252 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2696.425 2 0.825 0.000 3 9.735 0.000 4 0.000 0.000 5 0.000 2175.120 6 0.000 0.000 7 0.000 2003.563 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 187 0 0 0 0 0 0 0 0 0 0 0 1 86597 40 0 0 0 0 20 20 0 0 0 0 2 381 106659018 0 0 37 37 18746 18746 0 106621452 0 0 3 38584230 107373193 0 0 147 147 21605 21605 1053753 106275936 0 0 4 143 0 0 0 0 0 0 0 0 0 0 0 5 4208337 1756 0 0 0 0 878 878 0 0 0 0 6 545 0 0 0 0 0 0 0 0 0 0 0 7 5000124 2012 0 0 0 0 1006 1006 0 0 0 0 8 478 0 0 0 0 0 0 0 0 0 0 0 9 96 0 0 0 0 0 0 0 0 0 0 0 10 1149 0 0 0 0 0 0 0 0 0 0 0 11 117 0 0 0 0 0 0 0 0 0 0 0 12 706 0 0 0 0 0 0 0 0 0 0 0 13 105 0 0 0 0 0 0 0 0 0 0 0 14 636 0 0 0 0 0 0 0 0 0 0 0 15 151 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.996s ( 1066.806 blocks/s, 66.675 Mb/s)
Attachment 9: Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png

356	Fri Jun 4 09:47:44 2021	TD	aida15 - system console output - 00.02 Friday 4 June reboot
03/23:58:55\|kernel BUG at mm/slab.c:2974! 03/23:58:55\|Oops: Exception in kernel mode, sig: 5 [#1] 03/23:58:55\|PREEMPT Xilinx Virtex440 03/23:58:55\|Modules linked in: aidamem xdriver xh_spidev_register 03/23:58:55\|NIP: c009211c LR: c00920b0 CTR: c0038e70 03/23:58:55\|REGS: c631fc60 TRAP: 0700 Not tainted (2.6.31) 03/23:58:55\|MSR: 00021000 <ME,CE> CR: 22022048 XER: 00000000 03/23:58:56\|TASK = c627a0c0[375] 'AidaExecV9' THREAD: c631e000 03/23:58:56\|GPR00: 00000001 c631fd10 c627a0c0 c680daf0 00000020 00000001 c6940020 00000000 03/23:58:56\|GPR08: fffffff6 00010002 c680e400 c631e000 22008022 1005520c c03a0000 00000020 03/23:58:56\|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 03/23:58:56\|GPR24: 00100100 c631e000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400 03/23:58:56\|NIP [c009211c] cache_alloc_refill+0x130/0x608 03/23:58:56\|LR [c00920b0] cache_alloc_refill+0xc4/0x608 03/23:58:56\|Call Trace: 03/23:58:56\|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable) 03/23:58:56\|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc 03/23:58:56\|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8 03/23:58:56\|[c631fdb0] [c0042938] __send_signal+0x78/0x260 03/23:58:56\|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c 03/23:58:56\|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c 03/23:58:56\|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30 03/23:58:56\|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240 03/23:58:56\|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c 03/23:58:56\|[c631feb0] [c0040bf4] update_process_times+0x30/0x70 03/23:58:57\|[c631fed0] [c005a000] tick_periodic+0x34/0xe8 03/23:58:57\|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120 03/23:58:57\|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c 03/23:58:57\|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18 03/23:58:57\|Instruction dump: 03/23:58:57\|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 03/23:58:57\|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 03/23:58:57\|Kernel panic - not syncing: Fatal exception in interrupt 03/23:58:57\|Call Trace: 03/23:58:57\|[c631fab0] [c0005de8] show_stack+0x44/0x16c (unreliable) 03/23:58:57\|[c631faf0] [c00345bc] panic+0x94/0x168 03/23:58:57\|[c631fb40] [c000bd44] die+0x178/0x18c 03/23:58:57\|[c631fb60] [c000c000] _exception+0x164/0x1b4 03/23:58:57\|[c631fc50] [c000e978] ret_from_except_full+0x0/0x4c 03/23:58:57\|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 03/23:58:57\|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc 03/23:58:57\|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8 03/23:58:57\|[c631fdb0] [c0042938] __send_signal+0x78/0x260 03/23:58:57\|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c 03/23:58:58\|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c 03/23:58:58\|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30 03/23:58:58\|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240 03/23:58:58\|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c 03/23:58:58\|[c631feb0] [c0040bf4] update_process_times+0x30/0x70 03/23:58:58\|[c631fed0] [c005a000] tick_periodic+0x34/0xe8 03/23:58:58\|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120 03/23:58:58\|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c 03/23:58:58\|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18 03/23:58:58\|Rebooting in 180 seconds.. ISOL Version 1.00 Date 9th January 2017 Flash base address=FC000000 Set Flash to ASync Mode XST_SUCCESS\| Finished copying zImage to RAM 04/00:01:59\| Found 0 errors checking kernel image 04/00:02:00\|VHDL version number 0X03350706 Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012 Starting LMK 3200 setup 04/00:02:00\| Setting LMK03200 to standard clock settings -- External Clock 23Nov15 .... SPI Base Address=0x81400000 clk_control_reg=0x4 04/00:02:00\|Next step is SPIconfig Control 32(0x81400000)=0x180 SlaveSel(0x81400000)=0x3 Ctrl(0x81400000)=0xE6 Ctrl(0x81400000)=0x86 04/00:02:00\|SPIconfig done now to set up the LMK3200 registers 04/00:02:00\|LMK #0 : regInit[0]=0x80000000 04/00:02:00\|LMK #0 : regInit[1]=0x10070600 04/00:02:00\|LMK #0 : regInit[2]=0x60601 04/00:02:00\|LMK #0 : regInit[3]=0x60602 04/00:02:00\|LMK #0 : regInit[4]=0x60603 04/00:02:00\|LMK #0 : regInit[5]=0x70624 04/00:02:00\|LMK #0 : regInit[6]=0x70605 04/00:02:00\|LMK #0 : regInit[7]=0x70606 04/00:02:00\|LMK #0 : regInit[8]=0x70627 04/00:02:00\|LMK #0 : regInit[9]=0x10000908 04/00:02:00\|LMK #0 : regInit[10]=0xA0022A09 04/00:02:00\|LMK #0 : regInit[11]=0x82800B 04/00:02:01\|LMK #0 : regInit[12]=0x28C800D 04/00:02:01\|LMK #0 : regInit[13]=0x830020E 04/00:02:01\|LMK #0 : regInit[14]=0xC800180F Calibrate completed at 941 counts Setting Clock Control =0x0000000B, to set GOE and sync bit Ctrl @ SPIstop (0x81400000)=0x186 Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004 04/00:02:01\| Finished Clock setup LMK03200 completed LMK 3200 setup Loaded all four ASICs with default settings Setting the ADCs into calibration mode 04/00:02:01\| Control 32(0x81400400)=0x180 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86 Init : Config of AD9252 SPI ok 04/00:02:01\| Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised Cal DCMs not locked ADC calibrate failed Jumping to kernel simpleboot... 04/00:02:01\| zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0) Allocating 0x3b78cc bytes for kernel ... gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes 04/00:02:05\| Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M Finalizing device tree... flat tree at 0xbd1300 Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 04/00:02:11\|Using Xilinx Virtex440 machine description 04/00:02:12\|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011 04/00:02:12\|Zone PFN ranges: 04/00:02:12\| DMA 0x00000000 -> 0x00007000 04/00:02:12\| Normal 0x00007000 -> 0x00007000 04/00:02:12\|Movable zone start PFN for each node 04/00:02:12\|early_node_map[1] active PFN ranges 04/00:02:12\| 0: 0x00000000 -> 0x00007000 04/00:02:12\|MMU: Allocated 1088 bytes of context maps for 255 contexts 04/00:02:12\|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448 04/00:02:12\|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M 04/00:02:12\|PID hash table entries: 512 (order: 9, 2048 bytes) 04/00:02:12\|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) 04/00:02:12\|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) 04/00:02:12\|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init) 04/00:02:12\|Kernel virtual memory layout: 04/00:02:12\| * 0xffffe000..0xfffff000 : fixmap 04/00:02:12\| * 0xfde00000..0xfe000000 : consistent mem 04/00:02:12\| * 0xfde00000..0xfde00000 : early ioremap 04/00:02:13\| * 0xd1000000..0xfde00000 : vmalloc & ioremap 04/00:02:13\|NR_IRQS:512 04/00:02:13\|clocksource: timebase mult[a00000] shift[22] registered 04/00:02:13\|Console: colour dummy device 80x25 04/00:02:13\|Mount-cache hash table entries: 512 04/00:02:13\|NET: Registered protocol family 16 04/00:02:13\|PCI: Probing PCI hardware 04/00:02:13\|bio: create slab <bio-0> at 0 04/00:02:13\|NET: Registered protocol family 2 04/00:02:13\|IP route cache hash table entries: 1024 (order: 0, 4096 bytes) 04/00:02:13\|TCP established hash table entries: 4096 (order: 3, 32768 bytes) 04/00:02:13\|TCP bind hash table entries: 4096 (order: 2, 16384 bytes) 04/00:02:13\|TCP: Hash tables configured (established 4096 bind 4096) 04/00:02:13\|TCP reno registered 04/00:02:13\|NET: Registered protocol family 1 04/00:02:13\|ROMFS MTD (C) 2007 Red Hat, Inc. 04/00:02:13\|msgmni has been set to 214 04/00:02:13\|io scheduler noop registered 04/00:02:13\|io scheduler anticipatory registered 04/00:02:13\|io scheduler deadline registered 04/00:02:13\|io scheduler cfq registered (default) 04/00:02:13\|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 04/00:02:13\|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550 04/00:02:14\|console [ttyS0] enabled 04/00:02:14\|brd: module loaded 04/00:02:14\|loop: module loaded 04/00:02:14\|Device Tree Probing 'ethernet' 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode. 04/00:02:14\|XLlTemac: DCR address: 0x80 04/00:02:14\|XLlTemac: buffer descriptor size: 32768 (0x8000) 04/00:02:14\|XLlTemac: Allocating DMA descriptors with kmalloc 04/00:02:14\|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000 04/00:02:14\|XTemac: PHY detected at address 7. 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17 04/00:02:14\|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\|Using buffer write method 04/00:02:15\|cfi_cmdset_0001: Erase suspend on write enabled 04/00:02:15\|cmdlinepart partition parsing not available 04/00:02:15\|RedBoot partition parsing not available 04/00:02:15\|Creating 5 MTD partitions on "fc000000.flash": 04/00:02:15\|0x000000000000-0x000000500000 : "golden_firmware" 04/00:02:15\|0x000000500000-0x000000800000 : "golden_kernel" 04/00:02:15\|0x000000800000-0x000000d00000 : "user_firmware" 04/00:02:15\|0x000000d00000-0x000000fe0000 : "user_kernel" 04/00:02:15\|0x000000fe0000-0x000001000000 : "env_variables" 04/00:02:15\|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20 04/00:02:15\|SPI: XIlinx spi: bus number now 32766 04/00:02:15\|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21 04/00:02:15\|SPI: XIlinx spi: bus number now 32765 04/00:02:15\|mice: PS/2 mouse device common for all mice 04/00:02:15\|Device Tree Probing 'i2c' 04/00:02:15\| #0 at 0x81600000 mapped to 0xD1030000, irq=22 04/00:02:15\|at24 0-0050: 1024 byte 24c08 EEPROM (writable) 04/00:02:15\|TCP cubic registered 04/00:02:15\|NET: Registered protocol family 17 04/00:02:15\|RPC: Registered udp transport module. 04/00:02:15\|RPC: Registered tcp transport module. 04/00:02:16\|eth0: XLlTemac: Options: 0x3fa 04/00:02:16\|eth0: XLlTemac: allocating interrupt 19 for dma mode tx. 04/00:02:16\|eth0: XLlTemac: allocating interrupt 18 for dma mode rx. 04/00:02:16\|eth0: XLlTemac: speed set to 1000Mb/s 04/00:02:18\|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4 04/00:02:18\|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254 04/00:02:18\|Sending DHCP requests ., OK 04/00:02:19\|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.15 04/00:02:20\|IP-Config: Complete: 04/00:02:20\| device=eth0, addr=192.168.11.15, mask=255.255.255.0, gw=255.255.255.255, 04/00:02:20\| host=aida15, domain=dl.ac.uk, nis-domain=nuclear.physics, 04/00:02:20\| bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15 04/00:02:20\|Looking up port of RPC 100003/2 on 192.168.11.99 04/00:02:20\|Looking up port of RPC 100005/1 on 192.168.11.99 04/00:02:20\|VFS: Mounted root (nfs filesystem) on device 0:12. 04/00:02:20\|Freeing unused kernel memory: 168k init INIT: version 2.86 booting 04/00:02:20\|Starting sysinit... 04/00:02:21\| Welcome to DENX & STFC Daresbury Embedded Linux Environment 04/00:02:22\| Press 'I' to enter interactive startup. 04/00:02:22\|Setting clock (utc): Thu Jun 3 23:02:22 BST 2021 [ OK ] 04/00:02:22\|Building the cache [ OK ] 04/00:02:22\|Setting hostname aida15: [ OK ] 04/00:02:23\|Mounting local filesystems: [ OK ] 04/00:02:25\|Enabling /etc/fstab swaps: [ OK ] 04/00:02:27\|Finishing sysinit... INIT: Entering runlevel: 3 04/00:02:30\|Entering non-interactive startup 04/00:02:30\|FATAL: Module ipv6 not found. 04/00:02:31\|Bringing up loopback interface: [ OK ] 04/00:02:33\|FATAL: Module ipv6 not found. 04/00:02:34\|Starting system logger: [ OK ] 04/00:02:34\|Starting kernel logger: [ OK ] 04/00:02:35\|Starting rpcbind: [ OK ] 04/00:02:36\|Mounting NFS filesystems: [ OK ] 04/00:02:37\|Mounting other filesystems: [ OK ] 04/00:02:37\|Starting xinetd: [ OK ] 04/00:02:38\|Starting midas: Starting MIDAS Data Acquisition for aida15 04/00:02:38\|xaida: device parameters: base=0x81000000 size=0x200000 04/00:02:44\|Trying to free nonexistent resource <0000000081000000-00000000811fffff> 04/00:02:44\|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000 04/00:02:44\|xaida: driver assigned major number 254 04/00:02:44\|Trying to free nonexistent resource <0000000007000000-0000000007ffffff> 04/00:02:50\|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000 04/00:02:50\|AIDAMEM: aidamem: driver assigned major number 253 04/00:02:50\|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31 04/00:02:56\|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx 04/00:02:57\|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx 04/00:02:57\|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin 04/00:02:57\|Computer Name = aida15; Temp Directory = /tmp/tcl361 04/00:03:01\|package limit is not available: can't find package limit 04/00:03:04\|Running with default file descriptor limit 04/00:03:04\|package setuid is not available: can't find package setuid 04/00:03:05\|Running as user 0 group 0 04/00:03:05\| 04/00:03:06\|AIDA Data Acquisition Program Release 9_10.Apr 3 2019_11:34:31 starting 04/00:03:06\| 04/00:03:06\|Built without pthreads 04/00:03:06\| 04/00:03:06\|Creating NetVars 04/00:03:06\|Output buffer length = 65xaida: open: 04/00:03:06\|504; format option = 4AIDAMEM: aidamem_open: 04/00:03:06\|; transfer option = 3 04/00:03:06\|EB transfer option = 3 04/00:03:06\|NetVars created and initialised 04/00:03:06\|Statistics thread starting 04/00:03:06\|Statistics thread created 04/00:03:06\|Stat/Rate creation thread starting 04/00:03:06\|Data Acquisition task has PID 375 04/00:03:06\|Stat/Rate creation thread created 04/00:03:06\|Hit/Rate creation thread starting 04/00:03:06\|Hit/Rate creation thread created 04/00:03:07\|AIDA Heartbeat thread starting 04/00:03:07\|Heartbeat thread created 04/00:03:07\|Installing signal handlers 04/00:03:07\|Done 04/00:03:07\|ModuleNum = 0 04/00:03:07\|Aida Initialise complete. AidaExecV9_10: Build Apr 3 2019_11:34:31. HDL version : 03350706 04/00:03:07\|Spectra table initialised 04/00:03:07\|AIDA Data Acquisition now all ready to start 04/00:03:07\|SIGBUS, SIGSEGV and SIGPIPE traps setup 04/00:03:07\|/debug user "debug" password " bjkhcfyuqis" 04/00:03:14\|httpd started on port 8015 04/00:03:14\| 04/00:03:14\|Cannot use /MIDAS/config/TclHttpd/aida15@8015/startup.tcl 04/00:03:14\|Custom startup from /MIDAS/config/TclHttpd/aida15/startup.tcl 04/00:03:14\|XAIDA Access package 1.0 04/00:03:16\|/XAIDAAccessServer 04/00:03:16\|XAD9252 Access package 1.0 04/00:03:16\|/XAD9252AccessServer 04/00:03:16\|/DataBaseAccessServer 04/00:03:16\|[ OK ] 04/00:03:16\|/NetVarService 04/00:03:17\|/SigTaskService 04/00:03:17\|Loaded MemSasAccess 04/00:03:17\|/SpectrumService 04/00:03:17\|loading tcl/AIDARunControl.tcl for namespace :: 04/00:03:17\| 04/00:03:17\|DENX ELDK version 4.2 build 2008-04-01 04/00:03:17\|Linux 2.6.31 on a ppc 04/00:03:17\| 04/00:03:17\|aida15 login: /DataAcquisitionControlServer 04/00:03:17\|DefineMessage unknown 04/00:03:17\|Run Control Server Implementation for AIDA 04/00:03:17\|RunControlServer loaded 04/00:03:17\|loading Html/RunControl/implementation.tcl 04/00:03:18\|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory 04/00:03:18\|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist 04/00:03:18\|Created UI registers 04/00:03:18\|RunControl loaded 04/00:03:19\|loading Html/AIDA/RunControl/implementation.tcl for namespace :: 04/00:03:19\|AIDA RunControl loaded 04/00:03:19\|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl 04/00:03:19\|do_GetState returned z=0 and 8 04/09:31:05\|do_GetState returned z=0 and 8 04/09:31:17\|do_GetState returned z=0 and 8 04/09:31:30\|do_GetState returned z=0 and 8 04/09:31:43\|do_GetState returned z=0 and 8 04/09:31:55\|do_GetState returned z=0 and 8 04/09:32:07\|do_GetState returned z=0 and 8 04/09:32:19\|do_GetState returned z=0 and 8 04/09:32:32\|do_GetState returned z=0 and 8 04/09:32:45\|do_GetState returned z=0 and 8 04/09:32:57\|do_GetState returned z=0 and 8 04/09:33:10\|do_GetState returned z=0 and 8 04/09:33:22\|do_GetState returned z=0 and 8 04/09:33:35\|do_GetState returned z=0 and 8 04/09:33:47\|do_GetState returned z=0 and 8 04/09:34:00\|do_GetState returned z=0 and 8 04/09:34:13\|do_GetState returned z=0 and 8 04/09:34:25\|do_GetState returned z=0 and 8 04/09:34:38\|do_GetState returned z=0 and 8 04/09:34:50\|do_GetState returned z=0 and 8 04/09:35:03\|do_GetState returned z=0 and 8 04/09:35:16\|do_GetState returned z=0 and 8 04/09:35:28\|do_GetState returned z=0 and 8 04/09:35:41\|do_GetState returned z=0 and 8 04/09:35:53\|do_GetState returned z=0 and 8 04/09:36:06\|do_GetState returned z=0 and 8 04/09:36:18\|do_GetState returned z=0 and 8 04/09:36:31\|do_GetState returned z=0 and 8 04/09:36:43\|do_GetState returned z=0 and 8 04/09:36:56\|do_GetState returned z=0 and 8 04/09:37:09\|do_GetState returned z=0 and 8 04/09:37:21\|do_GetState returned z=0 and 8 04/09:37:34\|do_GetState returned z=0 and 8 04/09:37:46\|do_GetState returned z=0 and 8 04/09:37:59\|do_GetState returned z=0 and 8 04/09:38:12\|do_GetState returned z=0 and 8 04/09:38:24\|do_GetState returned z=0 and 8 04/09:38:37\|do_GetState returned z=0 and 8 04/09:38:49\|do_GetState returned z=0 and 8 04/09:39:02\|do_GetState returned z=0 and 8 04/09:39:15\|do_GetState returned z=0 and 8 04/09:39:27\|do_GetState returned z=0 and 8 04/09:39:40\|do_GetState returned z=0 and 8 04/09:39:52\|do_GetState returned z=0 and 8 04/09:40:05\|do_GetState returned z=0 and 8 04/09:40:18\|do_GetState returned z=0 and 8 04/09:40:30\|do_GetState returned z=0 and 8 04/09:40:43\|do_GetState returned z=0 and 8 04/09:40:55\|do_GetState returned z=0 and 8 04/09:41:08\|do_GetState returned z=0 and 8 04/09:41:21\|do_GetState returned z=0 and 8 04/09:41:33\|do_GetState returned z=0 and 8 04/09:41:46\|do_GetState returned z=0 and 8 04/09:41:58\|do_GetState returned z=0 and 8 04/09:42:11\|do_GetState returned z=0 and 8 04/09:42:23\|do_GetState returned z=0 and 8 04/09:42:36\|do_GetState returned z=0 and 8 04/09:42:48\|do_GetState returned z=0 and 8 04/09:43:01\|do_GetState returned z=0 and 8 04/09:43:14\|do_GetState returned z=0 and 8 04/09:43:26\|do_GetState returned z=0 and 8 04/09:43:39\|do_GetState returned z=0 and 8 04/09:43:51\|do_GetState returned z=0 and 8 04/09:44:04\|do_GetState returned z=0 and 8 04/09:44:17\|do_GetState returned z=0 and 8 04/09:44:29\|do_GetState returned z=0 and 8 04/09:44:42\|do_GetState returned z=0 and 8 04/09:44:54\|do_GetState returned z=0 and 8 04/09:45:07\|do_GetState returned z=0 and 8 04/09:45:19\|do_GetState returned z=0 and 8 04/09:45:32\|do_GetState returned z=0 and 8 04/09:45:44\|do_GetState returned z=0 and 8 04/09:45:57\|do_GetState returned z=0 and 8 04/09:46:09\|do_GetState returned z=0 and 8 04/09:46:22\|do_GetState returned z=0 and 8 04/09:46:34\|do_GetState returned z=0 and 8 04/09:46:47\|do_GetState returned z=0 and 8 04/09:47:00\|do_GetState returned z=0 and 8 04/09:47:12\|do_GetState returned z=0 and 8 04/09:47:25\|do_GetState returned z=0 and 8 04/09:47:37\|do_GetState returned z=0 and 8 04/09:47:50\|do_GetState returned z=0 and 8 04/09:48:03\|do_GetState returned z=0 and 8 04/09:48:15\|do_GetState returned z=0 and 8 04/09:48:28\|do_GetState returned z=0 and 8 04/09:48:40\|do_GetState returned z=0 and 8 04/09:48:53\|do_GetState returned z=0 and 8 04/09:49:05\|do_GetState returned z=0 and 8 04/09:49:18\|do_GetState returned z=0 and 8 04/09:49:31\|do_GetState returned z=0 and 8 04/09:49:43\|do_GetState returned z=0 and 8 04/09:49:56\|do_GetState returned z=0 and 8 04/09:50:08\|do_GetState returned z=0 and 8 04/09:50:21\|do_GetState returned z=0 and 8 04/09:50:33\|do_GetState returned z=0 and 8 04/09:50:46\|do_GetState returned z=0 and 8 04/09:50:58\|do_GetState returned z=0 and 8 04/09:51:11\|do_GetState returned z=0 and 8 04/09:51:23\|do_GetState returned z=0 and 8 04/09:51:36\|do_GetState returned z=0 and 8 04/09:51:49\|do_GetState returned z=0 and 8 04/09:52:01\|do_GetState returned z=0 and 8 04/09:52:14\|do_GetState returned z=0 and 8 04/09:52:26\|do_GetState returned z=0 and 8 04/09:52:39\|do_GetState returned z=0 and 8 04/09:52:51\|do_GetState returned z=0 and 8 04/09:53:04\|do_GetState returned z=0 and 8 04/09:53:16\|do_GetState returned z=0 and 8 04/09:53:29\|do_GetState returned z=0 and 8 04/09:53:42\|do_GetState returned z=0 and 8 04/09:53:54\|do_GetState returned z=0 and 8 04/09:54:07\|do_GetState returned z=0 and 8 04/09:54:19\|do_GetState returned z=0 and 8 04/09:54:32\|do_GetState returned z=0 and 8 04/09:54:44\|do_GetState returned z=0 and 8 04/09:54:57\|do_GetState returned z=0 and 8 04/09:55:09\|do_GetState returned z=0 and 8 04/09:55:22\|do_GetState returned z=0 and 8 04/09:55:34\|do_GetState returned z=0 and 8 04/09:55:47\|do_GetState returned z=0 and 8 04/09:56:00\|do_GetState returned z=0 and 8 04/09:56:12\|do_GetState returned z=0 and 8 04/09:56:25\|do_GetState returned z=0 and 8 04/09:56:37\|do_GetState returned z=0 and 8 04/09:56:50\|do_GetState returned z=0 and 8 04/09:57:02\|do_GetState returned z=0 and 8 04/09:57:15\|do_GetState returned z=0 and 8 04/09:57:27\|do_GetState returned z=0 and 8 04/09:57:40\|do_GetState returned z=0 and 8 04/09:57:53\|do_GetState returned z=0 and 8 04/09:58:05\|do_GetState returned z=0 and 8 04/09:58:18\|do_GetState returned z=0 and 8 04/09:58:30\|do_GetState returned z=0 and 8 04/09:58:43\|do_GetState returned z=0 and 8 04/09:58:56\|do_GetState returned z=0 and 8 04/09:59:08\|do_GetState returned z=0 and 8 04/09:59:21\|do_GetState returned z=0 and 8 04/09:59:33\|do_GetState returned z=0 and 8 04/09:59:46\|do_GetState returned z=0 and 8 04/09:59:58\|do_GetState returned z=0 and 8 04/10:00:11\|do_GetState returned z=0 and 8 04/10:00:23\|do_GetState returned z=0 and 8 04/10:00:36\|do_GetState returned z=0 and 8 04/10:00:48\|do_GetState returned z=0 and 8 04/10:01:01\|do_GetState returned z=0 and 8 04/10:01:13\|do_GetState returned z=0 and 8 04/10:01:26\|do_GetState returned z=0 and 8 04/10:01:38\|do_GetState returned z=0 and 8 04/10:01:51\|do_GetState returned z=0 and 8 04/10:02:04\|do_GetState returned z=0 and 8 04/10:02:16\|do_GetState returned z=0 and 8 04/10:02:29\|do_GetState returned z=0 and 8 04/10:02:41\|do_GetState returned z=0 and 8 04/10:02:54\|do_GetState returned z=0 and 8 04/10:03:06\|do_GetState returned z=0 and 8 04/10:03:19\|do_GetState returned z=0 and 8 04/10:03:31\|do_GetState returned z=0 and 8 04/10:03:44\|do_GetState returned z=0 and 8 04/10:03:57\|do_GetState returned z=0 and 8 04/10:04:09\|do_GetState returned z=0 and 8 04/10:04:22\|do_GetState returned z=0 and 8 04/10:04:34\|do_GetState returned z=0 and 8 04/10:04:47\|do_GetState returned z=0 and 8 04/10:04:59\|do_GetState returned z=0 and 8 04/10:05:12\|do_GetState returned z=0 and 8 04/10:05:25\|do_GetState returned z=0 and 8 04/10:05:37\|do_GetState returned z=0 and 8 04/10:05:50\|do_GetState returned z=0 and 8 04/10:06:02\|do_GetState returned z=0 and 8 04/10:06:15\|do_GetState returned z=0 and 8 04/10:06:27\|do_GetState returned z=0 and 8 04/10:06:40\|do_GetState returned z=0 and 8 04/10:06:52\|do_GetState returned z=0 and 8 04/10:07:05\|do_GetState returned z=0 and 8 04/10:07:18\|do_GetState returned z=0 and 8 04/10:07:30\|do_GetState returned z=0 and 8
355	Fri Jun 4 08:40:52 2021	TD	Friday 4 June
09.20 DAQ found stopped files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight aida15 rebooted at 00.02 regained control of DAQ and issued DAQ STOP by restarting NewMerger twice 11.00 DAQ reset and setup All system wide checks OK except Base Current Difference aida05 fault 0x4d3c : 0x4d3c : 0 aida05 : WR status 0x10 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 4 2 8 5 4 1 2 2 4 7 : 41352 aida02 : 0 3 3 6 6 4 1 2 2 4 7 : 41352 aida03 : 2 4 2 12 1 3 2 2 2 4 7 : 41352 aida04 : 2 2 1 7 4 3 2 2 2 4 7 : 41352 aida05 : 2 3 1 5 3 2 3 2 2 4 7 : 41360 aida06 : 2 4 2 6 4 3 2 2 2 4 7 : 41352 aida07 : 1 5 2 9 4 2 4 3 1 4 7 : 41324 aida08 : 0 3 1 7 4 3 4 3 1 4 7 : 41352 aida09 : 3 3 0 8 2 2 3 2 2 4 7 : 41380 aida10 : 1 2 2 11 3 2 3 3 3 3 7 : 41044 aida11 : 2 3 1 6 6 4 2 3 3 3 7 : 41072 aida12 : 4 2 1 8 3 3 3 3 1 4 7 : 41072 aida13 : 2 4 1 8 4 4 2 3 3 3 7 : 41016 aida14 : 1 4 1 9 4 4 2 3 3 3 7 : 41044 aida15 : 0 5 1 2 2 1 2 4 3 4 7 : 42872 aida16 : 1 3 2 9 4 4 4 4 2 3 7 : 41052 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 11.03 DAQ resumes file S496/R32 alpha background slow comparator 0x64 BNC PB-5 pulser OFF 11.09 1.8.W spectra - 20us FSR - attachments 1 & 2 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart 13.07 analysis of file S496/R32_16 - attachment 8 max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png

Attachment 2: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-04_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 980.png

Attachment 7: 981.png

Attachment 8: R32_16
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 49166189 ( 17465.0 Hz) Other data format: 212753815 ( 75575.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 137 ( 0.0 Hz) RESUME: 137 ( 0.0 Hz) SYNC100: 42486 ( 15.1 Hz) WR48-63: 42486 ( 15.1 Hz) FEE64 disc: 1037793 ( 368.6 Hz) MBS info: 211630776 ( 75176.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2815.130 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2759.088 2 0.741 0.000 3 4.946 0.000 4 0.000 0.000 5 0.000 2173.993 6 0.000 0.000 7 0.000 2104.432 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 184 0 0 0 0 0 0 0 0 0 0 0 1 115458 62 0 0 0 0 31 31 0 0 0 0 2 444 105932150 0 0 36 36 18754 18754 0 105894570 0 0 3 39683483 106817705 0 0 101 101 21752 21752 1037793 105736206 0 0 4 154 0 0 0 0 0 0 0 0 0 0 0 5 4413391 1852 0 0 0 0 926 926 0 0 0 0 6 586 0 0 0 0 0 0 0 0 0 0 0 7 4948958 2044 0 0 0 0 1022 1022 0 0 0 0 8 489 0 0 0 0 0 0 0 0 0 0 0 9 77 0 0 0 0 0 0 0 0 0 0 0 10 1172 0 0 0 0 0 0 0 0 0 0 0 11 163 2 0 0 0 0 1 1 0 0 0 0 12 754 0 0 0 0 0 0 0 0 0 0 0 13 94 0 0 0 0 0 0 0 0 0 0 0 14 605 0 0 0 0 0 0 0 0 0 0 0 15 177 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.027s ( 1102.409 blocks/s, 68.901 Mb/s)
354	Thu Jun 3 10:16:39 2021	TD	Thursday 3 June
11.19 Hot HEC channels multiple FEE64s ASIC check fixes System wide checks OK * except* Base Current Difference aida01 fault 0x8940 : 0x8941 : 1 aida02 fault 0x9b32 : 0x9b33 : 1 aida03 fault 0xf266 : 0xf267 : 1 aida04 fault 0x579b : 0x579c : 1 White Rabbit error counter test result: Passed 12, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 11 4 2 2 2 2 2 3 2 4 7 : 41516 aida02 : 10 7 2 0 1 2 2 3 2 4 7 : 41408 aida03 : 10 8 2 2 2 4 1 4 2 3 7 : 40008 aida04 : 5 4 4 2 2 3 1 4 2 3 7 : 39860 aida05 : 12 3 2 2 1 2 2 3 2 4 7 : 41448 aida06 : 15 9 2 2 3 2 1 3 2 4 7 : 41380 aida07 : 12 8 1 2 2 3 3 2 2 4 7 : 41408 aida08 : 9 6 3 1 2 3 3 2 2 4 7 : 41380 aida09 : 15 6 1 1 2 2 2 3 2 4 7 : 41500 aida10 : 15 6 3 2 1 3 3 2 1 4 7 : 40348 aida11 : 8 2 1 1 1 3 1 3 2 4 7 : 41248 aida12 : 18 3 2 1 3 3 2 2 2 4 7 : 41184 aida13 : 7 4 1 4 3 4 3 3 1 4 7 : 41100 aida14 : 18 6 1 4 3 4 2 2 1 4 7 : 40392 aida15 : 7 7 1 3 1 2 1 3 2 4 7 : 41220 aida16 : 4 4 2 3 3 3 2 4 1 4 7 : 41200 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Wed Jun 02 16:27:30 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 1.8.W spectra - 20us FSR - attachments 2 & 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart 11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36 13.09 analysis of file S496/R31_16 - attachment 8 max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-06-03_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 970.png

Attachment 7: 971.png

Attachment 8: R31_16
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 50135749 ( 17884.4 Hz) Other data format: 211784253 ( 75547.7 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 199 ( 0.1 Hz) RESUME: 199 ( 0.1 Hz) SYNC100: 42443 ( 15.1 Hz) WR48-63: 42443 ( 15.1 Hz) FEE64 disc: 1081490 ( 385.8 Hz) MBS info: 210617479 ( 75131.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 4 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2803.320 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2522.967 2 0.535 0.000 3 8.636 0.000 4 0.000 0.000 5 0.000 2119.634 6 0.000 0.000 7 0.000 2067.036 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 196 2 0 0 0 0 0 0 2 0 0 2 1 193203 74 0 0 0 0 37 37 0 0 0 0 2 471 105497692 0 0 26 26 18605 18605 0 105460430 0 0 3 40159698 106282506 0 0 173 173 21812 21812 1081487 105157049 0 0 4 128 0 0 0 0 0 0 0 0 0 0 0 5 4517663 1914 0 0 0 0 957 957 0 0 0 0 6 641 2 0 0 0 0 1 1 0 0 0 0 7 5260254 2060 0 0 0 0 1030 1030 0 0 0 0 8 463 0 0 0 0 0 0 0 0 0 0 0 9 104 0 0 0 0 0 0 0 0 0 0 0 10 1026 2 0 0 0 0 1 1 0 0 0 0 11 141 1 0 0 0 0 0 0 1 0 0 1 12 786 0 0 0 0 0 0 0 0 0 0 0 13 109 0 0 0 0 0 0 0 0 0 0 0 14 687 0 0 0 0 0 0 0 0 0 0 0 15 179 0 0 0 0 0 0 0 0 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.445s ( 1124.966 blocks/s, 70.310 Mb/s)
353	Wed Jun 2 14:53:05 2021	OH, TD	Merger/Tapeserver issues after reboot of aida-3
On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen. We were also unable to connected to MIDAS via ssh port forwarding. Since the restart we have been unable to write to file. The tapeserver will allocate/mount/open a file but nothing will be written to it. The merger will be receiving data but never shows as having any "Current links with data". /MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23 2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 The current disk structure is: [npg@aidas-gsi npg]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sdb 8:16 0 7.3T 0 disk ├─sdb1 8:17 0 200M 0 part ├─sdb2 8:18 0 500M 0 part └─sdb3 8:19 0 7.3T 0 part ├─vg_aidas2-lv_root (dm-2) 253:2 0 50G 0 lvm ├─vg_aidas2-lv_home (dm-3) 253:3 0 7.2T 0 lvm /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 └─vg_aidas2-lv_swap (dm-4) 253:4 0 7.8G 0 lvm sdc 8:32 0 7.3T 0 disk └─sdc1 8:33 0 7.3T 0 part /media/ThirdDrive sdd 8:48 0 7.3T 0 disk └─sdd1 8:49 0 7.3T 0 part /media/SecondDrive sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 200M 0 part /boot/efi ├─sda2 8:2 0 500M 0 part /boot └─sda3 8:3 0 465.1G 0 part ├─vg_aidasgsi-lv_root (dm-0) 253:0 0 50G 0 lvm / ├─vg_aidasgsi-lv_swap (dm-1) 253:1 0 7.8G 0 lvm [SWAP] └─vg_aidasgsi-lv_home (dm-5) 253:5 0 407.3G 0 lvm /home Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3 TapeServer config file - attachment 4 Merger options - attachment 5 Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png
Attachment 1: 210602_MergerTape_HTTPd_Startup
tidy up tclsh8.5_copy2: no process killed System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64 Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64 MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64 PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin Computer Name = aidas-gsi; Temp Directory = /tmp/tcl25624 package limit is not available: can't find package limit Running with default file descriptor limit package setuid is not available: can't find package setuid Could not change to user 50 group 50: not owner /debug user "debug" password "z5y30nlwy2e4" httpd started on port 8115 Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl /DataBaseAccessServer /NetVarService /SigTaskService Loaded MemSasAccess /SpectrumService /TapeServer loading tcl/NewMergerControl.tcl for namespace :: /DataAcquisitionControlServer DefineMessage unknown Run Control Server Implementation for MERGE RunControlServer loaded loading Html/RunControl/implementation.tcl /MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist Created UI registers RunControl loaded loading Html/NewMerger/RunControl/implementation.tcl for namespace :: ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist Created UI registers NewMerge Control loaded Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl Shared memory area located at 0x7f34d3028000 Tape Server comms table located at 0x7f34d3028000 MERGE command 30 setup MERGER Stop Merge Halt (1) Action has completed: MergeCommand 2 Halt Setup Merge Setup (1) Action has completed: MergeCommand 1 Setup MERGE command 32 go MERGER Go Merge Go (1) Action has completed: MergeCommand 3 Go link 0 GO (1) Action has completed: MergeCommand 22 {link 0 GO} 0 2 link 1 GO (1) Action has completed: MergeCommand 22 {link 1 GO} 1 2 link 2 GO (1) Action has completed: MergeCommand 22 {link 2 GO} 2 2 link 3 GO (1) Action has completed: MergeCommand 22 {link 3 GO} 3 2 link 4 GO (1) Action has completed: MergeCommand 22 {link 4 GO} 4 2 link 5 GO (1) Action has completed: MergeCommand 22 {link 5 GO} 5 2 link 6 GO (1) Action has completed: MergeCommand 22 {link 6 GO} 6 2 link 7 GO (1) Action has completed: MergeCommand 22 {link 7 GO} 7 2 link 8 GO (1) Action has completed: MergeCommand 22 {link 8 GO} 8 2 link 9 GO (1) Action has completed: MergeCommand 22 {link 9 GO} 9 2 link 10 GO (1) Action has completed: MergeCommand 22 {link 10 GO} 10 2 link 11 GO (1) Action has completed: MergeCommand 22 {link 11 GO} 11 2 link 12 GO (1) Action has completed: MergeCommand 22 {link 12 GO} 12 2 link 13 GO (1) Action has completed: MergeCommand 22 {link 13 GO} 13 2 link 14 GO (1) Action has completed: MergeCommand 22 {link 14 GO} 14 2 link 15 GO (1) Action has completed: MergeCommand 22 {link 15 GO} 15 2 Resume MERGER
Attachment 2: 210602_NewMerger_Startup
Tidy up MERGE: no process killed merge64.AD: no process killed Starting New Merger [1] 25659 MERGE Master (25659): Message logger not contacted. MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021 20:32:01 Creating NetVars Creating NetVars complete MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021 20:32:01 MERGE Master (25659): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0 MERGE Master (25659): File mapped object /SHM_11000 of size 536628 created MERGE Master (25659): Memory mapping 536628 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0a99b8000. MERGE Master (25659): File mapped object /SHM_11001 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0a39b6000. MERGE Master (25659): File mapped object /SHM_11002 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc09d9b4000. MERGE Master (25659): File mapped object /SHM_11003 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0979b2000. MERGE Master (25659): File mapped object /SHM_11004 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0919b0000. MERGE Master (25659): File mapped object /SHM_11005 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc08b9ae000. MERGE Master (25659): File mapped object /SHM_11006 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0859ac000. MERGE Master (25659): File mapped object /SHM_11007 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc07f9aa000. MERGE Master (25659): File mapped object /SHM_11008 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0799a8000. MERGE Master (25659): File mapped object /SHM_11009 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0739a6000. MERGE Master (25659): File mapped object /SHM_11010 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc06d9a4000. MERGE Master (25659): File mapped object /SHM_11011 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0679a2000. MERGE Master (25659): File mapped object /SHM_11012 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0619a0000. MERGE Master (25659): File mapped object /SHM_11013 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc05b99e000. MERGE Master (25659): File mapped object /SHM_11014 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc05599c000. MERGE Master (25659): File mapped object /SHM_11015 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc04f99a000. MERGE Master (25659): File mapped object /SHM_11016 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc049998000. MERGE Master (25659): Shared memory segment located at address 0x7fc049998000. MERGE Master (25659): Starting stats/rates task MERGE Master (25659): Stats/Rates task has pid 25661 MERGE Master (25659): Starting 16 link tasks MERGE Master (25659): Link task has pid 25662 MERGE Master (25659): Link task has pid 25663 MERGE Statistics/Rates (25661): Message logger not contacted. MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021 20:32:03 Creating NetVars Creating NetVars complete MERGE Master (25659): Link task has pid 25664 MERGE Data Link (25662): Message logger not contacted. MERGE Data Link (25662): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25662): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0 MERGE Data Link (25662): File mapped object /SHM_11000 accessed MERGE Data Link (25662): Memory mapping 536628 bytes MERGE Data Link (25662): Shared memory segment located at address 0x7f198ee7a000. MERGE Data Link (25662): Allocated 71720 bytes for local input buffer MERGE Data Link (25662): File mapped object /SHM_11001 accessed MERGE Data Link (25662): Memory mapping 100669480 bytes MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000. MERGE Master (25659): Link task has pid 25665 MERGE Data Link (25663): Message logger not contacted. MERGE Data Link (25663): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25663): Configuration: index = 1, SHM key=11000, TCP port base = 11002, module = 0, hardware = 0 MERGE Data Link (25663): File mapped object /SHM_11000 accessed MERGE Data Link (25663): Memory mapping 536628 bytes MERGE Data Link (25663): Shared memory segment located at address 0x7f3bee603000. MERGE Data Link (25663): Allocated 71720 bytes for local input buffer MERGE Data Link (25663): File mapped object /SHM_11002 accessed MERGE Data Link (25663): Memory mapping 100669480 bytes MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000. MERGE Master (25659): Link task has pid 25666 MERGE Data Link (25664): Message logger not contacted. MERGE Data Link (25665): Message logger not contacted. MERGE Data Link (25664): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 MERGE Data Link (25665): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars Creating NetVars complete MERGE Data Link (25665): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0 MERGE Data Link (25665): File mapped object /SHM_11000 accessed MERGE Data Link (25665): Memory mapping 536628 bytes MERGE Data Link (25665): Shared memory segment located at address 0x7f7418ae6000. MERGE Data Link (25665): Allocated 71720 bytes for local input buffer MERGE Data Link (25665): File mapped object /SHM_11004 accessed MERGE Data Link (25665): Memory mapping 100669480 bytes MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000. Creating NetVars complete MERGE Data Link (25664): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0 MERGE Data Link (25664): File mapped object /SHM_11000 accessed MERGE Data Link (25664): Memory mapping 536628 bytes MERGE Data Link (25664): Shared memory segment located at address 0x7f0da72c8000. MERGE Data Link (25664): Allocated 71720 bytes for local input buffer MERGE Data Link (25664): File mapped object /SHM_11003 accessed MERGE Data Link (25664): Memory mapping 100669480 bytes MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000. MERGE Master (25659): Link task has pid 25667 MERGE Master (25659): Link task has pid 25668 MERGE Data Link (25666): Message logger not contacted. MERGE Data Link (25666): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25666): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0 MERGE Data Link (25666): File mapped object /SHM_11000 accessed MERGE Data Link (25666): Memory mapping 536628 bytes MERGE Data Link (25666): Shared memory segment located at address 0x7fb7f4ec2000. MERGE Data Link (25666): Allocated 71720 bytes for local input buffer MERGE Data Link (25666): File mapped object /SHM_11005 accessed MERGE Data Link (25666): Memory mapping 100669480 bytes MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000. MERGE Master (25659): Link task has pid 25669 MERGE Data Link (25667): Message logger not contacted. MERGE Data Link (25667): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25667): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0 MERGE Data Link (25667): File mapped object /SHM_11000 accessed MERGE Data Link (25667): Memory mapping 536628 bytes MERGE Data Link (25667): Shared memory segment located at address 0x7f8b878c9000. MERGE Data Link (25667): Allocated 71720 bytes for local input buffer MERGE Data Link (25667): File mapped object /SHM_11006 accessed MERGE Data Link (25667): Memory mapping 100669480 bytes MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000. MERGE Master (25659): Link task has pid 25670 MERGE Data Link (25668): Message logger not contacted. MERGE Data Link (25668): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Master (25659): Link task has pid 25671 Creating NetVars complete MERGE Data Link (25668): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0 MERGE Data Link (25668): File mapped object /SHM_11000 accessed MERGE Data Link (25668): Memory mapping 536628 bytes MERGE Data Link (25668): Shared memory segment located at address 0x7f88c7c9a000. MERGE Data Link (25668): Allocated 71720 bytes for local input buffer MERGE Data Link (25668): File mapped object /SHM_11007 accessed MERGE Data Link (25668): Memory mapping 100669480 bytes MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000. MERGE Data Link (25669): Message logger not contacted. MERGE Data Link (25669): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Master (25659): Link task has pid 25672 Creating NetVars complete MERGE Data Link (25669): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0 MERGE Data Link (25669): File mapped object /SHM_11000 accessed MERGE Data Link (25669): Memory mapping 536628 bytes MERGE Data Link (25669): Shared memory segment located at address 0x7f87e7704000. MERGE Data Link (25669): Allocated 71720 bytes for local input buffer MERGE Data Link (25669): File mapped object /SHM_11008 accessed MERGE Data Link (25669): Memory mapping 100669480 bytes MERGE Data Link (25669): Shared memory segment located at address 0x7f87e1702000. MERGE Data Link (25671): Message logger not contacted. MERGE Data Link (25671): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25671): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0 MERGE Data Link (25671): File mapped object /SHM_11000 accessed MERGE Data Link (25671): Memory mapping 536628 bytes MERGE Data Link (25671): Shared memory segment located at address 0x7f38b4f59000. MERGE Data Link (25671): Allocated 71720 bytes for local input buffer MERGE Data Link (25671): File mapped object /SHM_11010 accessed MERGE Data Link (25671): Memory mapping 100669480 bytes MERGE Data Link (25671): Shared memory segment located at address 0x7f38aef57000. MERGE Data Link (25670): Message logger not contacted. MERGE Data Link (25670): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Master (25659): Link task has pid 25673 Creating NetVars complete MERGE Data Link (25670): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0 MERGE Data Link (25670): File mapped object /SHM_11000 accessed MERGE Data Link (25670): Memory mapping 536628 bytes MERGE Data Link (25672): Message logger not contacted. MERGE Data Link (25670): Shared memory segment located at address 0x7ff7744ce000. MERGE Data Link (25672): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Data Link (25670): Allocated 71720 bytes for local input buffer MERGE Data Link (25670): File mapped object /SHM_11009 accessed MERGE Data Link (25670): Memory mapping 100669480 bytes MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000. Creating NetVars complete MERGE Data Link (25672): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0 MERGE Data Link (25672): File mapped object /SHM_11000 accessed MERGE Data Link (25672): Memory mapping 536628 bytes MERGE Data Link (25672): Shared memory segment located at address 0x7f4bb287e000. MERGE Data Link (25672): Allocated 71720 bytes for local input buffer MERGE Data Link (25672): File mapped object /SHM_11011 accessed MERGE Data Link (25672): Memory mapping 100669480 bytes MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000. MERGE Master (25659): Link task has pid 25674 MERGE Data Link (25673): Message logger not contacted. MERGE Data Link (25673): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25673): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0 MERGE Data Link (25673): File mapped object /SHM_11000 accessed MERGE Data Link (25673): Memory mapping 536628 bytes MERGE Data Link (25673): Shared memory segment located at address 0x7fe3cd54a000. MERGE Data Link (25673): Allocated 71720 bytes for local input buffer MERGE Data Link (25673): File mapped object /SHM_11012 accessed MERGE Data Link (25673): Memory mapping 100669480 bytes MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000. MERGE Master (25659): Link task has pid 25675 MERGE Data Link (25674): Message logger not contacted. MERGE Data Link (25674): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25674): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0 MERGE Data Link (25674): File mapped object /SHM_11000 accessed MERGE Data Link (25674): Memory mapping 536628 bytes MERGE Data Link (25674): Shared memory segment located at address 0x7faec5dbd000. MERGE Data Link (25674): Allocated 71720 bytes for local input buffer MERGE Data Link (25674): File mapped object /SHM_11013 accessed MERGE Data Link (25674): Memory mapping 100669480 bytes MERGE Data Link (25674): Shared memory segment located at address 0x7faebfdbb000. MERGE Master (25659): Link task has pid 25676 MERGE Master (25659): Link task has pid 25677 MERGE Data Link (25675): Message logger not contacted. MERGE Data Link (25675): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25675): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0 MERGE Data Link (25675): File mapped object /SHM_11000 accessed MERGE Data Link (25675): Memory mapping 536628 bytes MERGE Data Link (25675): Shared memory segment located at address 0x7fef43120000. MERGE Data Link (25675): Allocated 71720 bytes for local input buffer MERGE Data Link (25675): File mapped object /SHM_11014 accessed MERGE Data Link (25675): Memory mapping 100669480 bytes MERGE Data Link (25675): Shared memory segment located at address 0x7fef3d11e000. MERGE Data Link (25676): Message logger not contacted. MERGE Data Link (25676): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25676): Configuration: index = 14, SHM key=11000, TCP port base = 11015, module = 13, hardware = 0 MERGE Data Link (25676): File mapped object /SHM_11000 accessed MERGE Data Link (25676): Memory mapping 536628 bytes MERGE Data Link (25676): Shared memory segment located at address 0x7f6aea271000. MERGE Data Link (25676): Allocated 71720 bytes for local input buffer MERGE Data Link (25676): File mapped object /SHM_11015 accessed MERGE Data Link (25676): Memory mapping 100669480 bytes MERGE Data Link (25676): Shared memory segment located at address 0x7f6ae426f000. MERGE Data Link (25677): Message logger not contacted. MERGE Data Link (25677): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25677): Configuration: index = 15, SHM key=11000, TCP port base = 11016, module = 14, hardware = 0 MERGE Data Link (25677): File mapped object /SHM_11000 accessed MERGE Data Link (25677): Memory mapping 536628 bytes MERGE Data Link (25677): Shared memory segment located at address 0x7fc4281e9000. MERGE Data Link (25677): Allocated 71720 bytes for local input buffer MERGE Data Link (25677): File mapped object /SHM_11016 accessed MERGE Data Link (25677): Memory mapping 100669480 bytes MERGE Data Link (25677): Shared memory segment located at address 0x7fc4221e7000. MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021 20:32:03 MERGE Statistics thread starting MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000. MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000. MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000. MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000. MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000. MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000. MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000. MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000. MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000. MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000. MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000. MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000. MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000. MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000. MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000. MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000. MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000. MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000. MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000. ... 201 more lines ...
Attachment 3: 210602_TapeServerStartup
Tidy up master(4596): Operation not permitted Starting Tape Server [1] 25639 MIDAS Tape Server: Message logger not contacted. MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021 12:22:14 MIDAS Tape Server: Unable to change scheduling priority - Permission denied MIDAS Tape Server: Using default startup MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205. MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Server: Shared memory ID is 3 MIDAS Tape Server: Shared memory segment located at address 7fdf19721000. MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305 MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002 MIDAS Tape Server: Message reporting level = 0x180fff8 MIDAS Tape Server: Message logging level = 0xfff8 MIDAS Tape Server: Tape Server Options = 0x0 MIDAS Tape Server: File device path base = /TapeData MIDAS Tape Server: Data buffer size = 65536 MIDAS Tape Server: Tape block size = 65536 MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created MIDAS Tape Server: Shared memory ID is 3 MIDAS Tape Server: Shared memory segment located at address 7fdf19320000. MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created MIDAS Tape Server: Shared memory ID is 3 MIDAS Tape Server: Shared memory segment located at address 7fdf19881000. MIDAS Tape Server: Capabilities restored. MIDAS Tape Server: Master global area initialised. MIDAS Tape Server: Stats task has pid 25641 MIDAS Tape Server: Driver process for /dev/file/0 has pid 25642 MIDAS Tape Server: Driver process for /dev/file/1 has pid 25643 MIDAS Tape Server: Driver process for /dev/null/0 has pid 25644 MIDAS Tape Server: Link task 0 has pid 25645 MIDAS Tape Server: Link task 1 has pid 25646 MIDAS Tape Server: Starting the RPC interface MIDAS Tape Statistics: Message logger not contacted. MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021 12:22:20 MIDAS Tape Statistics: Started with args 10205 MIDAS Tape Statistics: Configuration: SHM key=10205 MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Statistics: Shared memory ID is 3 MIDAS Tape Statistics: Shared memory segment located at address 7f9c551f7000. MIDAS Data Link (25645): Message logger not contacted. MIDAS Data Link (25645): MIDAS Tape Data Link Build Mar 16 2021 12:22:19 MIDAS Data Link (25645): Started with args 10205 10305 MIDAS Data Link (25645): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1 MIDAS Tape Driver (25644): Message logger not contacted. MIDAS Tape Driver (25644): Message logger not contacted. MIDAS Data Link (25646): Message logger not contacted. MIDAS Tape Driver (25644): Started with args 2 10205 MIDAS Data Link (25645): File mapped object /SHM_10205 of size 1331104 created MIDAS Data Link (25645): Shared memory ID is 3 MIDAS Data Link (25645): Shared memory segment located at address 7f6ba30ef000. MIDAS Data Link (25646): MIDAS Tape Data Link Build Mar 16 2021 12:22:19 MIDAS Tape Driver (25644): Configuration: driver=2, key=10205. MIDAS Data Link (25646): Started with args 10205 21001 21002 MIDAS Data Link (25645): Starting the network interface MIDAS Tape Driver (25644): File mapped object /SHM_10205 of size 1331104 created MIDAS Data Link (25646): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2 MIDAS Tape Driver (25644): Shared memory ID is 3 MIDAS Tape Driver (25644): Shared memory segment located at address 7f09d718d000. MIDAS Tape Driver (25644): Using device /dev/null/0 of type sink. MIDAS Data Link (25646): File mapped object /SHM_10205 of size 1331104 created MIDAS Data Link (25646): Shared memory ID is 3 MIDAS Data Link (25646): Shared memory segment located at address 7fa75ae3c000. MIDAS Data Link (25646): Starting the network interface MIDAS Tape Driver (25642): Message logger not contacted. MIDAS Data Link (25645): TCP socket receive buffer was 87380 - now 249856 MIDAS Tape Driver (25642): Message logger not contacted. MIDAS Tape Driver (25642): Started with args 0 10205 MIDAS Tape Driver (25642): Configuration: driver=0, key=10205. MIDAS Tape Driver (25643): Message logger not contacted. MIDAS Tape Driver (25642): File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Driver (25643): Message logger not contacted. MIDAS Tape Driver (25642): Shared memory ID is 3 MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856 MIDAS Data Link (25645): TCP socket send buffer was 16384 - now 249856 MIDAS Tape Driver (25643): Started with args 1 10205 MIDAS Tape Driver (25642): Shared memory segment located at address 7f48b1ea4000. MIDAS Tape Driver (25642): Using device /dev/file/0 of type file. MIDAS Data Link (25645): MIDAS Data Link thread 0 using TCP port 10305. MIDAS Data Link (25645): Entering server loop MIDAS Tape Driver (25643): Configuration: driver=1, key=10205. MIDAS Data Link (25645): thread 0 listening on port 10305 MIDAS Tape Driver (25643): File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Driver (25643): Shared memory ID is 3 MIDAS Tape Driver (25643): Shared memory segment located at address 7f6044052000. MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856 MIDAS Tape Driver (25643): Using device /dev/file/1 of type file. MIDAS Data Link (25646): MIDAS Data Link thread 0 using TCP port 21001. MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856 MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856 MIDAS Data Link (25646): MIDAS Data Link thread 1 using TCP port 21002. MIDAS Data Link (25646): Entering server loop MIDAS Data Link (25646): thread 0 listening on port 21001 MIDAS Data Link (25646): thread 1 listening on port 21002 MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205. MIDAS Tape Server: Entering server loop MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205. MIDAS Data Link (25645): thread 0 accepted connection from 127.0.0.1, port 37330 MIDAS Data Link (25645): buffer size changed to 65536 MIDAS Tape Server: device /dev/file/0 allocated. MIDAS Tape Server: Mounting volume S496 on device /dev/file/0. MIDAS Tape Server: Opening file R31 on device /dev/file/0. 2222 64 0 31056336 3333 31056336 2048000 1 15
Attachment 4: TS_configuration
#device configuration information # any line starting with a # is a comment and is ignored stats /MIDAS/TapeServer/Linux64/stats # devices available - list ends with a null line # format class_name device_name driver_task_path_name # class_name MUST be one of dlt, exabyte, scsitape, sink file /dev/file/0 /MIDAS/TapeServer/Linux64/driver file /dev/file/1 /MIDAS/TapeServer/Linux64/driver sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver #data link configuration information - list end with a null line # format link_task_path_name /MIDAS/TapeServer/Linux64/linkTCP 10305 /MIDAS/TapeServer/Linux64/linkTCP 21001 21002 #program options - list end with a null line #tapeserver_options 0x2 tapeserver_options 0x0 msg_reporting_level 0x0180fff8 # use 0x0080fff8 to enable msg logging msg_logging_level 0xfff8 # default for following is 16Kbytes - both MUST be the same at present data_buffer_size 64 tape_block_size 64 disc_file_size 2000 # file_path_base /TapeData #end of information
Attachment 5: CONTENTS
Index string MERGE.LinksAvailable&&RunNumber&&MERGE.LinksInUse&&MERGE.RunOptions Stat.shift string 0 0x0006dead string 0x0000 MERGE.LinksAvailable string 16 Stat.offset string 0 RunNumber string 0 Rate.channels string 1024 Stat.channels string 1024 MERGE.RunOptions string 3 MERGE.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1% Merge.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%
Attachment 6: 210602_merger.png

Goto page Previous 1, 2, 3 ... 17, 18, 19 ... 34, 35, 36 Next

ELOG V3.1.4-unknown