ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 17 of 35

Full | Summary | Threaded | Hide attachments

686 Entries

Goto page Previous 1, 2, 3 ... 16, 17, 18 ... 33, 34, 35 Next

Selected entries:

	ID	Date	Author	Subject
	376	Thu Aug 19 10:37:50 2021	TD	AIDA Noise
11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control) Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs Switch Cooknell SA1 Sum Amp output from negative to positive Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14 This probably indicates - noisy Cooknell SA1 - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1 > Investigate noise situation of AIDA (no DSSSDs connected) > > Figs 1-6: Waves, Pulser Peaks + Widths before any changes > - aida16 very noisy ASICS 1&2, normal ASICS 3&4 > - back-side aidas (02,04,06,08) worse resolution than front-side > - high-frequency noise still noticeable in waveforms > > Figs 7-8: Remove pulser in (from NIM rack) only > - no noticable change > > Figs 9-10: Remove all pulser cables from adapter boards > - looks improved > > Figs 11-12: HV Braid->GND Jumpers removed > > Fig 13-14: All HV cables removed > - aida06, aida14 isn't updating > - very noticable improvement > > Fig 15-16: Intermediate HV cables re-added > - signal worse in FEE64s with HV cables > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s. > > Pulser widths now: > - aida14 (-ve amplitude) = 13.5 channels > - aida08 (+ve amplitude) = 40 channels (?)
	375	Tue Aug 17 12:39:25 2021	NH, TD	AIDA Noise
Investigate noise situation of AIDA (no DSSSDs connected) Figs 1-6: Waves, Pulser Peaks + Widths before any changes - aida16 very noisy ASICS 1&2, normal ASICS 3&4 - back-side aidas (02,04,06,08) worse resolution than front-side - high-frequency noise still noticeable in waveforms Figs 7-8: Remove pulser in (from NIM rack) only - no noticable change Figs 9-10: Remove all pulser cables from adapter boards - looks improved Figs 11-12: HV Braid->GND Jumpers removed Fig 13-14: All HV cables removed - aida06, aida14 isn't updating - very noticable improvement Fig 15-16: Intermediate HV cables re-added - signal worse in FEE64s with HV cables - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s. Pulser widths now: - aida14 (-ve amplitude) = 13.5 channels - aida08 (+ve amplitude) = 40 channels (?)
Attachment 1: Screenshot-Waves1.png

Attachment 2: Screenshot-Waves2.png

Attachment 3: Screenshot-Pulsers.png

Attachment 4: Screenshot-Width11.png

Attachment 5: Screenshot-PulserWidthBack.png

Attachment 6: Screenshot-Width02.png

Attachment 7: Screenshot-PulserInRemov-Front.png

Attachment 8: Screenshot-PulserInRemov-Back.png

Attachment 9: PulserAllOut-Front.png

Attachment 10: PulserAllremov-Back.png

Attachment 11: HVJumpers-Front.png

Attachment 12: HVJumpers-Back.png

Attachment 13: AllHVRemoved-Front.png

Attachment 14: HVout-Back.png

Attachment 15: HVintermediate-Front.png

Attachment 16: HVintermediate-Back.png

	374	Mon Jul 5 14:36:15 2021	NH	Monday 5 July
AIDA Unbiased and powered down All FEEs were running before stoppage. None had rebooted. Aida05 did not respond until merger was restarted, all other DAQs responded fine. Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup AIDA will remain in Snout in lab for time being. FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now
	373	Tue Jun 22 13:01:19 2021	NH	AIDA Data on lustre
The following runs are now copied to lustre If necessary they can be removed from AIDA HDDs for space. There are also spare HDDs that can (and should) be swapped for the current ones /lustre/gamma/DESPEC_S452_FILES/AIDA S452 R1 R2 R3 R4 R5 R6 R7 R9 R11 R13 R14 R15 R16 R17 R20 R21 R22 R23 R25 R27 R28 R29 R30 R33 R34 R35 R36 R38 R39 R40 R42 R43 R46 R50 R56 R57 R58 /lustre/gamma/DESPEC_S460_FILES/AIDA S460 R1 R2 R4 R5 R7 R9 R11 R13 R14 R16 R20 R21 R23 R24 R25 R26 R27 R30 R31 R32 R33 R34 R35 R36 R38 R40 R45 R46 R48 R50 R51 R52 /lustre/gamma/DESPEC_S496_FILES/AIDA S496 R1 R2 R3 R4 R5 R6 R7 R9 R11 R13 R14 R15 R16 R17 R20 R21 R22 R23 R25 R27 R28 R29 R30 R33 R34 R35 R36 R38 R39 R40 R42 R43 R46 R50 R56 R57 R58
	372	Mon Jun 21 10:21:50 2021	NH	Monday 21 June
11.21 - DAQ continues OK (no storage mode) 11.21 - System checks ok except Base Current Difference aida01 fault 0x8944 : 0x8963 : 31 aida02 fault 0x9b36 : 0x9b55 : 31 aida03 fault 0xf26b : 0xf28a : 31 aida04 fault 0x57a0 : 0x57bf : 31 aida05 fault 0x4d3c : 0x4d95 : 89 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x3 : 3 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x9560 : 38240 aida13 fault 0x0 : 0x15b9 : 5561 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 33 12 14 2 4 4 2 2 2 4 6 : 37636 aida02 : 40 7 15 1 3 4 3 2 2 4 6 : 37800 aida03 : 39 7 9 6 5 3 1 3 2 4 6 : 37860 aida04 : 27 7 5 3 3 4 2 3 2 4 6 : 37908 aida05 : 33 5 5 3 6 4 3 3 2 4 6 : 38364 aida06 : 45 12 3 4 4 2 2 3 4 3 6 : 37828 aida07 : 31 11 5 4 4 2 3 2 2 4 6 : 37540 aida08 : 31 6 8 5 3 4 3 2 2 4 6 : 37772 aida09 : 45 4 9 3 2 3 3 2 2 4 6 : 37572 aida10 : 37 11 10 2 3 4 3 3 3 3 6 : 37260 aida11 : 40 9 11 3 3 4 3 3 1 4 6 : 37304 aida12 : 44 11 6 5 3 4 3 3 1 4 6 : 37320 aida13 : 41 11 9 3 5 3 3 3 1 4 6 : 37292 aida14 : 31 12 4 6 5 3 3 3 3 3 6 : 37276 aida15 : 17 9 7 1 2 2 3 2 1 4 7 : 40348 aida16 : 41 12 5 4 5 3 3 3 1 4 6 : 37268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 11.23 - figures Grafana temp & leakage - attachment 1 Weather cooler this week in Germany, expect S4 to cool down eventually Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 aida05 looks very noisy (?) ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 Looks some new time seq error since Thursday Experiment is runnign in S4 so no access possible this week Data is being archived to lustre (GSI PB file storage)
Attachment 1: firefox_2021-06-21_11-27-00.png

Attachment 2: AnyDesk_2021-06-21_11-24-28.png

Attachment 3: AnyDesk_2021-06-21_11-25-44.png

Attachment 4: AnyDesk_2021-06-21_11-26-07.png

Attachment 5: AnyDesk_2021-06-21_11-24-15.png

Attachment 6: AnyDesk_2021-06-21_11-24-43.png

Attachment 7: AnyDesk_2021-06-21_11-25-04.png

Attachment 8: AnyDesk_2021-06-21_11-25-15.png

	371	Thu Jun 17 11:14:04 2021	TD	Thursday 17 June
12.15 DAQ continues OK - no storage mode 12.15 System checks OK except Base Current Difference aida01 fault 0x8944 : 0x8963 : 31 aida02 fault 0x9b36 : 0x9b55 : 31 aida03 fault 0xf26b : 0xf28a : 31 aida04 fault 0x57a0 : 0x57bf : 31 aida05 fault 0x4d3c : 0x4d55 : 25 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x3e03 : 15875 aida13 fault 0x0 : 0x32d6 : 13014 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 31 11 9 4 4 4 2 2 2 4 6 : 37604 aida02 : 30 10 10 3 3 4 3 2 2 4 6 : 37768 aida03 : 35 11 8 6 5 3 2 2 3 3 6 : 36580 aida04 : 33 9 4 4 3 4 2 3 3 3 6 : 36940 aida05 : 41 9 7 3 4 4 2 2 2 4 6 : 37564 aida06 : 38 6 5 3 2 3 2 3 4 3 6 : 37752 aida07 : 25 12 6 5 4 2 3 2 2 4 6 : 37572 aida08 : 32 7 10 5 4 3 3 2 2 4 6 : 37752 aida09 : 41 8 6 4 2 3 3 2 2 4 6 : 37572 aida10 : 31 10 11 5 5 3 3 3 3 3 6 : 37340 aida11 : 29 7 12 3 4 4 3 3 1 4 6 : 37324 aida12 : 42 8 7 5 3 4 3 3 1 4 6 : 37304 aida13 : 43 10 4 5 5 3 3 3 1 4 6 : 37276 aida14 : 40 15 4 5 4 4 3 3 3 3 6 : 37368 aida15 : 13 5 8 2 1 2 3 2 1 4 7 : 40284 aida16 : 39 9 6 6 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 12.15 Hot HEC channels - ASIC check Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-17_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1008.png

Attachment 8: 1009.png

	370	Wed Jun 16 18:05:03 2021	TD	Wednesday 16 June
19.04 All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8960 : 28 aida02 fault 0x9b36 : 0x9b52 : 28 aida03 fault 0xf26b : 0xf287 : 28 aida04 fault 0x57a0 : 0x57bc : 28 aida05 fault 0x4d3c : 0x4d55 : 25 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x3e03 : 15875 aida13 fault 0x0 : 0x32d4 : 13012 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 53 10 4 4 4 4 2 2 3 3 6 : 36580 aida02 : 28 7 10 4 3 4 3 2 2 4 6 : 37768 aida03 : 41 10 7 6 5 3 1 3 3 3 6 : 36836 aida04 : 45 7 2 4 3 4 2 3 3 3 6 : 36940 aida05 : 35 8 8 4 4 4 2 2 2 4 6 : 37580 aida06 : 45 6 6 3 3 3 2 3 4 3 6 : 37860 aida07 : 37 10 6 4 5 2 3 2 2 4 6 : 37636 aida08 : 32 5 10 4 4 4 3 2 2 4 6 : 37832 aida09 : 24 5 8 7 1 3 3 2 2 4 6 : 37544 aida10 : 29 9 12 5 4 3 3 3 3 3 6 : 37276 aida11 : 33 7 7 6 3 4 3 3 1 4 6 : 37292 aida12 : 36 11 6 6 3 4 3 3 1 4 6 : 37320 aida13 : 37 7 6 6 5 3 3 3 1 4 6 : 37292 aida14 : 25 9 7 6 5 3 3 3 3 3 6 : 37276 aida15 : 3 3 4 3 0 3 3 2 1 4 7 : 40260 aida16 : 35 7 6 6 4 4 3 3 1 4 6 : 37348 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19.07 Hot HEC channels - ASIC check Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 significant increases of leakage current to c. 12uA - temperature c. 30 deg C Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-16_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 1005.png

Attachment 7: 1006.png

	369	Tue Jun 15 12:35:48 2021	TD	Tuesday 15 June
13.35 DAQ continues OK - no storage mode All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 37 10 8 5 4 4 2 2 2 4 6 : 37636 aida02 : 37 10 10 3 4 4 3 2 2 4 6 : 37860 aida03 : 41 8 7 7 5 3 1 3 2 4 6 : 37876 aida04 : 33 9 4 4 2 5 2 3 3 3 6 : 37004 aida05 : 37 9 9 3 5 4 2 2 2 4 6 : 37644 aida06 : 39 9 6 5 4 2 2 3 4 3 6 : 37860 aida07 : 29 12 6 5 4 2 3 2 2 4 6 : 37588 aida08 : 25 7 9 5 4 3 3 2 2 4 6 : 37708 aida09 : 34 5 9 6 2 3 3 2 2 4 6 : 37632 aida10 : 34 12 11 5 4 3 3 3 3 3 6 : 37304 aida11 : 38 8 7 6 3 4 3 3 1 4 6 : 37320 aida12 : 34 10 8 6 3 4 3 3 1 4 6 : 37336 aida13 : 33 11 5 6 4 4 3 3 1 4 6 : 37356 aida14 : 31 10 5 6 5 3 3 3 3 3 6 : 37276 aida15 : 14 6 7 4 1 2 3 2 1 4 7 : 40344 aida16 : 33 8 6 7 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 13.40 All histograms, stats & merger stats zero'd
Attachment 1: Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-15_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1002.png

Attachment 8: 1003.png

	368	Tue Jun 15 12:33:46 2021	TD	Monday 14 June

	367	Sun Jun 13 17:29:04 2021	TD	Sunday 13 June
18.15 DAQ continues file S496/R32_946 Hot HEC channels multiple FEE64s from c. 06.00 this morning ASIC check fixes Out of disk space on /TapeData filesystem - switch to no storage mode System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 26 16 4 6 4 4 2 2 2 4 6 : 37608 aida02 : 29 11 9 3 3 4 3 2 2 4 6 : 37756 aida03 : 32 9 6 8 5 3 1 3 3 3 6 : 36840 aida04 : 32 8 3 4 3 4 2 3 3 3 6 : 36912 aida05 : 36 8 6 4 5 4 2 2 2 4 6 : 37616 aida06 : 25 13 4 3 2 3 2 3 4 3 6 : 37740 aida07 : 28 13 4 5 5 2 3 2 2 4 6 : 37624 aida08 : 29 5 9 4 3 4 3 2 2 4 6 : 37740 aida09 : 41 8 4 7 1 3 3 2 2 4 6 : 37572 aida10 : 45 11 7 5 5 3 3 3 3 3 6 : 37340 aida11 : 33 13 5 6 3 4 3 3 1 4 6 : 37308 aida12 : 39 8 6 6 4 4 3 3 1 4 6 : 37372 aida13 : 28 14 3 6 4 4 3 3 1 4 6 : 37328 aida14 : 30 11 3 6 4 4 3 3 3 3 6 : 37312 aida15 : 3 3 4 3 0 3 3 2 1 4 7 : 40260 aida16 : 28 9 5 6 5 3 3 3 1 4 6 : 37256 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png

Attachment 2: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png

Attachment 5: Screenshot_2021-06-13_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1000.png

Attachment 8: 1001.png

	366	Sat Jun 12 14:53:56 2021	TD	aida15 - system console output - 15.33 Sat 12 June reboot
04/11:03:40\|get_ASICBlk : Blk=1 : bytes = 196096 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500240 : 1 => 0F07C000 04/11:03:41\|Last Data : Databuffer 273286648 : 0 => 80500240 : 1 => 0AFF8000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:42\|Machine check in kernel mode. 12/15:30:08\|I-Cache Parity Error 12/15:30:08\|Oops: Machine check, sig: 7 [#1] 12/15:30:08\|PREEMPT Xilinx Virtex440 12/15:30:08\|Modules linked in: aidamem xdriver xh_spidev_register 12/15:30:08\|NIP: c0055c54 LR: c00558fc CTR: c000ae5c 12/15:30:08\|REGS: c6ff9f10 TRAP: 0214 Not tainted (2.6.31) 12/15:30:08\|MSR: 00021000 <ME,CE> CR: 82000028 XER: 20000000 12/15:30:08\|TASK = c6ac44e0[375] 'AidaExecV9' THREAD: c6338000 12/15:30:08\|GPR00: 00000000 c6339cd0 c6ac44e0 ffffffff ffe79600 ffffffff ffe79600 ffffffff 12/15:30:08\|GPR08: 00000000 0000001f ffffffe0 00000000 00000000 1005520c 1dcd6500 00000000 12/15:30:08\|GPR16: 00000000 1009df98 100f0000 c0390000 000007a1 7fe9e580 ffffffff ffffffff 12/15:30:08\|GPR24: c0390000 00000001 00000000 001366ca c036f0d0 ffffffe1 ffffffff ffec9936 12/15:30:08\|NIP [c0055c54] update_wall_time+0x40c/0xd3c 12/15:30:09\|LR [c00558fc] update_wall_time+0xb4/0xd3c 12/15:30:09\|Call Trace: 12/15:30:09\|[c6339cd0] [c0055f44] update_wall_time+0x6fc/0xd3c (unreliable) 12/15:30:09\|[c6339d90] [c0040b84] do_timer+0x38/0x4c 12/15:30:09\|[c6339da0] [c005a088] tick_periodic+0xbc/0xe8 12/15:30:09\|[c6339db0] [c005a0d4] tick_handle_periodic+0x20/0x120 12/15:30:09\|[c6339df0] [c000af70] timer_interrupt+0xa4/0x10c 12/15:30:09\|[c6339e10] [c000e9c4] ret_from_except+0x0/0x18 12/15:30:09\|[c6339ed0] [d10751dc] aidamem_read+0x28/0x108 [aidamem] 12/15:30:09\|[c6339ef0] [c0097e80] vfs_read+0xb4/0x188 12/15:30:09\|[c6339f10] [c00982f8] sys_read+0x4c/0x90 12/15:30:09\|[c6339f40] [c000e364] ret_from_syscall+0x0/0x3c 12/15:30:09\|Instruction dump: 12/15:30:09\|39600000 354bffe0 418008b0 7cc35030 38800000 2f8a0000 419c08bc 7ffe5030 12/15:30:09\|3aa00000 93c10048 92a1004c 83c10048 <83e1004c> 7d095830 4800056c 419e06bc 12/15:30:09\|Kernel panic - not syncing: Fatal exception in interrupt 12/15:30:09\|Call Trace: 12/15:30:09\|Rebooting in 180 seconds.. ISOL Version 1.00 Date 9th January 2017 Flash base address=FC000000 Set Flash to ASync Mode XST_SUCCESS\| Finished copying zImage to RAM 12/15:33:10\| Found 0 errors checking kernel image 12/15:33:11\|VHDL version number 0X03350706 Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012 Starting LMK 3200 setup 12/15:33:11\| Setting LMK03200 to standard clock settings -- External Clock 23Nov15 .... SPI Base Address=0x81400000 clk_control_reg=0x4 12/15:33:11\|Next step is SPIconfig Control 32(0x81400000)=0x180 SlaveSel(0x81400000)=0x3 Ctrl(0x81400000)=0xE6 Ctrl(0x81400000)=0x86 12/15:33:11\|SPIconfig done now to set up the LMK3200 registers 12/15:33:12\|LMK #0 : regInit[0]=0x80000000 12/15:33:12\|LMK #0 : regInit[1]=0x10070600 12/15:33:12\|LMK #0 : regInit[2]=0x60601 12/15:33:12\|LMK #0 : regInit[3]=0x60602 12/15:33:12\|LMK #0 : regInit[4]=0x60603 12/15:33:12\|LMK #0 : regInit[5]=0x70624 12/15:33:12\|LMK #0 : regInit[6]=0x70605 12/15:33:12\|LMK #0 : regInit[7]=0x70606 12/15:33:12\|LMK #0 : regInit[8]=0x70627 12/15:33:12\|LMK #0 : regInit[9]=0x10000908 12/15:33:12\|LMK #0 : regInit[10]=0xA0022A09 12/15:33:12\|LMK #0 : regInit[11]=0x82800B 12/15:33:12\|LMK #0 : regInit[12]=0x28C800D 12/15:33:12\|LMK #0 : regInit[13]=0x830020E 12/15:33:12\|LMK #0 : regInit[14]=0xC800180F Calibrate completed at 941 counts Setting Clock Control =0x0000000B, to set GOE and sync bit Ctrl @ SPIstop (0x81400000)=0x186 Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004 12/15:33:12\| Finished Clock setup LMK03200 completed LMK 3200 setup Loaded all four ASICs with default settings Setting the ADCs into calibration mode 12/15:33:12\| Control 32(0x81400400)=0x180 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86 Init : Config of AD9252 SPI ok 12/15:33:13\| Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised ADCs calibrated 12/15:33:13\| Control 32(0x81400400)=0x186 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86Config of AD9252 SPI ok 12/15:33:13\| Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot... 12/15:33:13\| zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0) Allocating 0x3b78cc bytes for kernel ... gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes 12/15:33:16\| Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M Finalizing device tree... flat tree at 0xbd1300 Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 12/15:33:23\|Using Xilinx Virtex440 machine description 12/15:33:23\|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011 12/15:33:23\|Zone PFN ranges: 12/15:33:23\| DMA 0x00000000 -> 0x00007000 12/15:33:23\| Normal 0x00007000 -> 0x00007000 12/15:33:23\|Movable zone start PFN for each node 12/15:33:23\|early_node_map[1] active PFN ranges 12/15:33:23\| 0: 0x00000000 -> 0x00007000 12/15:33:23\|MMU: Allocated 1088 bytes of context maps for 255 contexts 12/15:33:23\|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448 12/15:33:24\|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M 12/15:33:24\|PID hash table entries: 512 (order: 9, 2048 bytes) 12/15:33:24\|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) 12/15:33:24\|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) 12/15:33:24\|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init) 12/15:33:24\|Kernel virtual memory layout: 12/15:33:24\| * 0xffffe000..0xfffff000 : fixmap 12/15:33:24\| * 0xfde00000..0xfe000000 : consistent mem 12/15:33:24\| * 0xfde00000..0xfde00000 : early ioremap 12/15:33:24\| * 0xd1000000..0xfde00000 : vmalloc & ioremap 12/15:33:24\|NR_IRQS:512 12/15:33:24\|clocksource: timebase mult[a00000] shift[22] registered 12/15:33:24\|Console: colour dummy device 80x25 12/15:33:24\|Mount-cache hash table entries: 512 12/15:33:24\|NET: Registered protocol family 16 12/15:33:24\|PCI: Probing PCI hardware 12/15:33:24\|bio: create slab <bio-0> at 0 12/15:33:24\|NET: Registered protocol family 2 12/15:33:24\|IP route cache hash table entries: 1024 (order: 0, 4096 bytes) 12/15:33:24\|TCP established hash table entries: 4096 (order: 3, 32768 bytes) 12/15:33:25\|TCP bind hash table entries: 4096 (order: 2, 16384 bytes) 12/15:33:25\|TCP: Hash tables configured (established 4096 bind 4096) 12/15:33:25\|TCP reno registered 12/15:33:25\|NET: Registered protocol family 1 12/15:33:25\|ROMFS MTD (C) 2007 Red Hat, Inc. 12/15:33:25\|msgmni has been set to 214 12/15:33:25\|io scheduler noop registered 12/15:33:25\|io scheduler anticipatory registered 12/15:33:25\|io scheduler deadline registered 12/15:33:25\|io scheduler cfq registered (default) 12/15:33:25\|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 12/15:33:25\|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550 12/15:33:25\|console [ttyS0] enabled 12/15:33:25\|brd: module loaded 12/15:33:25\|loop: module loaded 12/15:33:25\|Device Tree Probing 'ethernet' 12/15:33:25\|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 12/15:33:25\|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode. 12/15:33:25\|XLlTemac: DCR address: 0x80 12/15:33:25\|XLlTemac: buffer descriptor size: 32768 (0x8000) 12/15:33:25\|XLlTemac: Allocating DMA descriptors with kmalloc 12/15:33:25\|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000 12/15:33:26\|XTemac: PHY detected at address 7. 12/15:33:26\|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17 12/15:33:26\|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\|Using buffer write method 12/15:33:26\|cfi_cmdset_0001: Erase suspend on write enabled 12/15:33:26\|cmdlinepart partition parsing not available 12/15:33:26\|RedBoot partition parsing not available
	365	Sat Jun 12 13:56:00 2021	TD	Saturday 12 June
14.11 DAQ continues file S496/R32_697 High data rate from c. 06.00 this morning ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8948 : 4 aida02 fault 0x9b36 : 0x9b3a : 4 aida03 fault 0xf26b : 0xf26f : 4 aida04 fault 0x57a0 : 0x57a4 : 4 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x21be : 8638 aida13 fault 0x0 : 0x2294 : 8852 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 33 10 9 2 5 4 2 2 2 4 6 : 37604 aida02 : 46 12 7 2 4 4 3 2 2 4 6 : 37832 aida03 : 34 10 9 5 4 3 1 3 3 3 6 : 36744 aida04 : 35 12 10 1 3 4 2 3 3 3 6 : 36972 aida05 : 37 9 8 3 4 4 2 2 2 4 6 : 37564 aida06 : 34 10 5 5 4 2 2 3 2 4 6 : 37832 aida07 : 31 11 1 7 4 2 3 2 2 4 6 : 37572 aida08 : 40 7 4 5 3 4 3 2 2 4 6 : 37752 aida09 : 39 9 8 4 1 3 3 2 2 4 6 : 37540 aida10 : 43 12 8 4 4 3 3 3 1 4 6 : 37260 aida11 : 34 14 10 5 3 4 3 3 1 4 6 : 37368 aida12 : 46 12 3 4 4 4 3 3 1 4 6 : 37320 aida13 : 41 11 5 4 5 3 3 3 1 4 6 : 37260 aida14 : 39 10 2 6 4 4 3 3 1 4 6 : 37324 aida15 : 19 12 4 3 2 2 3 2 2 4 6 : 37324 aida16 : 45 10 6 4 5 3 3 3 1 4 6 : 37284 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 15.05 analysis of file S496/R32_698 - attachment 9 max dead time 0.05% aida04 15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365 successful DAQ stop aida15 DAQ reset, setup, go successful DAQ go
Attachment 1: Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-12_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 998.png

Attachment 8: 999.png

Attachment 9: R32_698
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 44970784 ( 15685.0 Hz) Other data format: 216949222 ( 75668.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 62 ( 0.0 Hz) RESUME: 62 ( 0.0 Hz) SYNC100: 42680 ( 14.9 Hz) WR48-63: 42680 ( 14.9 Hz) FEE64 disc: 985985 ( 343.9 Hz) MBS info: 215877753 ( 75294.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2867.115 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2299.418 2 0.673 0.000 3 1.368 0.000 4 0.000 0.000 5 0.000 2495.101 6 0.000 0.000 7 0.000 2566.695 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 467 2 0 0 0 0 1 1 0 0 0 0 1 72420 30 0 0 0 0 15 15 0 0 0 0 2 935 107990737 0 0 33 33 19385 19385 0 107951901 0 0 3 38882631 108955999 0 0 29 29 22052 22052 985985 107925852 0 0 4 496 0 0 0 0 0 0 0 0 0 0 0 5 3224207 1336 0 0 0 0 668 668 0 0 0 0 6 1520 0 0 0 0 0 0 0 0 0 0 0 7 2784837 1118 0 0 0 0 559 559 0 0 0 0 8 473 0 0 0 0 0 0 0 0 0 0 0 9 173 0 0 0 0 0 0 0 0 0 0 0 10 554 0 0 0 0 0 0 0 0 0 0 0 11 317 0 0 0 0 0 0 0 0 0 0 0 12 752 0 0 0 0 0 0 0 0 0 0 0 13 204 0 0 0 0 0 0 0 0 0 0 0 14 302 0 0 0 0 0 0 0 0 0 0 0 15 496 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.945s ( 1068.615 blocks/s, 66.788 Mb/s)
	363	Thu Jun 10 11:32:27 2021	TD	Thursday 10 June
12.26 DAQ continues file S496/R32_428 High data rate from c. 06.00 this morning ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 11 6 3 4 4 2 2 2 4 6 : 37540 aida02 : 30 8 7 3 4 4 3 2 2 4 6 : 37768 aida03 : 38 8 13 3 4 3 1 3 3 3 6 : 36744 aida04 : 37 9 7 2 3 4 2 3 3 3 6 : 36940 aida05 : 29 5 9 3 3 3 1 3 2 4 6 : 37580 aida06 : 38 6 2 3 3 3 2 3 2 4 6 : 37768 aida07 : 31 9 2 7 4 2 3 2 2 4 6 : 37572 aida08 : 34 6 3 5 4 4 3 2 2 4 6 : 37768 aida09 : 29 2 12 5 1 3 3 2 2 4 6 : 37540 aida10 : 24 8 15 3 5 3 3 3 1 4 6 : 37296 aida11 : 35 4 15 3 3 4 3 3 1 4 6 : 37308 aida12 : 36 5 3 5 6 4 3 3 1 4 6 : 37384 aida13 : 31 10 4 3 6 4 3 3 1 4 6 : 37356 aida14 : 37 7 1 4 5 4 3 3 1 4 6 : 37276 aida15 : 34 10 6 3 4 1 2 4 2 4 6 : 38168 aida16 : 31 9 3 3 5 4 3 3 1 4 6 : 37268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 202 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 15.07 analysis of file S496/R32_431 - attachment 9 max dead time aida04 0.33%
Attachment 1: Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-10_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 996.png

Attachment 8: 997.png

Attachment 9: R32_431
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 54180182 ( 19758.0 Hz) Other data format: 207739819 ( 75756.7 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 105 ( 0.0 Hz) RESUME: 105 ( 0.0 Hz) SYNC100: 42213 ( 15.4 Hz) WR48-63: 42213 ( 15.4 Hz) FEE64 disc: 1515114 ( 552.5 Hz) MBS info: 206140069 ( 75173.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 6 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2742.196 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2662.854 2 0.384 0.000 3 9.009 0.000 4 0.000 0.000 5 0.000 1805.119 6 0.000 0.000 7 0.000 1797.783 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 559 0 0 0 0 0 0 0 0 0 0 0 1 413404 177 0 0 0 0 84 84 9 0 0 0 2 1153 103268358 0 0 19 19 17963 17963 0 103232394 0 0 3 42054837 104466440 0 0 86 86 21757 21757 1515079 102907675 0 0 4 515 0 0 0 0 0 0 0 0 0 0 0 5 5561752 2359 0 0 0 0 1175 1175 9 0 0 0 6 1738 2 0 0 0 0 1 1 0 0 0 0 7 6142236 2472 0 0 0 0 1232 1232 8 0 0 0 8 565 1 0 0 0 0 0 0 1 0 0 2 9 190 0 0 0 0 0 0 0 0 0 0 0 10 694 0 0 0 0 0 0 0 0 0 0 0 11 417 6 0 0 0 0 0 0 6 0 0 4 12 967 4 0 0 0 0 1 1 2 0 0 0 13 244 0 0 0 0 0 0 0 0 0 0 0 14 285 0 0 0 0 0 0 0 0 0 0 0 15 626 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.160s ( 1097.388 blocks/s, 68.587 Mb/s)
	362	Wed Jun 9 19:17:21 2021	TD	FEE64 system console logs

Attachment 1: logs-090621.tar.gz
	361	Wed Jun 9 11:27:47 2021	TD	Wednesday 9 June
12.30 DAQ continues OK file S496/R32_373 System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 13 5 12 7 2 4 2 2 1 4 7 : 40572 aida02 : 11 6 8 6 1 3 3 2 1 4 7 : 40540 aida03 : 26 5 11 5 1 2 2 3 2 3 7 : 39712 aida04 : 18 7 8 3 2 3 2 3 2 3 7 : 39776 aida05 : 11 5 9 4 1 2 2 3 1 4 7 : 40612 aida06 : 17 3 6 5 5 2 1 3 1 4 7 : 40604 aida07 : 9 3 4 9 1 3 3 2 1 4 7 : 40540 aida08 : 11 8 5 5 3 3 3 2 1 4 7 : 40604 aida09 : 13 5 8 5 2 3 3 2 1 4 7 : 40572 aida10 : 13 8 13 5 1 2 2 2 1 4 7 : 40228 aida11 : 11 5 14 6 3 3 3 3 2 3 7 : 40244 aida12 : 10 9 5 5 3 3 2 2 1 4 7 : 40352 aida13 : 15 7 7 4 3 4 3 3 2 3 7 : 40228 aida14 : 9 6 7 6 3 4 3 3 2 3 7 : 40260 aida15 : 14 3 8 5 1 2 2 4 1 4 7 : 41136 aida16 : 15 4 9 4 4 4 3 3 2 3 7 : 40300 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 1.8.L specttra - attachments 1 & 2 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3 Lost activity monitor - attachment 4 1.8.W spectra - 20us FSR - attachments 5 & 6 ADC data items - attachment 7 FEE64 temperatures OK - attachment 8 DSSSD bias & leakage currents OK - attachment 9 Merger/Tape Server/Merger statistics - attachment 10 no merger errors reported since previous restart 16.28 analysis of file S496/R32_372 - attachment 11 max deadtime 0.07% aida04
Attachment 1: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png

Attachment 2: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png

Attachment 3: Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png

Attachment 5: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png

Attachment 6: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png

Attachment 7: Screenshot_2021-06-09_Statistics_aidas-gsi.png

Attachment 8: Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png

Attachment 9: 991.png

Attachment 10: 992.png

Attachment 11: R32_372
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 65571173 ( 25420.1 Hz) Other data format: 196348830 ( 76118.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 52 ( 0.0 Hz) RESUME: 52 ( 0.0 Hz) SYNC100: 41608 ( 16.1 Hz) WR48-63: 41608 ( 16.1 Hz) FEE64 disc: 2050316 ( 794.8 Hz) MBS info: 194215194 ( 75291.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 284 ( 0.1 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2579.504 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.015 1853.986 2 0.548 0.000 3 1.886 0.000 4 0.000 0.000 5 0.000 1457.725 6 0.000 0.000 7 0.000 1391.289 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 1323.904 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 836.274 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 6664 532 0 0 0 0 1 1 530 0 0 3 1 4648488 85466 0 0 1 1 947 947 83570 0 0 92 2 5675 97165745 0 0 27 27 16487 16487 7 97132710 0 2 3 47009038 98898439 0 0 24 24 21346 21346 1773239 97082460 0 42 4 3608 4 0 0 0 0 0 0 4 0 0 1 5 6949043 95991 0 0 0 0 1383 1383 93225 0 0 60 6 5001 30 0 0 0 0 1 1 4 24 0 0 7 6897601 102341 0 0 0 0 1436 1436 99469 0 0 74 8 1797 1 0 0 0 0 0 0 1 0 0 1 9 2911 4 0 0 0 0 1 1 2 0 0 3 10 8722 137 0 0 0 0 1 1 135 0 0 0 11 5985 36 0 0 0 0 2 2 32 0 0 4 12 7238 34 0 0 0 0 1 1 32 0 0 0 13 7345 52 0 0 0 0 0 0 52 0 0 0 14 4898 4 0 0 0 0 0 0 4 0 0 1 15 7159 14 0 0 0 0 2 2 10 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 30.055s ( 1064.726 blocks/s, 66.545 Mb/s)
	360	Tue Jun 8 10:51:55 2021	TD	Tuesday 8 June
11.52 DAQ continues file S496/R32_340 High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning 11.53 ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 3 12 8 3 3 2 2 1 4 7 : 40508 aida02 : 5 5 10 6 1 3 3 2 1 4 7 : 40540 aida03 : 25 7 12 5 1 2 2 3 2 3 7 : 39740 aida04 : 19 6 11 5 1 3 2 3 2 3 7 : 39820 aida05 : 9 6 7 5 0 2 2 3 1 4 7 : 40548 aida06 : 10 8 7 4 3 3 1 3 1 4 7 : 40600 aida07 : 9 3 4 8 1 3 3 2 1 4 7 : 40508 aida08 : 12 5 8 5 3 3 3 2 1 4 7 : 40632 aida09 : 15 4 8 5 1 3 3 2 1 4 7 : 40508 aida10 : 23 5 12 5 1 2 2 2 1 4 7 : 40228 aida11 : 15 9 11 6 2 4 3 3 2 3 7 : 40308 aida12 : 14 9 4 5 3 3 2 2 1 4 7 : 40352 aida13 : 15 9 6 3 4 4 3 3 2 3 7 : 40260 aida14 : 11 5 7 6 4 4 3 3 2 3 7 : 40324 aida15 : 16 8 7 4 1 2 2 4 1 4 7 : 41136 aida16 : 15 8 7 4 3 4 3 3 2 3 7 : 40236 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 16.28 analysis of file S496/R32_339 - attachment 9 max deadtime 0.24% aida04
Attachment 1: Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-08_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 990.png

Attachment 8: 991.png

Attachment 9: R32_339
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 52833088 ( 19139.6 Hz) Other data format: 209086916 ( 75745.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 101 ( 0.0 Hz) RESUME: 101 ( 0.0 Hz) SYNC100: 42281 ( 15.3 Hz) WR48-63: 42281 ( 15.3 Hz) FEE64 disc: 1254898 ( 454.6 Hz) MBS info: 207747254 ( 75259.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2760.405 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2672.516 2 0.610 0.000 3 6.711 0.000 4 0.000 0.000 5 0.000 2111.274 6 0.000 0.000 7 0.000 1910.870 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 589 0 0 0 0 0 0 0 0 0 0 0 1 221336 98 0 0 0 0 49 49 0 0 0 0 2 1118 104025022 0 0 30 30 18193 18193 0 103988576 0 0 3 42352603 105057740 0 0 71 71 22011 22011 1254898 103758678 0 0 4 548 0 0 0 0 0 0 0 0 0 0 0 5 4562177 1856 0 0 0 0 928 928 0 0 0 0 6 1810 0 0 0 0 0 0 0 0 0 0 0 7 5688898 2196 0 0 0 0 1098 1098 0 0 0 0 8 593 0 0 0 0 0 0 0 0 0 0 0 9 215 2 0 0 0 0 1 1 0 0 0 0 10 675 2 0 0 0 0 1 1 0 0 0 0 11 387 0 0 0 0 0 0 0 0 0 0 0 12 964 0 0 0 0 0 0 0 0 0 0 0 13 267 0 0 0 0 0 0 0 0 0 0 0 14 369 0 0 0 0 0 0 0 0 0 0 0 15 539 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.656s ( 1116.685 blocks/s, 69.793 Mb/s)
	359	Mon Jun 7 10:33:48 2021	TD	Monday 7 June
11.30 DAQ continues OK - file S496/R32_144 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 5 11 6 3 4 2 2 1 4 7 : 40572 aida02 : 1 3 6 7 2 3 3 2 1 4 7 : 40540 aida03 : 21 7 7 8 2 4 1 3 2 3 7 : 39804 aida04 : 13 7 5 6 1 3 2 3 2 3 7 : 39740 aida05 : 2 8 6 5 1 2 2 3 1 4 7 : 40584 aida06 : 10 4 5 5 4 3 1 3 1 4 7 : 40632 aida07 : 9 3 4 7 3 3 3 2 1 4 7 : 40604 aida08 : 11 2 6 4 4 3 3 2 1 4 7 : 40604 aida09 : 10 2 5 4 2 2 2 3 1 4 7 : 40584 aida10 : 14 5 8 6 2 1 1 3 1 4 7 : 40352 aida11 : 8 6 9 5 4 4 3 3 2 3 7 : 40320 aida12 : 10 5 3 7 3 3 2 2 1 4 7 : 40352 aida13 : 7 5 4 6 5 4 3 3 2 3 7 : 40324 aida14 : 7 5 4 8 4 4 3 3 2 3 7 : 40324 aida15 : 8 4 7 4 2 2 2 4 1 4 7 : 41136 aida16 : 13 5 7 6 3 4 3 3 2 3 7 : 40268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 11.35 analysis of file S496/R32_143 - attachment 9 max deadtime 0.29% aida03
Attachment 1: Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-07_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 986.png

Attachment 8: 987.png

Attachment 9: R32_143
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 57707988 ( 21988.9 Hz) Other data format: 204212013 ( 77812.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 85 ( 0.0 Hz) RESUME: 85 ( 0.0 Hz) SYNC100: 41772 ( 15.9 Hz) WR48-63: 41772 ( 15.9 Hz) FEE64 disc: 1109863 ( 422.9 Hz) MBS info: 203018436 ( 77357.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2624.418 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2432.488 2 0.665 0.000 3 7.525 0.000 4 0.000 0.000 5 0.000 1464.970 6 0.000 0.000 7 0.000 1288.051 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 112 0 0 0 0 0 0 0 0 0 0 0 1 1940572 788 0 0 0 0 394 394 0 0 0 0 2 442 101676817 0 0 30 30 17294 17294 0 101642169 0 0 3 40857770 102528388 0 0 55 55 21074 21074 1109863 101376267 0 0 4 129 0 0 0 0 0 0 0 0 0 0 0 5 7006930 2878 0 0 0 0 1439 1439 0 0 0 0 6 548 0 0 0 0 0 0 0 0 0 0 0 7 7898329 3142 0 0 0 0 1571 1571 0 0 0 0 8 285 0 0 0 0 0 0 0 0 0 0 0 9 71 0 0 0 0 0 0 0 0 0 0 0 10 1135 0 0 0 0 0 0 0 0 0 0 0 11 122 0 0 0 0 0 0 0 0 0 0 0 12 679 0 0 0 0 0 0 0 0 0 0 0 13 92 0 0 0 0 0 0 0 0 0 0 0 14 666 0 0 0 0 0 0 0 0 0 0 0 15 106 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.551s ( 1082.882 blocks/s, 67.680 Mb/s)
	358	Sun Jun 6 14:15:05 2021	TD	Sunday 6 June
14.15 DAQ continues file S496/R32_117 Hot HEC channels multiple FEE64s from c. 06.00 this morning ASIC check fixes System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 12 12 3 3 4 2 2 1 4 7 : 40572 aida02 : 7 8 10 5 2 3 3 2 1 4 7 : 40604 aida03 : 27 10 4 8 1 4 1 3 2 3 7 : 39740 aida04 : 18 7 4 5 1 3 2 3 2 3 7 : 39712 aida05 : 15 13 8 4 1 3 1 3 1 4 7 : 40548 aida06 : 14 8 8 6 3 2 1 3 1 4 7 : 40568 aida07 : 11 4 5 6 2 3 3 2 1 4 7 : 40540 aida08 : 15 6 5 3 4 3 3 2 1 4 7 : 40604 aida09 : 18 4 4 3 3 2 2 3 1 4 7 : 40648 aida10 : 12 6 8 6 1 1 1 3 1 4 7 : 40288 aida11 : 16 8 6 6 4 4 3 3 2 3 7 : 40352 aida12 : 10 7 4 8 3 4 3 3 2 3 7 : 40288 aida13 : 15 7 5 4 5 4 3 3 2 3 7 : 40324 aida14 : 9 4 6 7 3 4 3 3 2 3 7 : 40260 aida15 : 12 4 8 3 2 2 2 4 1 4 7 : 41136 aida16 : 11 8 6 6 3 4 3 3 2 3 7 : 40268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 16.28 analysis of file S496/R32_118 - attachment 9 max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-06_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 984.png

Attachment 8: 985.png

Attachment 9: R32_118
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 48860593 ( 17844.1 Hz) Other data format: 213059412 ( 77810.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 101 ( 0.0 Hz) RESUME: 101 ( 0.0 Hz) SYNC100: 42200 ( 15.4 Hz) WR48-63: 42200 ( 15.4 Hz) FEE64 disc: 1112345 ( 406.2 Hz) MBS info: 211862465 ( 77373.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2738.188 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2613.952 2 0.575 0.000 3 7.745 0.000 4 0.000 0.000 5 0.000 2154.815 6 0.000 0.000 7 0.000 1921.162 8 0.000 777.447 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 155 0 0 0 0 0 0 0 0 0 0 0 1 66625 28 0 0 0 0 14 14 0 0 0 0 2 415 106108157 0 0 26 26 18899 18899 0 106070307 0 0 3 39298810 106947309 0 0 75 75 21328 21328 1112345 105792158 0 0 4 171 0 0 0 0 0 0 0 0 0 0 0 5 4255149 1684 0 0 0 0 842 842 0 0 0 0 6 596 0 0 0 0 0 0 0 0 0 0 0 7 5234928 2226 0 0 0 0 1113 1113 0 0 0 0 8 493 4 0 0 0 0 2 2 0 0 0 0 9 92 0 0 0 0 0 0 0 0 0 0 0 10 1212 2 0 0 0 0 1 1 0 0 0 0 11 160 0 0 0 0 0 0 0 0 0 0 0 12 781 0 0 0 0 0 0 0 0 0 0 0 13 89 0 0 0 0 0 0 0 0 0 0 0 14 724 2 0 0 0 0 1 1 0 0 0 0 15 193 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.738s ( 1113.497 blocks/s, 69.594 Mb/s)
	357	Sat Jun 5 11:20:15 2021	TD	Saturday 5 June
12.25 DAQ continues OK - file S496/R32_48 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 10 6 1 2 4 4 2 2 1 4 7 : 40360 aida02 : 4 3 3 3 3 4 2 2 1 4 7 : 40312 aida03 : 13 5 4 4 2 4 1 3 2 3 7 : 39580 aida04 : 16 5 3 3 2 4 1 3 2 3 7 : 39544 aida05 : 8 6 1 2 3 3 1 3 1 4 7 : 40416 aida06 : 4 3 2 1 3 3 1 3 1 4 7 : 40360 aida07 : 6 6 2 4 4 2 3 2 1 4 7 : 40424 aida08 : 6 4 1 1 4 3 3 2 1 4 7 : 40424 aida09 : 16 1 3 1 0 2 2 3 1 4 7 : 40344 aida10 : 2 5 1 4 1 3 2 4 2 3 7 : 40064 aida11 : 14 3 2 1 4 4 3 3 2 3 7 : 40080 aida12 : 11 2 2 4 4 4 3 3 2 3 7 : 40156 aida13 : 11 3 2 2 3 4 3 3 2 3 7 : 40036 aida14 : 6 5 0 4 3 4 3 3 2 3 7 : 40064 aida15 : 11 6 3 1 3 1 2 4 1 4 7 : 40940 aida16 : 12 5 0 4 4 4 3 3 2 3 7 : 40152 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 1.8.W spectra - 20us FSR - attachments 2 & 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart analysis of file S496/R32_47 - attachment 8 max deadtime 0.35% aida04 13.56 Lost activity monitor - see attachment 9
Attachment 1: Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-06-05_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 982.png

Attachment 7: 983.png

Attachment 8: R32_47
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 47883982 ( 17391.8 Hz) Other data format: 214036019 ( 77739.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 184 ( 0.1 Hz) RESUME: 184 ( 0.1 Hz) SYNC100: 42255 ( 15.3 Hz) WR48-63: 42255 ( 15.3 Hz) FEE64 disc: 1053753 ( 382.7 Hz) MBS info: 212897388 ( 77325.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2753.252 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2696.425 2 0.825 0.000 3 9.735 0.000 4 0.000 0.000 5 0.000 2175.120 6 0.000 0.000 7 0.000 2003.563 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 187 0 0 0 0 0 0 0 0 0 0 0 1 86597 40 0 0 0 0 20 20 0 0 0 0 2 381 106659018 0 0 37 37 18746 18746 0 106621452 0 0 3 38584230 107373193 0 0 147 147 21605 21605 1053753 106275936 0 0 4 143 0 0 0 0 0 0 0 0 0 0 0 5 4208337 1756 0 0 0 0 878 878 0 0 0 0 6 545 0 0 0 0 0 0 0 0 0 0 0 7 5000124 2012 0 0 0 0 1006 1006 0 0 0 0 8 478 0 0 0 0 0 0 0 0 0 0 0 9 96 0 0 0 0 0 0 0 0 0 0 0 10 1149 0 0 0 0 0 0 0 0 0 0 0 11 117 0 0 0 0 0 0 0 0 0 0 0 12 706 0 0 0 0 0 0 0 0 0 0 0 13 105 0 0 0 0 0 0 0 0 0 0 0 14 636 0 0 0 0 0 0 0 0 0 0 0 15 151 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.996s ( 1066.806 blocks/s, 66.675 Mb/s)
Attachment 9: Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png

	356	Fri Jun 4 09:47:44 2021	TD	aida15 - system console output - 00.02 Friday 4 June reboot
03/23:58:55\|kernel BUG at mm/slab.c:2974! 03/23:58:55\|Oops: Exception in kernel mode, sig: 5 [#1] 03/23:58:55\|PREEMPT Xilinx Virtex440 03/23:58:55\|Modules linked in: aidamem xdriver xh_spidev_register 03/23:58:55\|NIP: c009211c LR: c00920b0 CTR: c0038e70 03/23:58:55\|REGS: c631fc60 TRAP: 0700 Not tainted (2.6.31) 03/23:58:55\|MSR: 00021000 <ME,CE> CR: 22022048 XER: 00000000 03/23:58:56\|TASK = c627a0c0[375] 'AidaExecV9' THREAD: c631e000 03/23:58:56\|GPR00: 00000001 c631fd10 c627a0c0 c680daf0 00000020 00000001 c6940020 00000000 03/23:58:56\|GPR08: fffffff6 00010002 c680e400 c631e000 22008022 1005520c c03a0000 00000020 03/23:58:56\|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 03/23:58:56\|GPR24: 00100100 c631e000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400 03/23:58:56\|NIP [c009211c] cache_alloc_refill+0x130/0x608 03/23:58:56\|LR [c00920b0] cache_alloc_refill+0xc4/0x608 03/23:58:56\|Call Trace: 03/23:58:56\|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable) 03/23:58:56\|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc 03/23:58:56\|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8 03/23:58:56\|[c631fdb0] [c0042938] __send_signal+0x78/0x260 03/23:58:56\|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c 03/23:58:56\|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c 03/23:58:56\|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30 03/23:58:56\|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240 03/23:58:56\|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c 03/23:58:56\|[c631feb0] [c0040bf4] update_process_times+0x30/0x70 03/23:58:57\|[c631fed0] [c005a000] tick_periodic+0x34/0xe8 03/23:58:57\|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120 03/23:58:57\|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c 03/23:58:57\|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18 03/23:58:57\|Instruction dump: 03/23:58:57\|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 03/23:58:57\|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 03/23:58:57\|Kernel panic - not syncing: Fatal exception in interrupt 03/23:58:57\|Call Trace: 03/23:58:57\|[c631fab0] [c0005de8] show_stack+0x44/0x16c (unreliable) 03/23:58:57\|[c631faf0] [c00345bc] panic+0x94/0x168 03/23:58:57\|[c631fb40] [c000bd44] die+0x178/0x18c 03/23:58:57\|[c631fb60] [c000c000] _exception+0x164/0x1b4 03/23:58:57\|[c631fc50] [c000e978] ret_from_except_full+0x0/0x4c 03/23:58:57\|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 03/23:58:57\|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc 03/23:58:57\|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8 03/23:58:57\|[c631fdb0] [c0042938] __send_signal+0x78/0x260 03/23:58:57\|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c 03/23:58:58\|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c 03/23:58:58\|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30 03/23:58:58\|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240 03/23:58:58\|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c 03/23:58:58\|[c631feb0] [c0040bf4] update_process_times+0x30/0x70 03/23:58:58\|[c631fed0] [c005a000] tick_periodic+0x34/0xe8 03/23:58:58\|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120 03/23:58:58\|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c 03/23:58:58\|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18 03/23:58:58\|Rebooting in 180 seconds.. ISOL Version 1.00 Date 9th January 2017 Flash base address=FC000000 Set Flash to ASync Mode XST_SUCCESS\| Finished copying zImage to RAM 04/00:01:59\| Found 0 errors checking kernel image 04/00:02:00\|VHDL version number 0X03350706 Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012 Starting LMK 3200 setup 04/00:02:00\| Setting LMK03200 to standard clock settings -- External Clock 23Nov15 .... SPI Base Address=0x81400000 clk_control_reg=0x4 04/00:02:00\|Next step is SPIconfig Control 32(0x81400000)=0x180 SlaveSel(0x81400000)=0x3 Ctrl(0x81400000)=0xE6 Ctrl(0x81400000)=0x86 04/00:02:00\|SPIconfig done now to set up the LMK3200 registers 04/00:02:00\|LMK #0 : regInit[0]=0x80000000 04/00:02:00\|LMK #0 : regInit[1]=0x10070600 04/00:02:00\|LMK #0 : regInit[2]=0x60601 04/00:02:00\|LMK #0 : regInit[3]=0x60602 04/00:02:00\|LMK #0 : regInit[4]=0x60603 04/00:02:00\|LMK #0 : regInit[5]=0x70624 04/00:02:00\|LMK #0 : regInit[6]=0x70605 04/00:02:00\|LMK #0 : regInit[7]=0x70606 04/00:02:00\|LMK #0 : regInit[8]=0x70627 04/00:02:00\|LMK #0 : regInit[9]=0x10000908 04/00:02:00\|LMK #0 : regInit[10]=0xA0022A09 04/00:02:00\|LMK #0 : regInit[11]=0x82800B 04/00:02:01\|LMK #0 : regInit[12]=0x28C800D 04/00:02:01\|LMK #0 : regInit[13]=0x830020E 04/00:02:01\|LMK #0 : regInit[14]=0xC800180F Calibrate completed at 941 counts Setting Clock Control =0x0000000B, to set GOE and sync bit Ctrl @ SPIstop (0x81400000)=0x186 Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004 04/00:02:01\| Finished Clock setup LMK03200 completed LMK 3200 setup Loaded all four ASICs with default settings Setting the ADCs into calibration mode 04/00:02:01\| Control 32(0x81400400)=0x180 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86 Init : Config of AD9252 SPI ok 04/00:02:01\| Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised Cal DCMs not locked ADC calibrate failed Jumping to kernel simpleboot... 04/00:02:01\| zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0) Allocating 0x3b78cc bytes for kernel ... gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes 04/00:02:05\| Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M Finalizing device tree... flat tree at 0xbd1300 Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 04/00:02:11\|Using Xilinx Virtex440 machine description 04/00:02:12\|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011 04/00:02:12\|Zone PFN ranges: 04/00:02:12\| DMA 0x00000000 -> 0x00007000 04/00:02:12\| Normal 0x00007000 -> 0x00007000 04/00:02:12\|Movable zone start PFN for each node 04/00:02:12\|early_node_map[1] active PFN ranges 04/00:02:12\| 0: 0x00000000 -> 0x00007000 04/00:02:12\|MMU: Allocated 1088 bytes of context maps for 255 contexts 04/00:02:12\|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448 04/00:02:12\|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M 04/00:02:12\|PID hash table entries: 512 (order: 9, 2048 bytes) 04/00:02:12\|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) 04/00:02:12\|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) 04/00:02:12\|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init) 04/00:02:12\|Kernel virtual memory layout: 04/00:02:12\| * 0xffffe000..0xfffff000 : fixmap 04/00:02:12\| * 0xfde00000..0xfe000000 : consistent mem 04/00:02:12\| * 0xfde00000..0xfde00000 : early ioremap 04/00:02:13\| * 0xd1000000..0xfde00000 : vmalloc & ioremap 04/00:02:13\|NR_IRQS:512 04/00:02:13\|clocksource: timebase mult[a00000] shift[22] registered 04/00:02:13\|Console: colour dummy device 80x25 04/00:02:13\|Mount-cache hash table entries: 512 04/00:02:13\|NET: Registered protocol family 16 04/00:02:13\|PCI: Probing PCI hardware 04/00:02:13\|bio: create slab <bio-0> at 0 04/00:02:13\|NET: Registered protocol family 2 04/00:02:13\|IP route cache hash table entries: 1024 (order: 0, 4096 bytes) 04/00:02:13\|TCP established hash table entries: 4096 (order: 3, 32768 bytes) 04/00:02:13\|TCP bind hash table entries: 4096 (order: 2, 16384 bytes) 04/00:02:13\|TCP: Hash tables configured (established 4096 bind 4096) 04/00:02:13\|TCP reno registered 04/00:02:13\|NET: Registered protocol family 1 04/00:02:13\|ROMFS MTD (C) 2007 Red Hat, Inc. 04/00:02:13\|msgmni has been set to 214 04/00:02:13\|io scheduler noop registered 04/00:02:13\|io scheduler anticipatory registered 04/00:02:13\|io scheduler deadline registered 04/00:02:13\|io scheduler cfq registered (default) 04/00:02:13\|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 04/00:02:13\|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550 04/00:02:14\|console [ttyS0] enabled 04/00:02:14\|brd: module loaded 04/00:02:14\|loop: module loaded 04/00:02:14\|Device Tree Probing 'ethernet' 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode. 04/00:02:14\|XLlTemac: DCR address: 0x80 04/00:02:14\|XLlTemac: buffer descriptor size: 32768 (0x8000) 04/00:02:14\|XLlTemac: Allocating DMA descriptors with kmalloc 04/00:02:14\|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000 04/00:02:14\|XTemac: PHY detected at address 7. 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17 04/00:02:14\|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\|Using buffer write method 04/00:02:15\|cfi_cmdset_0001: Erase suspend on write enabled 04/00:02:15\|cmdlinepart partition parsing not available 04/00:02:15\|RedBoot partition parsing not available 04/00:02:15\|Creating 5 MTD partitions on "fc000000.flash": 04/00:02:15\|0x000000000000-0x000000500000 : "golden_firmware" 04/00:02:15\|0x000000500000-0x000000800000 : "golden_kernel" 04/00:02:15\|0x000000800000-0x000000d00000 : "user_firmware" 04/00:02:15\|0x000000d00000-0x000000fe0000 : "user_kernel" 04/00:02:15\|0x000000fe0000-0x000001000000 : "env_variables" 04/00:02:15\|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20 04/00:02:15\|SPI: XIlinx spi: bus number now 32766 04/00:02:15\|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21 04/00:02:15\|SPI: XIlinx spi: bus number now 32765 04/00:02:15\|mice: PS/2 mouse device common for all mice 04/00:02:15\|Device Tree Probing 'i2c' 04/00:02:15\| #0 at 0x81600000 mapped to 0xD1030000, irq=22 04/00:02:15\|at24 0-0050: 1024 byte 24c08 EEPROM (writable) 04/00:02:15\|TCP cubic registered 04/00:02:15\|NET: Registered protocol family 17 04/00:02:15\|RPC: Registered udp transport module. 04/00:02:15\|RPC: Registered tcp transport module. 04/00:02:16\|eth0: XLlTemac: Options: 0x3fa 04/00:02:16\|eth0: XLlTemac: allocating interrupt 19 for dma mode tx. 04/00:02:16\|eth0: XLlTemac: allocating interrupt 18 for dma mode rx. 04/00:02:16\|eth0: XLlTemac: speed set to 1000Mb/s 04/00:02:18\|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4 04/00:02:18\|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254 04/00:02:18\|Sending DHCP requests ., OK 04/00:02:19\|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.15 04/00:02:20\|IP-Config: Complete: 04/00:02:20\| device=eth0, addr=192.168.11.15, mask=255.255.255.0, gw=255.255.255.255, 04/00:02:20\| host=aida15, domain=dl.ac.uk, nis-domain=nuclear.physics, 04/00:02:20\| bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15 04/00:02:20\|Looking up port of RPC 100003/2 on 192.168.11.99 04/00:02:20\|Looking up port of RPC 100005/1 on 192.168.11.99 04/00:02:20\|VFS: Mounted root (nfs filesystem) on device 0:12. 04/00:02:20\|Freeing unused kernel memory: 168k init INIT: version 2.86 booting 04/00:02:20\|Starting sysinit... 04/00:02:21\| Welcome to DENX & STFC Daresbury Embedded Linux Environment 04/00:02:22\| Press 'I' to enter interactive startup. 04/00:02:22\|Setting clock (utc): Thu Jun 3 23:02:22 BST 2021 [ OK ] 04/00:02:22\|Building the cache [ OK ] 04/00:02:22\|Setting hostname aida15: [ OK ] 04/00:02:23\|Mounting local filesystems: [ OK ] 04/00:02:25\|Enabling /etc/fstab swaps: [ OK ] 04/00:02:27\|Finishing sysinit... INIT: Entering runlevel: 3 04/00:02:30\|Entering non-interactive startup 04/00:02:30\|FATAL: Module ipv6 not found. 04/00:02:31\|Bringing up loopback interface: [ OK ] 04/00:02:33\|FATAL: Module ipv6 not found. 04/00:02:34\|Starting system logger: [ OK ] 04/00:02:34\|Starting kernel logger: [ OK ] 04/00:02:35\|Starting rpcbind: [ OK ] 04/00:02:36\|Mounting NFS filesystems: [ OK ] 04/00:02:37\|Mounting other filesystems: [ OK ] 04/00:02:37\|Starting xinetd: [ OK ] 04/00:02:38\|Starting midas: Starting MIDAS Data Acquisition for aida15 04/00:02:38\|xaida: device parameters: base=0x81000000 size=0x200000 04/00:02:44\|Trying to free nonexistent resource <0000000081000000-00000000811fffff> 04/00:02:44\|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000 04/00:02:44\|xaida: driver assigned major number 254 04/00:02:44\|Trying to free nonexistent resource <0000000007000000-0000000007ffffff> 04/00:02:50\|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000 04/00:02:50\|AIDAMEM: aidamem: driver assigned major number 253 04/00:02:50\|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31 04/00:02:56\|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx 04/00:02:57\|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx 04/00:02:57\|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin 04/00:02:57\|Computer Name = aida15; Temp Directory = /tmp/tcl361 04/00:03:01\|package limit is not available: can't find package limit 04/00:03:04\|Running with default file descriptor limit 04/00:03:04\|package setuid is not available: can't find package setuid 04/00:03:05\|Running as user 0 group 0 04/00:03:05\| 04/00:03:06\|AIDA Data Acquisition Program Release 9_10.Apr 3 2019_11:34:31 starting 04/00:03:06\| 04/00:03:06\|Built without pthreads 04/00:03:06\| 04/00:03:06\|Creating NetVars 04/00:03:06\|Output buffer length = 65xaida: open: 04/00:03:06\|504; format option = 4AIDAMEM: aidamem_open: 04/00:03:06\|; transfer option = 3 04/00:03:06\|EB transfer option = 3 04/00:03:06\|NetVars created and initialised 04/00:03:06\|Statistics thread starting 04/00:03:06\|Statistics thread created 04/00:03:06\|Stat/Rate creation thread starting 04/00:03:06\|Data Acquisition task has PID 375 04/00:03:06\|Stat/Rate creation thread created 04/00:03:06\|Hit/Rate creation thread starting 04/00:03:06\|Hit/Rate creation thread created 04/00:03:07\|AIDA Heartbeat thread starting 04/00:03:07\|Heartbeat thread created 04/00:03:07\|Installing signal handlers 04/00:03:07\|Done 04/00:03:07\|ModuleNum = 0 04/00:03:07\|Aida Initialise complete. AidaExecV9_10: Build Apr 3 2019_11:34:31. HDL version : 03350706 04/00:03:07\|Spectra table initialised 04/00:03:07\|AIDA Data Acquisition now all ready to start 04/00:03:07\|SIGBUS, SIGSEGV and SIGPIPE traps setup 04/00:03:07\|/debug user "debug" password " bjkhcfyuqis" 04/00:03:14\|httpd started on port 8015 04/00:03:14\| 04/00:03:14\|Cannot use /MIDAS/config/TclHttpd/aida15@8015/startup.tcl 04/00:03:14\|Custom startup from /MIDAS/config/TclHttpd/aida15/startup.tcl 04/00:03:14\|XAIDA Access package 1.0 04/00:03:16\|/XAIDAAccessServer 04/00:03:16\|XAD9252 Access package 1.0 04/00:03:16\|/XAD9252AccessServer 04/00:03:16\|/DataBaseAccessServer 04/00:03:16\|[ OK ] 04/00:03:16\|/NetVarService 04/00:03:17\|/SigTaskService 04/00:03:17\|Loaded MemSasAccess 04/00:03:17\|/SpectrumService 04/00:03:17\|loading tcl/AIDARunControl.tcl for namespace :: 04/00:03:17\| 04/00:03:17\|DENX ELDK version 4.2 build 2008-04-01 04/00:03:17\|Linux 2.6.31 on a ppc 04/00:03:17\| 04/00:03:17\|aida15 login: /DataAcquisitionControlServer 04/00:03:17\|DefineMessage unknown 04/00:03:17\|Run Control Server Implementation for AIDA 04/00:03:17\|RunControlServer loaded 04/00:03:17\|loading Html/RunControl/implementation.tcl 04/00:03:18\|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory 04/00:03:18\|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist 04/00:03:18\|Created UI registers 04/00:03:18\|RunControl loaded 04/00:03:19\|loading Html/AIDA/RunControl/implementation.tcl for namespace :: 04/00:03:19\|AIDA RunControl loaded 04/00:03:19\|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl 04/00:03:19\|do_GetState returned z=0 and 8 04/09:31:05\|do_GetState returned z=0 and 8 04/09:31:17\|do_GetState returned z=0 and 8 04/09:31:30\|do_GetState returned z=0 and 8 04/09:31:43\|do_GetState returned z=0 and 8 04/09:31:55\|do_GetState returned z=0 and 8 04/09:32:07\|do_GetState returned z=0 and 8 04/09:32:19\|do_GetState returned z=0 and 8 04/09:32:32\|do_GetState returned z=0 and 8 04/09:32:45\|do_GetState returned z=0 and 8 04/09:32:57\|do_GetState returned z=0 and 8 04/09:33:10\|do_GetState returned z=0 and 8 04/09:33:22\|do_GetState returned z=0 and 8 04/09:33:35\|do_GetState returned z=0 and 8 04/09:33:47\|do_GetState returned z=0 and 8 04/09:34:00\|do_GetState returned z=0 and 8 04/09:34:13\|do_GetState returned z=0 and 8 04/09:34:25\|do_GetState returned z=0 and 8 04/09:34:38\|do_GetState returned z=0 and 8 04/09:34:50\|do_GetState returned z=0 and 8 04/09:35:03\|do_GetState returned z=0 and 8 04/09:35:16\|do_GetState returned z=0 and 8 04/09:35:28\|do_GetState returned z=0 and 8 04/09:35:41\|do_GetState returned z=0 and 8 04/09:35:53\|do_GetState returned z=0 and 8 04/09:36:06\|do_GetState returned z=0 and 8 04/09:36:18\|do_GetState returned z=0 and 8 04/09:36:31\|do_GetState returned z=0 and 8 04/09:36:43\|do_GetState returned z=0 and 8 04/09:36:56\|do_GetState returned z=0 and 8 04/09:37:09\|do_GetState returned z=0 and 8 04/09:37:21\|do_GetState returned z=0 and 8 04/09:37:34\|do_GetState returned z=0 and 8 04/09:37:46\|do_GetState returned z=0 and 8 04/09:37:59\|do_GetState returned z=0 and 8 04/09:38:12\|do_GetState returned z=0 and 8 04/09:38:24\|do_GetState returned z=0 and 8 04/09:38:37\|do_GetState returned z=0 and 8 04/09:38:49\|do_GetState returned z=0 and 8 04/09:39:02\|do_GetState returned z=0 and 8 04/09:39:15\|do_GetState returned z=0 and 8 04/09:39:27\|do_GetState returned z=0 and 8 04/09:39:40\|do_GetState returned z=0 and 8 04/09:39:52\|do_GetState returned z=0 and 8 04/09:40:05\|do_GetState returned z=0 and 8 04/09:40:18\|do_GetState returned z=0 and 8 04/09:40:30\|do_GetState returned z=0 and 8 04/09:40:43\|do_GetState returned z=0 and 8 04/09:40:55\|do_GetState returned z=0 and 8 04/09:41:08\|do_GetState returned z=0 and 8 04/09:41:21\|do_GetState returned z=0 and 8 04/09:41:33\|do_GetState returned z=0 and 8 04/09:41:46\|do_GetState returned z=0 and 8 04/09:41:58\|do_GetState returned z=0 and 8 04/09:42:11\|do_GetState returned z=0 and 8 04/09:42:23\|do_GetState returned z=0 and 8 04/09:42:36\|do_GetState returned z=0 and 8 04/09:42:48\|do_GetState returned z=0 and 8 04/09:43:01\|do_GetState returned z=0 and 8 04/09:43:14\|do_GetState returned z=0 and 8 04/09:43:26\|do_GetState returned z=0 and 8 04/09:43:39\|do_GetState returned z=0 and 8 04/09:43:51\|do_GetState returned z=0 and 8 04/09:44:04\|do_GetState returned z=0 and 8 04/09:44:17\|do_GetState returned z=0 and 8 04/09:44:29\|do_GetState returned z=0 and 8 04/09:44:42\|do_GetState returned z=0 and 8 04/09:44:54\|do_GetState returned z=0 and 8 04/09:45:07\|do_GetState returned z=0 and 8 04/09:45:19\|do_GetState returned z=0 and 8 04/09:45:32\|do_GetState returned z=0 and 8 04/09:45:44\|do_GetState returned z=0 and 8 04/09:45:57\|do_GetState returned z=0 and 8 04/09:46:09\|do_GetState returned z=0 and 8 04/09:46:22\|do_GetState returned z=0 and 8 04/09:46:34\|do_GetState returned z=0 and 8 04/09:46:47\|do_GetState returned z=0 and 8 04/09:47:00\|do_GetState returned z=0 and 8 04/09:47:12\|do_GetState returned z=0 and 8 04/09:47:25\|do_GetState returned z=0 and 8 04/09:47:37\|do_GetState returned z=0 and 8 04/09:47:50\|do_GetState returned z=0 and 8 04/09:48:03\|do_GetState returned z=0 and 8 04/09:48:15\|do_GetState returned z=0 and 8 04/09:48:28\|do_GetState returned z=0 and 8 04/09:48:40\|do_GetState returned z=0 and 8 04/09:48:53\|do_GetState returned z=0 and 8 04/09:49:05\|do_GetState returned z=0 and 8 04/09:49:18\|do_GetState returned z=0 and 8 04/09:49:31\|do_GetState returned z=0 and 8 04/09:49:43\|do_GetState returned z=0 and 8 04/09:49:56\|do_GetState returned z=0 and 8 04/09:50:08\|do_GetState returned z=0 and 8 04/09:50:21\|do_GetState returned z=0 and 8 04/09:50:33\|do_GetState returned z=0 and 8 04/09:50:46\|do_GetState returned z=0 and 8 04/09:50:58\|do_GetState returned z=0 and 8 04/09:51:11\|do_GetState returned z=0 and 8 04/09:51:23\|do_GetState returned z=0 and 8 04/09:51:36\|do_GetState returned z=0 and 8 04/09:51:49\|do_GetState returned z=0 and 8 04/09:52:01\|do_GetState returned z=0 and 8 04/09:52:14\|do_GetState returned z=0 and 8 04/09:52:26\|do_GetState returned z=0 and 8 04/09:52:39\|do_GetState returned z=0 and 8 04/09:52:51\|do_GetState returned z=0 and 8 04/09:53:04\|do_GetState returned z=0 and 8 04/09:53:16\|do_GetState returned z=0 and 8 04/09:53:29\|do_GetState returned z=0 and 8 04/09:53:42\|do_GetState returned z=0 and 8 04/09:53:54\|do_GetState returned z=0 and 8 04/09:54:07\|do_GetState returned z=0 and 8 04/09:54:19\|do_GetState returned z=0 and 8 04/09:54:32\|do_GetState returned z=0 and 8 04/09:54:44\|do_GetState returned z=0 and 8 04/09:54:57\|do_GetState returned z=0 and 8 04/09:55:09\|do_GetState returned z=0 and 8 04/09:55:22\|do_GetState returned z=0 and 8 04/09:55:34\|do_GetState returned z=0 and 8 04/09:55:47\|do_GetState returned z=0 and 8 04/09:56:00\|do_GetState returned z=0 and 8 04/09:56:12\|do_GetState returned z=0 and 8 04/09:56:25\|do_GetState returned z=0 and 8 04/09:56:37\|do_GetState returned z=0 and 8 04/09:56:50\|do_GetState returned z=0 and 8 04/09:57:02\|do_GetState returned z=0 and 8 04/09:57:15\|do_GetState returned z=0 and 8 04/09:57:27\|do_GetState returned z=0 and 8 04/09:57:40\|do_GetState returned z=0 and 8 04/09:57:53\|do_GetState returned z=0 and 8 04/09:58:05\|do_GetState returned z=0 and 8 04/09:58:18\|do_GetState returned z=0 and 8 04/09:58:30\|do_GetState returned z=0 and 8 04/09:58:43\|do_GetState returned z=0 and 8 04/09:58:56\|do_GetState returned z=0 and 8 04/09:59:08\|do_GetState returned z=0 and 8 04/09:59:21\|do_GetState returned z=0 and 8 04/09:59:33\|do_GetState returned z=0 and 8 04/09:59:46\|do_GetState returned z=0 and 8 04/09:59:58\|do_GetState returned z=0 and 8 04/10:00:11\|do_GetState returned z=0 and 8 04/10:00:23\|do_GetState returned z=0 and 8 04/10:00:36\|do_GetState returned z=0 and 8 04/10:00:48\|do_GetState returned z=0 and 8 04/10:01:01\|do_GetState returned z=0 and 8 04/10:01:13\|do_GetState returned z=0 and 8 04/10:01:26\|do_GetState returned z=0 and 8 04/10:01:38\|do_GetState returned z=0 and 8 04/10:01:51\|do_GetState returned z=0 and 8 04/10:02:04\|do_GetState returned z=0 and 8 04/10:02:16\|do_GetState returned z=0 and 8 04/10:02:29\|do_GetState returned z=0 and 8 04/10:02:41\|do_GetState returned z=0 and 8 04/10:02:54\|do_GetState returned z=0 and 8 04/10:03:06\|do_GetState returned z=0 and 8 04/10:03:19\|do_GetState returned z=0 and 8 04/10:03:31\|do_GetState returned z=0 and 8 04/10:03:44\|do_GetState returned z=0 and 8 04/10:03:57\|do_GetState returned z=0 and 8 04/10:04:09\|do_GetState returned z=0 and 8 04/10:04:22\|do_GetState returned z=0 and 8 04/10:04:34\|do_GetState returned z=0 and 8 04/10:04:47\|do_GetState returned z=0 and 8 04/10:04:59\|do_GetState returned z=0 and 8 04/10:05:12\|do_GetState returned z=0 and 8 04/10:05:25\|do_GetState returned z=0 and 8 04/10:05:37\|do_GetState returned z=0 and 8 04/10:05:50\|do_GetState returned z=0 and 8 04/10:06:02\|do_GetState returned z=0 and 8 04/10:06:15\|do_GetState returned z=0 and 8 04/10:06:27\|do_GetState returned z=0 and 8 04/10:06:40\|do_GetState returned z=0 and 8 04/10:06:52\|do_GetState returned z=0 and 8 04/10:07:05\|do_GetState returned z=0 and 8 04/10:07:18\|do_GetState returned z=0 and 8 04/10:07:30\|do_GetState returned z=0 and 8

Goto page Previous 1, 2, 3 ... 16, 17, 18 ... 33, 34, 35 Next

ELOG V3.1.4-unknown