ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 17 of 35

Full | Summary | Threaded | Hide attachments

687 Entries

Goto page Previous 1, 2, 3 ... 16, 17, 18 ... 33, 34, 35 Next

ID	Date	Author	Subject
377	Thu Aug 19 10:47:13 2021	TD	MIDAS spectra

Attachment 1: AIDA_spectra_190821.tar.gz
376	Thu Aug 19 10:37:50 2021	TD	AIDA Noise
11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control) Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs Switch Cooknell SA1 Sum Amp output from negative to positive Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14 This probably indicates - noisy Cooknell SA1 - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1 > Investigate noise situation of AIDA (no DSSSDs connected) > > Figs 1-6: Waves, Pulser Peaks + Widths before any changes > - aida16 very noisy ASICS 1&2, normal ASICS 3&4 > - back-side aidas (02,04,06,08) worse resolution than front-side > - high-frequency noise still noticeable in waveforms > > Figs 7-8: Remove pulser in (from NIM rack) only > - no noticable change > > Figs 9-10: Remove all pulser cables from adapter boards > - looks improved > > Figs 11-12: HV Braid->GND Jumpers removed > > Fig 13-14: All HV cables removed > - aida06, aida14 isn't updating > - very noticable improvement > > Fig 15-16: Intermediate HV cables re-added > - signal worse in FEE64s with HV cables > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s. > > Pulser widths now: > - aida14 (-ve amplitude) = 13.5 channels > - aida08 (+ve amplitude) = 40 channels (?)
375	Tue Aug 17 12:39:25 2021	NH, TD	AIDA Noise
Investigate noise situation of AIDA (no DSSSDs connected) Figs 1-6: Waves, Pulser Peaks + Widths before any changes - aida16 very noisy ASICS 1&2, normal ASICS 3&4 - back-side aidas (02,04,06,08) worse resolution than front-side - high-frequency noise still noticeable in waveforms Figs 7-8: Remove pulser in (from NIM rack) only - no noticable change Figs 9-10: Remove all pulser cables from adapter boards - looks improved Figs 11-12: HV Braid->GND Jumpers removed Fig 13-14: All HV cables removed - aida06, aida14 isn't updating - very noticable improvement Fig 15-16: Intermediate HV cables re-added - signal worse in FEE64s with HV cables - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s. Pulser widths now: - aida14 (-ve amplitude) = 13.5 channels - aida08 (+ve amplitude) = 40 channels (?)
Attachment 1: Screenshot-Waves1.png

Attachment 2: Screenshot-Waves2.png

Attachment 3: Screenshot-Pulsers.png

Attachment 4: Screenshot-Width11.png

Attachment 5: Screenshot-PulserWidthBack.png

Attachment 6: Screenshot-Width02.png

Attachment 7: Screenshot-PulserInRemov-Front.png

Attachment 8: Screenshot-PulserInRemov-Back.png

Attachment 9: PulserAllOut-Front.png

Attachment 10: PulserAllremov-Back.png

Attachment 11: HVJumpers-Front.png

Attachment 12: HVJumpers-Back.png

Attachment 13: AllHVRemoved-Front.png

Attachment 14: HVout-Back.png

Attachment 15: HVintermediate-Front.png

Attachment 16: HVintermediate-Back.png

374	Mon Jul 5 14:36:15 2021	NH	Monday 5 July
AIDA Unbiased and powered down All FEEs were running before stoppage. None had rebooted. Aida05 did not respond until merger was restarted, all other DAQs responded fine. Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup AIDA will remain in Snout in lab for time being. FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now
373	Tue Jun 22 13:01:19 2021	NH	AIDA Data on lustre
The following runs are now copied to lustre If necessary they can be removed from AIDA HDDs for space. There are also spare HDDs that can (and should) be swapped for the current ones /lustre/gamma/DESPEC_S452_FILES/AIDA S452 R1 R2 R3 R4 R5 R6 R7 R9 R11 R13 R14 R15 R16 R17 R20 R21 R22 R23 R25 R27 R28 R29 R30 R33 R34 R35 R36 R38 R39 R40 R42 R43 R46 R50 R56 R57 R58 /lustre/gamma/DESPEC_S460_FILES/AIDA S460 R1 R2 R4 R5 R7 R9 R11 R13 R14 R16 R20 R21 R23 R24 R25 R26 R27 R30 R31 R32 R33 R34 R35 R36 R38 R40 R45 R46 R48 R50 R51 R52 /lustre/gamma/DESPEC_S496_FILES/AIDA S496 R1 R2 R3 R4 R5 R6 R7 R9 R11 R13 R14 R15 R16 R17 R20 R21 R22 R23 R25 R27 R28 R29 R30 R33 R34 R35 R36 R38 R39 R40 R42 R43 R46 R50 R56 R57 R58
372	Mon Jun 21 10:21:50 2021	NH	Monday 21 June
11.21 - DAQ continues OK (no storage mode) 11.21 - System checks ok except Base Current Difference aida01 fault 0x8944 : 0x8963 : 31 aida02 fault 0x9b36 : 0x9b55 : 31 aida03 fault 0xf26b : 0xf28a : 31 aida04 fault 0x57a0 : 0x57bf : 31 aida05 fault 0x4d3c : 0x4d95 : 89 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x3 : 3 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x9560 : 38240 aida13 fault 0x0 : 0x15b9 : 5561 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 33 12 14 2 4 4 2 2 2 4 6 : 37636 aida02 : 40 7 15 1 3 4 3 2 2 4 6 : 37800 aida03 : 39 7 9 6 5 3 1 3 2 4 6 : 37860 aida04 : 27 7 5 3 3 4 2 3 2 4 6 : 37908 aida05 : 33 5 5 3 6 4 3 3 2 4 6 : 38364 aida06 : 45 12 3 4 4 2 2 3 4 3 6 : 37828 aida07 : 31 11 5 4 4 2 3 2 2 4 6 : 37540 aida08 : 31 6 8 5 3 4 3 2 2 4 6 : 37772 aida09 : 45 4 9 3 2 3 3 2 2 4 6 : 37572 aida10 : 37 11 10 2 3 4 3 3 3 3 6 : 37260 aida11 : 40 9 11 3 3 4 3 3 1 4 6 : 37304 aida12 : 44 11 6 5 3 4 3 3 1 4 6 : 37320 aida13 : 41 11 9 3 5 3 3 3 1 4 6 : 37292 aida14 : 31 12 4 6 5 3 3 3 3 3 6 : 37276 aida15 : 17 9 7 1 2 2 3 2 1 4 7 : 40348 aida16 : 41 12 5 4 5 3 3 3 1 4 6 : 37268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 11.23 - figures Grafana temp & leakage - attachment 1 Weather cooler this week in Germany, expect S4 to cool down eventually Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 aida05 looks very noisy (?) ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 Looks some new time seq error since Thursday Experiment is runnign in S4 so no access possible this week Data is being archived to lustre (GSI PB file storage)
Attachment 1: firefox_2021-06-21_11-27-00.png

Attachment 2: AnyDesk_2021-06-21_11-24-28.png

Attachment 3: AnyDesk_2021-06-21_11-25-44.png

Attachment 4: AnyDesk_2021-06-21_11-26-07.png

Attachment 5: AnyDesk_2021-06-21_11-24-15.png

Attachment 6: AnyDesk_2021-06-21_11-24-43.png

Attachment 7: AnyDesk_2021-06-21_11-25-04.png

Attachment 8: AnyDesk_2021-06-21_11-25-15.png

371	Thu Jun 17 11:14:04 2021	TD	Thursday 17 June
12.15 DAQ continues OK - no storage mode 12.15 System checks OK except Base Current Difference aida01 fault 0x8944 : 0x8963 : 31 aida02 fault 0x9b36 : 0x9b55 : 31 aida03 fault 0xf26b : 0xf28a : 31 aida04 fault 0x57a0 : 0x57bf : 31 aida05 fault 0x4d3c : 0x4d55 : 25 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x3e03 : 15875 aida13 fault 0x0 : 0x32d6 : 13014 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 31 11 9 4 4 4 2 2 2 4 6 : 37604 aida02 : 30 10 10 3 3 4 3 2 2 4 6 : 37768 aida03 : 35 11 8 6 5 3 2 2 3 3 6 : 36580 aida04 : 33 9 4 4 3 4 2 3 3 3 6 : 36940 aida05 : 41 9 7 3 4 4 2 2 2 4 6 : 37564 aida06 : 38 6 5 3 2 3 2 3 4 3 6 : 37752 aida07 : 25 12 6 5 4 2 3 2 2 4 6 : 37572 aida08 : 32 7 10 5 4 3 3 2 2 4 6 : 37752 aida09 : 41 8 6 4 2 3 3 2 2 4 6 : 37572 aida10 : 31 10 11 5 5 3 3 3 3 3 6 : 37340 aida11 : 29 7 12 3 4 4 3 3 1 4 6 : 37324 aida12 : 42 8 7 5 3 4 3 3 1 4 6 : 37304 aida13 : 43 10 4 5 5 3 3 3 1 4 6 : 37276 aida14 : 40 15 4 5 4 4 3 3 3 3 6 : 37368 aida15 : 13 5 8 2 1 2 3 2 1 4 7 : 40284 aida16 : 39 9 6 6 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 12.15 Hot HEC channels - ASIC check Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-17_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1008.png

Attachment 8: 1009.png

370	Wed Jun 16 18:05:03 2021	TD	Wednesday 16 June
19.04 All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8960 : 28 aida02 fault 0x9b36 : 0x9b52 : 28 aida03 fault 0xf26b : 0xf287 : 28 aida04 fault 0x57a0 : 0x57bc : 28 aida05 fault 0x4d3c : 0x4d55 : 25 aida05 : WR status 0x10 aida06 fault 0x640c : 0x6425 : 25 aida07 fault 0x255e : 0x2577 : 25 aida08 fault 0xffbd : 0xffd6 : 25 aida09 fault 0x87e : 0x895 : 23 aida10 fault 0xc557 : 0xc56e : 23 aida11 fault 0x683e : 0x6855 : 23 aida12 fault 0x1c35 : 0x1c4c : 23 aida13 fault 0x8f41 : 0x8f58 : 23 aida14 fault 0x1b47 : 0x1b5e : 23 aida15 fault 0x4e4 : 0x127e : 3482 aida16 fault 0xf570 : 0xf587 : 23 White Rabbit error counter test result: Passed 0, Failed 16 Base Current Difference aida01 fault 0x0 : 0x2 : 2 aida02 fault 0x0 : 0x2 : 2 aida05 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida09 fault 0x0 : 0x2 : 2 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x2 : 2 aida12 fault 0x0 : 0x3e03 : 15875 aida13 fault 0x0 : 0x32d4 : 13012 aida14 fault 0x0 : 0x2 : 2 aida15 fault 0x0 : 0x2 : 2 aida16 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 4, Failed 12 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 53 10 4 4 4 4 2 2 3 3 6 : 36580 aida02 : 28 7 10 4 3 4 3 2 2 4 6 : 37768 aida03 : 41 10 7 6 5 3 1 3 3 3 6 : 36836 aida04 : 45 7 2 4 3 4 2 3 3 3 6 : 36940 aida05 : 35 8 8 4 4 4 2 2 2 4 6 : 37580 aida06 : 45 6 6 3 3 3 2 3 4 3 6 : 37860 aida07 : 37 10 6 4 5 2 3 2 2 4 6 : 37636 aida08 : 32 5 10 4 4 4 3 2 2 4 6 : 37832 aida09 : 24 5 8 7 1 3 3 2 2 4 6 : 37544 aida10 : 29 9 12 5 4 3 3 3 3 3 6 : 37276 aida11 : 33 7 7 6 3 4 3 3 1 4 6 : 37292 aida12 : 36 11 6 6 3 4 3 3 1 4 6 : 37320 aida13 : 37 7 6 6 5 3 3 3 1 4 6 : 37292 aida14 : 25 9 7 6 5 3 3 3 3 3 6 : 37276 aida15 : 3 3 4 3 0 3 3 2 1 4 7 : 40260 aida16 : 35 7 6 6 4 4 3 3 1 4 6 : 37348 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19.07 Hot HEC channels - ASIC check Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents - attachment 7 significant increases of leakage current to c. 12uA - temperature c. 30 deg C Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-16_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 1005.png

Attachment 7: 1006.png

369	Tue Jun 15 12:35:48 2021	TD	Tuesday 15 June
13.35 DAQ continues OK - no storage mode All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 37 10 8 5 4 4 2 2 2 4 6 : 37636 aida02 : 37 10 10 3 4 4 3 2 2 4 6 : 37860 aida03 : 41 8 7 7 5 3 1 3 2 4 6 : 37876 aida04 : 33 9 4 4 2 5 2 3 3 3 6 : 37004 aida05 : 37 9 9 3 5 4 2 2 2 4 6 : 37644 aida06 : 39 9 6 5 4 2 2 3 4 3 6 : 37860 aida07 : 29 12 6 5 4 2 3 2 2 4 6 : 37588 aida08 : 25 7 9 5 4 3 3 2 2 4 6 : 37708 aida09 : 34 5 9 6 2 3 3 2 2 4 6 : 37632 aida10 : 34 12 11 5 4 3 3 3 3 3 6 : 37304 aida11 : 38 8 7 6 3 4 3 3 1 4 6 : 37320 aida12 : 34 10 8 6 3 4 3 3 1 4 6 : 37336 aida13 : 33 11 5 6 4 4 3 3 1 4 6 : 37356 aida14 : 31 10 5 6 5 3 3 3 3 3 6 : 37276 aida15 : 14 6 7 4 1 2 3 2 1 4 7 : 40344 aida16 : 33 8 6 7 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 13.40 All histograms, stats & merger stats zero'd
Attachment 1: Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-15_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1002.png

Attachment 8: 1003.png

368	Tue Jun 15 12:33:46 2021	TD	Monday 14 June

367	Sun Jun 13 17:29:04 2021	TD	Sunday 13 June
18.15 DAQ continues file S496/R32_946 Hot HEC channels multiple FEE64s from c. 06.00 this morning ASIC check fixes Out of disk space on /TapeData filesystem - switch to no storage mode System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 26 16 4 6 4 4 2 2 2 4 6 : 37608 aida02 : 29 11 9 3 3 4 3 2 2 4 6 : 37756 aida03 : 32 9 6 8 5 3 1 3 3 3 6 : 36840 aida04 : 32 8 3 4 3 4 2 3 3 3 6 : 36912 aida05 : 36 8 6 4 5 4 2 2 2 4 6 : 37616 aida06 : 25 13 4 3 2 3 2 3 4 3 6 : 37740 aida07 : 28 13 4 5 5 2 3 2 2 4 6 : 37624 aida08 : 29 5 9 4 3 4 3 2 2 4 6 : 37740 aida09 : 41 8 4 7 1 3 3 2 2 4 6 : 37572 aida10 : 45 11 7 5 5 3 3 3 3 3 6 : 37340 aida11 : 33 13 5 6 3 4 3 3 1 4 6 : 37308 aida12 : 39 8 6 6 4 4 3 3 1 4 6 : 37372 aida13 : 28 14 3 6 4 4 3 3 1 4 6 : 37328 aida14 : 30 11 3 6 4 4 3 3 3 3 6 : 37312 aida15 : 3 3 4 3 0 3 3 2 1 4 7 : 40260 aida16 : 28 9 5 6 5 3 3 3 1 4 6 : 37256 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png

Attachment 2: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png

Attachment 5: Screenshot_2021-06-13_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1000.png

Attachment 8: 1001.png

366	Sat Jun 12 14:53:56 2021	TD	aida15 - system console output - 15.33 Sat 12 June reboot
04/11:03:40\|get_ASICBlk : Blk=1 : bytes = 196096 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500240 : 1 => 0F07C000 04/11:03:41\|Last Data : Databuffer 273286648 : 0 => 80500240 : 1 => 0AFF8000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:41\| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 \| 00000000 : 00000000 04/11:03:42\|Machine check in kernel mode. 12/15:30:08\|I-Cache Parity Error 12/15:30:08\|Oops: Machine check, sig: 7 [#1] 12/15:30:08\|PREEMPT Xilinx Virtex440 12/15:30:08\|Modules linked in: aidamem xdriver xh_spidev_register 12/15:30:08\|NIP: c0055c54 LR: c00558fc CTR: c000ae5c 12/15:30:08\|REGS: c6ff9f10 TRAP: 0214 Not tainted (2.6.31) 12/15:30:08\|MSR: 00021000 <ME,CE> CR: 82000028 XER: 20000000 12/15:30:08\|TASK = c6ac44e0[375] 'AidaExecV9' THREAD: c6338000 12/15:30:08\|GPR00: 00000000 c6339cd0 c6ac44e0 ffffffff ffe79600 ffffffff ffe79600 ffffffff 12/15:30:08\|GPR08: 00000000 0000001f ffffffe0 00000000 00000000 1005520c 1dcd6500 00000000 12/15:30:08\|GPR16: 00000000 1009df98 100f0000 c0390000 000007a1 7fe9e580 ffffffff ffffffff 12/15:30:08\|GPR24: c0390000 00000001 00000000 001366ca c036f0d0 ffffffe1 ffffffff ffec9936 12/15:30:08\|NIP [c0055c54] update_wall_time+0x40c/0xd3c 12/15:30:09\|LR [c00558fc] update_wall_time+0xb4/0xd3c 12/15:30:09\|Call Trace: 12/15:30:09\|[c6339cd0] [c0055f44] update_wall_time+0x6fc/0xd3c (unreliable) 12/15:30:09\|[c6339d90] [c0040b84] do_timer+0x38/0x4c 12/15:30:09\|[c6339da0] [c005a088] tick_periodic+0xbc/0xe8 12/15:30:09\|[c6339db0] [c005a0d4] tick_handle_periodic+0x20/0x120 12/15:30:09\|[c6339df0] [c000af70] timer_interrupt+0xa4/0x10c 12/15:30:09\|[c6339e10] [c000e9c4] ret_from_except+0x0/0x18 12/15:30:09\|[c6339ed0] [d10751dc] aidamem_read+0x28/0x108 [aidamem] 12/15:30:09\|[c6339ef0] [c0097e80] vfs_read+0xb4/0x188 12/15:30:09\|[c6339f10] [c00982f8] sys_read+0x4c/0x90 12/15:30:09\|[c6339f40] [c000e364] ret_from_syscall+0x0/0x3c 12/15:30:09\|Instruction dump: 12/15:30:09\|39600000 354bffe0 418008b0 7cc35030 38800000 2f8a0000 419c08bc 7ffe5030 12/15:30:09\|3aa00000 93c10048 92a1004c 83c10048 <83e1004c> 7d095830 4800056c 419e06bc 12/15:30:09\|Kernel panic - not syncing: Fatal exception in interrupt 12/15:30:09\|Call Trace: 12/15:30:09\|Rebooting in 180 seconds.. ISOL Version 1.00 Date 9th January 2017 Flash base address=FC000000 Set Flash to ASync Mode XST_SUCCESS\| Finished copying zImage to RAM 12/15:33:10\| Found 0 errors checking kernel image 12/15:33:11\|VHDL version number 0X03350706 Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012 Starting LMK 3200 setup 12/15:33:11\| Setting LMK03200 to standard clock settings -- External Clock 23Nov15 .... SPI Base Address=0x81400000 clk_control_reg=0x4 12/15:33:11\|Next step is SPIconfig Control 32(0x81400000)=0x180 SlaveSel(0x81400000)=0x3 Ctrl(0x81400000)=0xE6 Ctrl(0x81400000)=0x86 12/15:33:11\|SPIconfig done now to set up the LMK3200 registers 12/15:33:12\|LMK #0 : regInit[0]=0x80000000 12/15:33:12\|LMK #0 : regInit[1]=0x10070600 12/15:33:12\|LMK #0 : regInit[2]=0x60601 12/15:33:12\|LMK #0 : regInit[3]=0x60602 12/15:33:12\|LMK #0 : regInit[4]=0x60603 12/15:33:12\|LMK #0 : regInit[5]=0x70624 12/15:33:12\|LMK #0 : regInit[6]=0x70605 12/15:33:12\|LMK #0 : regInit[7]=0x70606 12/15:33:12\|LMK #0 : regInit[8]=0x70627 12/15:33:12\|LMK #0 : regInit[9]=0x10000908 12/15:33:12\|LMK #0 : regInit[10]=0xA0022A09 12/15:33:12\|LMK #0 : regInit[11]=0x82800B 12/15:33:12\|LMK #0 : regInit[12]=0x28C800D 12/15:33:12\|LMK #0 : regInit[13]=0x830020E 12/15:33:12\|LMK #0 : regInit[14]=0xC800180F Calibrate completed at 941 counts Setting Clock Control =0x0000000B, to set GOE and sync bit Ctrl @ SPIstop (0x81400000)=0x186 Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004 12/15:33:12\| Finished Clock setup LMK03200 completed LMK 3200 setup Loaded all four ASICs with default settings Setting the ADCs into calibration mode 12/15:33:12\| Control 32(0x81400400)=0x180 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86 Init : Config of AD9252 SPI ok 12/15:33:13\| Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised ADCs calibrated 12/15:33:13\| Control 32(0x81400400)=0x186 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86Config of AD9252 SPI ok 12/15:33:13\| Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot... 12/15:33:13\| zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0) Allocating 0x3b78cc bytes for kernel ... gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes 12/15:33:16\| Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M Finalizing device tree... flat tree at 0xbd1300 Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 12/15:33:23\|Using Xilinx Virtex440 machine description 12/15:33:23\|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011 12/15:33:23\|Zone PFN ranges: 12/15:33:23\| DMA 0x00000000 -> 0x00007000 12/15:33:23\| Normal 0x00007000 -> 0x00007000 12/15:33:23\|Movable zone start PFN for each node 12/15:33:23\|early_node_map[1] active PFN ranges 12/15:33:23\| 0: 0x00000000 -> 0x00007000 12/15:33:23\|MMU: Allocated 1088 bytes of context maps for 255 contexts 12/15:33:23\|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448 12/15:33:24\|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M 12/15:33:24\|PID hash table entries: 512 (order: 9, 2048 bytes) 12/15:33:24\|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) 12/15:33:24\|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) 12/15:33:24\|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init) 12/15:33:24\|Kernel virtual memory layout: 12/15:33:24\| * 0xffffe000..0xfffff000 : fixmap 12/15:33:24\| * 0xfde00000..0xfe000000 : consistent mem 12/15:33:24\| * 0xfde00000..0xfde00000 : early ioremap 12/15:33:24\| * 0xd1000000..0xfde00000 : vmalloc & ioremap 12/15:33:24\|NR_IRQS:512 12/15:33:24\|clocksource: timebase mult[a00000] shift[22] registered 12/15:33:24\|Console: colour dummy device 80x25 12/15:33:24\|Mount-cache hash table entries: 512 12/15:33:24\|NET: Registered protocol family 16 12/15:33:24\|PCI: Probing PCI hardware 12/15:33:24\|bio: create slab <bio-0> at 0 12/15:33:24\|NET: Registered protocol family 2 12/15:33:24\|IP route cache hash table entries: 1024 (order: 0, 4096 bytes) 12/15:33:24\|TCP established hash table entries: 4096 (order: 3, 32768 bytes) 12/15:33:25\|TCP bind hash table entries: 4096 (order: 2, 16384 bytes) 12/15:33:25\|TCP: Hash tables configured (established 4096 bind 4096) 12/15:33:25\|TCP reno registered 12/15:33:25\|NET: Registered protocol family 1 12/15:33:25\|ROMFS MTD (C) 2007 Red Hat, Inc. 12/15:33:25\|msgmni has been set to 214 12/15:33:25\|io scheduler noop registered 12/15:33:25\|io scheduler anticipatory registered 12/15:33:25\|io scheduler deadline registered 12/15:33:25\|io scheduler cfq registered (default) 12/15:33:25\|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 12/15:33:25\|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550 12/15:33:25\|console [ttyS0] enabled 12/15:33:25\|brd: module loaded 12/15:33:25\|loop: module loaded 12/15:33:25\|Device Tree Probing 'ethernet' 12/15:33:25\|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 12/15:33:25\|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode. 12/15:33:25\|XLlTemac: DCR address: 0x80 12/15:33:25\|XLlTemac: buffer descriptor size: 32768 (0x8000) 12/15:33:25\|XLlTemac: Allocating DMA descriptors with kmalloc 12/15:33:25\|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000 12/15:33:26\|XTemac: PHY detected at address 7. 12/15:33:26\|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17 12/15:33:26\|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\| Intel/Sharp Extended Query Table at 0x010A 12/15:33:26\|Using buffer write method 12/15:33:26\|cfi_cmdset_0001: Erase suspend on write enabled 12/15:33:26\|cmdlinepart partition parsing not available 12/15:33:26\|RedBoot partition parsing not available
365	Sat Jun 12 13:56:00 2021	TD	Saturday 12 June
14.11 DAQ continues file S496/R32_697 High data rate from c. 06.00 this morning ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8948 : 4 aida02 fault 0x9b36 : 0x9b3a : 4 aida03 fault 0xf26b : 0xf26f : 4 aida04 fault 0x57a0 : 0x57a4 : 4 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x21be : 8638 aida13 fault 0x0 : 0x2294 : 8852 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 33 10 9 2 5 4 2 2 2 4 6 : 37604 aida02 : 46 12 7 2 4 4 3 2 2 4 6 : 37832 aida03 : 34 10 9 5 4 3 1 3 3 3 6 : 36744 aida04 : 35 12 10 1 3 4 2 3 3 3 6 : 36972 aida05 : 37 9 8 3 4 4 2 2 2 4 6 : 37564 aida06 : 34 10 5 5 4 2 2 3 2 4 6 : 37832 aida07 : 31 11 1 7 4 2 3 2 2 4 6 : 37572 aida08 : 40 7 4 5 3 4 3 2 2 4 6 : 37752 aida09 : 39 9 8 4 1 3 3 2 2 4 6 : 37540 aida10 : 43 12 8 4 4 3 3 3 1 4 6 : 37260 aida11 : 34 14 10 5 3 4 3 3 1 4 6 : 37368 aida12 : 46 12 3 4 4 4 3 3 1 4 6 : 37320 aida13 : 41 11 5 4 5 3 3 3 1 4 6 : 37260 aida14 : 39 10 2 6 4 4 3 3 1 4 6 : 37324 aida15 : 19 12 4 3 2 2 3 2 2 4 6 : 37324 aida16 : 45 10 6 4 5 3 3 3 1 4 6 : 37284 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 15.05 analysis of file S496/R32_698 - attachment 9 max dead time 0.05% aida04 15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365 successful DAQ stop aida15 DAQ reset, setup, go successful DAQ go
Attachment 1: Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-12_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 998.png

Attachment 8: 999.png

Attachment 9: R32_698
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 44970784 ( 15685.0 Hz) Other data format: 216949222 ( 75668.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 62 ( 0.0 Hz) RESUME: 62 ( 0.0 Hz) SYNC100: 42680 ( 14.9 Hz) WR48-63: 42680 ( 14.9 Hz) FEE64 disc: 985985 ( 343.9 Hz) MBS info: 215877753 ( 75294.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2867.115 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2299.418 2 0.673 0.000 3 1.368 0.000 4 0.000 0.000 5 0.000 2495.101 6 0.000 0.000 7 0.000 2566.695 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 467 2 0 0 0 0 1 1 0 0 0 0 1 72420 30 0 0 0 0 15 15 0 0 0 0 2 935 107990737 0 0 33 33 19385 19385 0 107951901 0 0 3 38882631 108955999 0 0 29 29 22052 22052 985985 107925852 0 0 4 496 0 0 0 0 0 0 0 0 0 0 0 5 3224207 1336 0 0 0 0 668 668 0 0 0 0 6 1520 0 0 0 0 0 0 0 0 0 0 0 7 2784837 1118 0 0 0 0 559 559 0 0 0 0 8 473 0 0 0 0 0 0 0 0 0 0 0 9 173 0 0 0 0 0 0 0 0 0 0 0 10 554 0 0 0 0 0 0 0 0 0 0 0 11 317 0 0 0 0 0 0 0 0 0 0 0 12 752 0 0 0 0 0 0 0 0 0 0 0 13 204 0 0 0 0 0 0 0 0 0 0 0 14 302 0 0 0 0 0 0 0 0 0 0 0 15 496 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.945s ( 1068.615 blocks/s, 66.788 Mb/s)
363	Thu Jun 10 11:32:27 2021	TD	Thursday 10 June
12.26 DAQ continues file S496/R32_428 High data rate from c. 06.00 this morning ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 11 6 3 4 4 2 2 2 4 6 : 37540 aida02 : 30 8 7 3 4 4 3 2 2 4 6 : 37768 aida03 : 38 8 13 3 4 3 1 3 3 3 6 : 36744 aida04 : 37 9 7 2 3 4 2 3 3 3 6 : 36940 aida05 : 29 5 9 3 3 3 1 3 2 4 6 : 37580 aida06 : 38 6 2 3 3 3 2 3 2 4 6 : 37768 aida07 : 31 9 2 7 4 2 3 2 2 4 6 : 37572 aida08 : 34 6 3 5 4 4 3 2 2 4 6 : 37768 aida09 : 29 2 12 5 1 3 3 2 2 4 6 : 37540 aida10 : 24 8 15 3 5 3 3 3 1 4 6 : 37296 aida11 : 35 4 15 3 3 4 3 3 1 4 6 : 37308 aida12 : 36 5 3 5 6 4 3 3 1 4 6 : 37384 aida13 : 31 10 4 3 6 4 3 3 1 4 6 : 37356 aida14 : 37 7 1 4 5 4 3 3 1 4 6 : 37276 aida15 : 34 10 6 3 4 1 2 4 2 4 6 : 38168 aida16 : 31 9 3 3 5 4 3 3 1 4 6 : 37268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 202 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 15.07 analysis of file S496/R32_431 - attachment 9 max dead time aida04 0.33%
Attachment 1: Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-10_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 996.png

Attachment 8: 997.png

Attachment 9: R32_431
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 54180182 ( 19758.0 Hz) Other data format: 207739819 ( 75756.7 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 105 ( 0.0 Hz) RESUME: 105 ( 0.0 Hz) SYNC100: 42213 ( 15.4 Hz) WR48-63: 42213 ( 15.4 Hz) FEE64 disc: 1515114 ( 552.5 Hz) MBS info: 206140069 ( 75173.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 6 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2742.196 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2662.854 2 0.384 0.000 3 9.009 0.000 4 0.000 0.000 5 0.000 1805.119 6 0.000 0.000 7 0.000 1797.783 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 559 0 0 0 0 0 0 0 0 0 0 0 1 413404 177 0 0 0 0 84 84 9 0 0 0 2 1153 103268358 0 0 19 19 17963 17963 0 103232394 0 0 3 42054837 104466440 0 0 86 86 21757 21757 1515079 102907675 0 0 4 515 0 0 0 0 0 0 0 0 0 0 0 5 5561752 2359 0 0 0 0 1175 1175 9 0 0 0 6 1738 2 0 0 0 0 1 1 0 0 0 0 7 6142236 2472 0 0 0 0 1232 1232 8 0 0 0 8 565 1 0 0 0 0 0 0 1 0 0 2 9 190 0 0 0 0 0 0 0 0 0 0 0 10 694 0 0 0 0 0 0 0 0 0 0 0 11 417 6 0 0 0 0 0 0 6 0 0 4 12 967 4 0 0 0 0 1 1 2 0 0 0 13 244 0 0 0 0 0 0 0 0 0 0 0 14 285 0 0 0 0 0 0 0 0 0 0 0 15 626 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.160s ( 1097.388 blocks/s, 68.587 Mb/s)
362	Wed Jun 9 19:17:21 2021	TD	FEE64 system console logs

Attachment 1: logs-090621.tar.gz
361	Wed Jun 9 11:27:47 2021	TD	Wednesday 9 June
12.30 DAQ continues OK file S496/R32_373 System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 13 5 12 7 2 4 2 2 1 4 7 : 40572 aida02 : 11 6 8 6 1 3 3 2 1 4 7 : 40540 aida03 : 26 5 11 5 1 2 2 3 2 3 7 : 39712 aida04 : 18 7 8 3 2 3 2 3 2 3 7 : 39776 aida05 : 11 5 9 4 1 2 2 3 1 4 7 : 40612 aida06 : 17 3 6 5 5 2 1 3 1 4 7 : 40604 aida07 : 9 3 4 9 1 3 3 2 1 4 7 : 40540 aida08 : 11 8 5 5 3 3 3 2 1 4 7 : 40604 aida09 : 13 5 8 5 2 3 3 2 1 4 7 : 40572 aida10 : 13 8 13 5 1 2 2 2 1 4 7 : 40228 aida11 : 11 5 14 6 3 3 3 3 2 3 7 : 40244 aida12 : 10 9 5 5 3 3 2 2 1 4 7 : 40352 aida13 : 15 7 7 4 3 4 3 3 2 3 7 : 40228 aida14 : 9 6 7 6 3 4 3 3 2 3 7 : 40260 aida15 : 14 3 8 5 1 2 2 4 1 4 7 : 41136 aida16 : 15 4 9 4 4 4 3 3 2 3 7 : 40300 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 1.8.L specttra - attachments 1 & 2 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3 Lost activity monitor - attachment 4 1.8.W spectra - 20us FSR - attachments 5 & 6 ADC data items - attachment 7 FEE64 temperatures OK - attachment 8 DSSSD bias & leakage currents OK - attachment 9 Merger/Tape Server/Merger statistics - attachment 10 no merger errors reported since previous restart 16.28 analysis of file S496/R32_372 - attachment 11 max deadtime 0.07% aida04
Attachment 1: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png

Attachment 2: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png

Attachment 3: Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png

Attachment 5: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png

Attachment 6: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png

Attachment 7: Screenshot_2021-06-09_Statistics_aidas-gsi.png

Attachment 8: Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png

Attachment 9: 991.png

Attachment 10: 992.png

Attachment 11: R32_372
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 65571173 ( 25420.1 Hz) Other data format: 196348830 ( 76118.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 52 ( 0.0 Hz) RESUME: 52 ( 0.0 Hz) SYNC100: 41608 ( 16.1 Hz) WR48-63: 41608 ( 16.1 Hz) FEE64 disc: 2050316 ( 794.8 Hz) MBS info: 194215194 ( 75291.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 284 ( 0.1 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2579.504 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.015 1853.986 2 0.548 0.000 3 1.886 0.000 4 0.000 0.000 5 0.000 1457.725 6 0.000 0.000 7 0.000 1391.289 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 1323.904 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 836.274 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 6664 532 0 0 0 0 1 1 530 0 0 3 1 4648488 85466 0 0 1 1 947 947 83570 0 0 92 2 5675 97165745 0 0 27 27 16487 16487 7 97132710 0 2 3 47009038 98898439 0 0 24 24 21346 21346 1773239 97082460 0 42 4 3608 4 0 0 0 0 0 0 4 0 0 1 5 6949043 95991 0 0 0 0 1383 1383 93225 0 0 60 6 5001 30 0 0 0 0 1 1 4 24 0 0 7 6897601 102341 0 0 0 0 1436 1436 99469 0 0 74 8 1797 1 0 0 0 0 0 0 1 0 0 1 9 2911 4 0 0 0 0 1 1 2 0 0 3 10 8722 137 0 0 0 0 1 1 135 0 0 0 11 5985 36 0 0 0 0 2 2 32 0 0 4 12 7238 34 0 0 0 0 1 1 32 0 0 0 13 7345 52 0 0 0 0 0 0 52 0 0 0 14 4898 4 0 0 0 0 0 0 4 0 0 1 15 7159 14 0 0 0 0 2 2 10 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 30.055s ( 1064.726 blocks/s, 66.545 Mb/s)
360	Tue Jun 8 10:51:55 2021	TD	Tuesday 8 June
11.52 DAQ continues file S496/R32_340 High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning 11.53 ASIC check System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8947 : 3 aida02 fault 0x9b36 : 0x9b39 : 3 aida03 fault 0xf26b : 0xf26e : 3 aida04 fault 0x57a0 : 0x57a3 : 3 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 3 12 8 3 3 2 2 1 4 7 : 40508 aida02 : 5 5 10 6 1 3 3 2 1 4 7 : 40540 aida03 : 25 7 12 5 1 2 2 3 2 3 7 : 39740 aida04 : 19 6 11 5 1 3 2 3 2 3 7 : 39820 aida05 : 9 6 7 5 0 2 2 3 1 4 7 : 40548 aida06 : 10 8 7 4 3 3 1 3 1 4 7 : 40600 aida07 : 9 3 4 8 1 3 3 2 1 4 7 : 40508 aida08 : 12 5 8 5 3 3 3 2 1 4 7 : 40632 aida09 : 15 4 8 5 1 3 3 2 1 4 7 : 40508 aida10 : 23 5 12 5 1 2 2 2 1 4 7 : 40228 aida11 : 15 9 11 6 2 4 3 3 2 3 7 : 40308 aida12 : 14 9 4 5 3 3 2 2 1 4 7 : 40352 aida13 : 15 9 6 3 4 4 3 3 2 3 7 : 40260 aida14 : 11 5 7 6 4 4 3 3 2 3 7 : 40324 aida15 : 16 8 7 4 1 2 2 4 1 4 7 : 41136 aida16 : 15 8 7 4 3 4 3 3 2 3 7 : 40236 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 16.28 analysis of file S496/R32_339 - attachment 9 max deadtime 0.24% aida04
Attachment 1: Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-08_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 990.png

Attachment 8: 991.png

Attachment 9: R32_339
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 52833088 ( 19139.6 Hz) Other data format: 209086916 ( 75745.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 101 ( 0.0 Hz) RESUME: 101 ( 0.0 Hz) SYNC100: 42281 ( 15.3 Hz) WR48-63: 42281 ( 15.3 Hz) FEE64 disc: 1254898 ( 454.6 Hz) MBS info: 207747254 ( 75259.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2760.405 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2672.516 2 0.610 0.000 3 6.711 0.000 4 0.000 0.000 5 0.000 2111.274 6 0.000 0.000 7 0.000 1910.870 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 589 0 0 0 0 0 0 0 0 0 0 0 1 221336 98 0 0 0 0 49 49 0 0 0 0 2 1118 104025022 0 0 30 30 18193 18193 0 103988576 0 0 3 42352603 105057740 0 0 71 71 22011 22011 1254898 103758678 0 0 4 548 0 0 0 0 0 0 0 0 0 0 0 5 4562177 1856 0 0 0 0 928 928 0 0 0 0 6 1810 0 0 0 0 0 0 0 0 0 0 0 7 5688898 2196 0 0 0 0 1098 1098 0 0 0 0 8 593 0 0 0 0 0 0 0 0 0 0 0 9 215 2 0 0 0 0 1 1 0 0 0 0 10 675 2 0 0 0 0 1 1 0 0 0 0 11 387 0 0 0 0 0 0 0 0 0 0 0 12 964 0 0 0 0 0 0 0 0 0 0 0 13 267 0 0 0 0 0 0 0 0 0 0 0 14 369 0 0 0 0 0 0 0 0 0 0 0 15 539 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.656s ( 1116.685 blocks/s, 69.793 Mb/s)
359	Mon Jun 7 10:33:48 2021	TD	Monday 7 June
11.30 DAQ continues OK - file S496/R32_144 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 5 11 6 3 4 2 2 1 4 7 : 40572 aida02 : 1 3 6 7 2 3 3 2 1 4 7 : 40540 aida03 : 21 7 7 8 2 4 1 3 2 3 7 : 39804 aida04 : 13 7 5 6 1 3 2 3 2 3 7 : 39740 aida05 : 2 8 6 5 1 2 2 3 1 4 7 : 40584 aida06 : 10 4 5 5 4 3 1 3 1 4 7 : 40632 aida07 : 9 3 4 7 3 3 3 2 1 4 7 : 40604 aida08 : 11 2 6 4 4 3 3 2 1 4 7 : 40604 aida09 : 10 2 5 4 2 2 2 3 1 4 7 : 40584 aida10 : 14 5 8 6 2 1 1 3 1 4 7 : 40352 aida11 : 8 6 9 5 4 4 3 3 2 3 7 : 40320 aida12 : 10 5 3 7 3 3 2 2 1 4 7 : 40352 aida13 : 7 5 4 6 5 4 3 3 2 3 7 : 40324 aida14 : 7 5 4 8 4 4 3 3 2 3 7 : 40324 aida15 : 8 4 7 4 2 2 2 4 1 4 7 : 41136 aida16 : 13 5 7 6 3 4 3 3 2 3 7 : 40268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 11.35 analysis of file S496/R32_143 - attachment 9 max deadtime 0.29% aida03
Attachment 1: Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-07_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 986.png

Attachment 8: 987.png

Attachment 9: R32_143
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 57707988 ( 21988.9 Hz) Other data format: 204212013 ( 77812.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 85 ( 0.0 Hz) RESUME: 85 ( 0.0 Hz) SYNC100: 41772 ( 15.9 Hz) WR48-63: 41772 ( 15.9 Hz) FEE64 disc: 1109863 ( 422.9 Hz) MBS info: 203018436 ( 77357.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2624.418 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2432.488 2 0.665 0.000 3 7.525 0.000 4 0.000 0.000 5 0.000 1464.970 6 0.000 0.000 7 0.000 1288.051 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 112 0 0 0 0 0 0 0 0 0 0 0 1 1940572 788 0 0 0 0 394 394 0 0 0 0 2 442 101676817 0 0 30 30 17294 17294 0 101642169 0 0 3 40857770 102528388 0 0 55 55 21074 21074 1109863 101376267 0 0 4 129 0 0 0 0 0 0 0 0 0 0 0 5 7006930 2878 0 0 0 0 1439 1439 0 0 0 0 6 548 0 0 0 0 0 0 0 0 0 0 0 7 7898329 3142 0 0 0 0 1571 1571 0 0 0 0 8 285 0 0 0 0 0 0 0 0 0 0 0 9 71 0 0 0 0 0 0 0 0 0 0 0 10 1135 0 0 0 0 0 0 0 0 0 0 0 11 122 0 0 0 0 0 0 0 0 0 0 0 12 679 0 0 0 0 0 0 0 0 0 0 0 13 92 0 0 0 0 0 0 0 0 0 0 0 14 666 0 0 0 0 0 0 0 0 0 0 0 15 106 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.551s ( 1082.882 blocks/s, 67.680 Mb/s)
358	Sun Jun 6 14:15:05 2021	TD	Sunday 6 June
14.15 DAQ continues file S496/R32_117 Hot HEC channels multiple FEE64s from c. 06.00 this morning ASIC check fixes System wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 12 12 3 3 4 2 2 1 4 7 : 40572 aida02 : 7 8 10 5 2 3 3 2 1 4 7 : 40604 aida03 : 27 10 4 8 1 4 1 3 2 3 7 : 39740 aida04 : 18 7 4 5 1 3 2 3 2 3 7 : 39712 aida05 : 15 13 8 4 1 3 1 3 1 4 7 : 40548 aida06 : 14 8 8 6 3 2 1 3 1 4 7 : 40568 aida07 : 11 4 5 6 2 3 3 2 1 4 7 : 40540 aida08 : 15 6 5 3 4 3 3 2 1 4 7 : 40604 aida09 : 18 4 4 3 3 2 2 3 1 4 7 : 40648 aida10 : 12 6 8 6 1 1 1 3 1 4 7 : 40288 aida11 : 16 8 6 6 4 4 3 3 2 3 7 : 40352 aida12 : 10 7 4 8 3 4 3 3 2 3 7 : 40288 aida13 : 15 7 5 4 5 4 3 3 2 3 7 : 40324 aida14 : 9 4 6 7 3 4 3 3 2 3 7 : 40260 aida15 : 12 4 8 3 2 2 2 4 1 4 7 : 41136 aida16 : 11 8 6 6 3 4 3 3 2 3 7 : 40268 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 16.28 analysis of file S496/R32_118 - attachment 9 max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-06_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 984.png

Attachment 8: 985.png

Attachment 9: R32_118
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 48860593 ( 17844.1 Hz) Other data format: 213059412 ( 77810.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 101 ( 0.0 Hz) RESUME: 101 ( 0.0 Hz) SYNC100: 42200 ( 15.4 Hz) WR48-63: 42200 ( 15.4 Hz) FEE64 disc: 1112345 ( 406.2 Hz) MBS info: 211862465 ( 77373.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2738.188 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2613.952 2 0.575 0.000 3 7.745 0.000 4 0.000 0.000 5 0.000 2154.815 6 0.000 0.000 7 0.000 1921.162 8 0.000 777.447 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 155 0 0 0 0 0 0 0 0 0 0 0 1 66625 28 0 0 0 0 14 14 0 0 0 0 2 415 106108157 0 0 26 26 18899 18899 0 106070307 0 0 3 39298810 106947309 0 0 75 75 21328 21328 1112345 105792158 0 0 4 171 0 0 0 0 0 0 0 0 0 0 0 5 4255149 1684 0 0 0 0 842 842 0 0 0 0 6 596 0 0 0 0 0 0 0 0 0 0 0 7 5234928 2226 0 0 0 0 1113 1113 0 0 0 0 8 493 4 0 0 0 0 2 2 0 0 0 0 9 92 0 0 0 0 0 0 0 0 0 0 0 10 1212 2 0 0 0 0 1 1 0 0 0 0 11 160 0 0 0 0 0 0 0 0 0 0 0 12 781 0 0 0 0 0 0 0 0 0 0 0 13 89 0 0 0 0 0 0 0 0 0 0 0 14 724 2 0 0 0 0 1 1 0 0 0 0 15 193 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.738s ( 1113.497 blocks/s, 69.594 Mb/s)
357	Sat Jun 5 11:20:15 2021	TD	Saturday 5 June
12.25 DAQ continues OK - file S496/R32_48 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8946 : 2 aida02 fault 0x9b36 : 0x9b38 : 2 aida03 fault 0xf26b : 0xf26d : 2 aida04 fault 0x57a0 : 0x57a2 : 2 aida05 fault 0x4d3c : 0x4d3d : 1 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640d : 1 aida07 fault 0x255e : 0x255f : 1 aida08 fault 0xffbd : 0xffbe : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 10 6 1 2 4 4 2 2 1 4 7 : 40360 aida02 : 4 3 3 3 3 4 2 2 1 4 7 : 40312 aida03 : 13 5 4 4 2 4 1 3 2 3 7 : 39580 aida04 : 16 5 3 3 2 4 1 3 2 3 7 : 39544 aida05 : 8 6 1 2 3 3 1 3 1 4 7 : 40416 aida06 : 4 3 2 1 3 3 1 3 1 4 7 : 40360 aida07 : 6 6 2 4 4 2 3 2 1 4 7 : 40424 aida08 : 6 4 1 1 4 3 3 2 1 4 7 : 40424 aida09 : 16 1 3 1 0 2 2 3 1 4 7 : 40344 aida10 : 2 5 1 4 1 3 2 4 2 3 7 : 40064 aida11 : 14 3 2 1 4 4 3 3 2 3 7 : 40080 aida12 : 11 2 2 4 4 4 3 3 2 3 7 : 40156 aida13 : 11 3 2 2 3 4 3 3 2 3 7 : 40036 aida14 : 6 5 0 4 3 4 3 3 2 3 7 : 40064 aida15 : 11 6 3 1 3 1 2 4 1 4 7 : 40940 aida16 : 12 5 0 4 4 4 3 3 2 3 7 : 40152 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 1.8.W spectra - 20us FSR - attachments 2 & 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart analysis of file S496/R32_47 - attachment 8 max deadtime 0.35% aida04 13.56 Lost activity monitor - see attachment 9
Attachment 1: Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-06-05_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 982.png

Attachment 7: 983.png

Attachment 8: R32_47
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 47883982 ( 17391.8 Hz) Other data format: 214036019 ( 77739.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 184 ( 0.1 Hz) RESUME: 184 ( 0.1 Hz) SYNC100: 42255 ( 15.3 Hz) WR48-63: 42255 ( 15.3 Hz) FEE64 disc: 1053753 ( 382.7 Hz) MBS info: 212897388 ( 77325.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2753.252 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2696.425 2 0.825 0.000 3 9.735 0.000 4 0.000 0.000 5 0.000 2175.120 6 0.000 0.000 7 0.000 2003.563 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 187 0 0 0 0 0 0 0 0 0 0 0 1 86597 40 0 0 0 0 20 20 0 0 0 0 2 381 106659018 0 0 37 37 18746 18746 0 106621452 0 0 3 38584230 107373193 0 0 147 147 21605 21605 1053753 106275936 0 0 4 143 0 0 0 0 0 0 0 0 0 0 0 5 4208337 1756 0 0 0 0 878 878 0 0 0 0 6 545 0 0 0 0 0 0 0 0 0 0 0 7 5000124 2012 0 0 0 0 1006 1006 0 0 0 0 8 478 0 0 0 0 0 0 0 0 0 0 0 9 96 0 0 0 0 0 0 0 0 0 0 0 10 1149 0 0 0 0 0 0 0 0 0 0 0 11 117 0 0 0 0 0 0 0 0 0 0 0 12 706 0 0 0 0 0 0 0 0 0 0 0 13 105 0 0 0 0 0 0 0 0 0 0 0 14 636 0 0 0 0 0 0 0 0 0 0 0 15 151 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.996s ( 1066.806 blocks/s, 66.675 Mb/s)
Attachment 9: Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png

Goto page Previous 1, 2, 3 ... 16, 17, 18 ... 33, 34, 35 Next

ELOG V3.1.4-unknown