ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Sat Jun 12 13:56:00 2021

Author*:

Subject*:

14.11 DAQ continues file S496/R32_697
      High data rate from c. 06.00 this morning
      ASIC check

System wide checks OK *except*

Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8948 : 	 4  
aida02 fault 	 0x9b36 : 	 0x9b3a : 	 4  
aida03 fault 	 0xf26b : 	 0xf26f : 	 4  
aida04 fault 	 0x57a0 : 	 0x57a4 : 	 4  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x21be : 	 8638  
aida13 fault 	 0x0 : 	 0x2294 : 	 8852  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     10      9      2      5      4      2      2      2      4      6   : 37604
aida02 :     46     12      7      2      4      4      3      2      2      4      6   : 37832
aida03 :     34     10      9      5      4      3      1      3      3      3      6   : 36744
aida04 :     35     12     10      1      3      4      2      3      3      3      6   : 36972
aida05 :     37      9      8      3      4      4      2      2      2      4      6   : 37564
aida06 :     34     10      5      5      4      2      2      3      2      4      6   : 37832
aida07 :     31     11      1      7      4      2      3      2      2      4      6   : 37572
aida08 :     40      7      4      5      3      4      3      2      2      4      6   : 37752
aida09 :     39      9      8      4      1      3      3      2      2      4      6   : 37540
aida10 :     43     12      8      4      4      3      3      3      1      4      6   : 37260
aida11 :     34     14     10      5      3      4      3      3      1      4      6   : 37368
aida12 :     46     12      3      4      4      4      3      3      1      4      6   : 37320
aida13 :     41     11      5      4      5      3      3      3      1      4      6   : 37260
aida14 :     39     10      2      6      4      4      3      3      1      4      6   : 37324
aida15 :     19     12      4      3      2      2      3      2      2      4      6   : 37324
aida16 :     45     10      6      4      5      3      3      3      1      4      6   : 37284

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.05 analysis of file S496/R32_698 - attachment 9
      max dead time 0.05% aida04

15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365

successful DAQ stop
      aida15 DAQ reset, setup, go 
      successful DAQ go

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1:

Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png Original size: 2108x1310

Attachment 2:

Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png Original size: 2400x1159

Attachment 3:

Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png Original size: 2385x1163

Attachment 4:

Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png Original size: 2400x1160

Attachment 5:

Screenshot_2021-06-12_Statistics_aidas-gsi.png Original size: 2400x1159

Attachment 6:

Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png Original size: 2400x1159

Attachment 7:

998.png Original size: 502x344

Attachment 8:

999.png Original size: 3840x1080

Attachment 9:

R32_698

 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   44970784 (   15685.0 Hz)
        Other data format:  216949222 (   75668.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         62 (       0.0 Hz)
                               RESUME:         62 (       0.0 Hz)
                              SYNC100:      42680 (      14.9 Hz)
                              WR48-63:      42680 (      14.9 Hz)
                           FEE64 disc:     985985 (     343.9 Hz)
                             MBS info:  215877753 (   75294.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2867.115 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2299.418
  2                0.673                0.000
  3                1.368                0.000
  4                0.000                0.000
  5                0.000             2495.101
  6                0.000                0.000
  7                0.000             2566.695
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        467          2          0          0          0          0          1          1          0          0          0          0
  1      72420         30          0          0          0          0         15         15          0          0          0          0
  2        935  107990737          0          0         33         33      19385      19385          0  107951901          0          0
  3   38882631  108955999          0          0         29         29      22052      22052     985985  107925852          0          0
  4        496          0          0          0          0          0          0          0          0          0          0          0
  5    3224207       1336          0          0          0          0        668        668          0          0          0          0
  6       1520          0          0          0          0          0          0          0          0          0          0          0
  7    2784837       1118          0          0          0          0        559        559          0          0          0          0
  8        473          0          0          0          0          0          0          0          0          0          0          0
  9        173          0          0          0          0          0          0          0          0          0          0          0
 10        554          0          0          0          0          0          0          0          0          0          0          0
 11        317          0          0          0          0          0          0          0          0          0          0          0
 12        752          0          0          0          0          0          0          0          0          0          0          0
 13        204          0          0          0          0          0          0          0          0          0          0          0
 14        302          0          0          0          0          0          0          0          0          0          0          0
 15        496          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.945s ( 1068.615 blocks/s,  66.788 Mb/s)

Attachment 10:

Drop attachments here...

ELOG V3.1.4-unknown