AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC  ELOG logo
Message ID: 354     Entry time: Thu Jun 3 10:16:39 2021
Author: TD 
Subject: Thursday 3 June 
11.19 Hot HEC channels multiple FEE64s
      ASIC check fixes

      System wide checks OK * except*

		 Base 		Current 	Difference
aida01 fault 	 0x8940 : 	 0x8941 : 	 1  
aida02 fault 	 0x9b32 : 	 0x9b33 : 	 1  
aida03 fault 	 0xf266 : 	 0xf267 : 	 1  
aida04 fault 	 0x579b : 	 0x579c : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      4      2      2      2      2      2      3      2      4      7   : 41516
aida02 :     10      7      2      0      1      2      2      3      2      4      7   : 41408
aida03 :     10      8      2      2      2      4      1      4      2      3      7   : 40008
aida04 :      5      4      4      2      2      3      1      4      2      3      7   : 39860
aida05 :     12      3      2      2      1      2      2      3      2      4      7   : 41448
aida06 :     15      9      2      2      3      2      1      3      2      4      7   : 41380
aida07 :     12      8      1      2      2      3      3      2      2      4      7   : 41408
aida08 :      9      6      3      1      2      3      3      2      2      4      7   : 41380
aida09 :     15      6      1      1      2      2      2      3      2      4      7   : 41500
aida10 :     15      6      3      2      1      3      3      2      1      4      7   : 40348
aida11 :      8      2      1      1      1      3      1      3      2      4      7   : 41248
aida12 :     18      3      2      1      3      3      2      2      2      4      7   : 41184
aida13 :      7      4      1      4      3      4      3      3      1      4      7   : 41100
aida14 :     18      6      1      4      3      4      2      2      1      4      7   : 40392
aida15 :      7      7      1      3      1      2      1      3      2      4      7   : 41220
aida16 :      4      4      2      3      3      3      2      4      1      4      7   : 41200

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Jun 02 16:27:30 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36

13.09 analysis of file S496/R31_16 - attachment 8
       max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png  284 kB  Uploaded Thu Jun 3 11:27:50 2021  | Hide | Hide all
Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png  285 kB  Uploaded Thu Jun 3 11:27:50 2021  | Hide | Hide all
Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png  366 kB  Uploaded Thu Jun 3 11:27:50 2021  | Hide | Hide all
Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-06-03_Statistics_aidas-gsi.png  96 kB  Uploaded Thu Jun 3 11:27:50 2021  | Hide | Hide all
Screenshot_2021-06-03_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png  146 kB  Uploaded Thu Jun 3 11:27:50 2021  | Hide | Hide all
Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 970.png  11 kB  Uploaded Thu Jun 3 11:28:20 2021  | Hide | Hide all
970.png
Attachment 7: 971.png  611 kB  Uploaded Thu Jun 3 11:28:20 2021  | Hide | Hide all
971.png
Attachment 8: R31_16  11 kB  Uploaded Thu Jun 3 13:45:16 2021  | Hide | Hide all
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   50135749 (   17884.4 Hz)
        Other data format:  211784253 (   75547.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        199 (       0.1 Hz)
                               RESUME:        199 (       0.1 Hz)
                              SYNC100:      42443 (      15.1 Hz)
                              WR48-63:      42443 (      15.1 Hz)
                           FEE64 disc:    1081490 (     385.8 Hz)
                             MBS info:  210617479 (   75131.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          4 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2803.320 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2522.967
  2                0.535                0.000
  3                8.636                0.000
  4                0.000                0.000
  5                0.000             2119.634
  6                0.000                0.000
  7                0.000             2067.036
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        196          2          0          0          0          0          0          0          2          0          0          2
  1     193203         74          0          0          0          0         37         37          0          0          0          0
  2        471  105497692          0          0         26         26      18605      18605          0  105460430          0          0
  3   40159698  106282506          0          0        173        173      21812      21812    1081487  105157049          0          0
  4        128          0          0          0          0          0          0          0          0          0          0          0
  5    4517663       1914          0          0          0          0        957        957          0          0          0          0
  6        641          2          0          0          0          0          1          1          0          0          0          0
  7    5260254       2060          0          0          0          0       1030       1030          0          0          0          0
  8        463          0          0          0          0          0          0          0          0          0          0          0
  9        104          0          0          0          0          0          0          0          0          0          0          0
 10       1026          2          0          0          0          0          1          1          0          0          0          0
 11        141          1          0          0          0          0          0          0          1          0          0          1
 12        786          0          0          0          0          0          0          0          0          0          0          0
 13        109          0          0          0          0          0          0          0          0          0          0          0
 14        687          0          0          0          0          0          0          0          0          0          0          0
 15        179          0          0          0          0          0          0          0          0          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.445s ( 1124.966 blocks/s,  70.310 Mb/s)
ELOG V3.1.4-unknown