AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC  ELOG logo
Message ID: 346     Entry time: Mon May 24 09:57:07 2021
Author: TD 
Subject: Monday 24 May 
07.30 Hot HEC channels multiple FEE64s
      ASIC check fixes

10.56 DAQ continues OK file S496/R18_75

      System wide checks OK * except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      2      7      2      1      4      3      2      1      4      7   : 40452
aida02 :     13      3      3      1      3      4      2      2      2      4      7   : 41308
aida03 :     17      6      2      2      1      4      2      3      2      3      7   : 39700
aida04 :      8      6      1      0      1      3      1      4      2      3      7   : 39712
aida05 :     10      3      5      3      4      3      3      2      1      4      7   : 40560
aida06 :      4      4      2      0      2      3      2      3      3      4      7   : 42576
aida07 :      5      3      2      3      1      3      1      3      2      4      7   : 41324
aida08 :     11      5      1      1      2      3      3      2      2      4      7   : 41348
aida09 :      7      4      3      0      2      3      1      3      2      4      7   : 41324
aida10 :     11      4      2      2      2      3      2      4      1      4      7   : 41132
aida11 :     18      5      0      2      2      4      2      2      1      4      7   : 40240
aida12 :      2      3      0      1      2      3      2      2      2      4      7   : 41024
aida13 :      4      5      1      2      2      3      2      4      1      4      7   : 41096
aida14 :      9      3      1      2      1      4      2      2      2      4      7   : 41164
aida15 :      2      3      1      3      2      2      3      3      2      4      7   : 41744
aida16 :      5      5      2      0      3      3      2      4      1      4      7   : 41116

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
      1.8.H spectra - attachments 6-8
      1.8.L spectra - attachments 9 & 10
      Stat spectra - attachments 11 & 12
      1.8.W spectra - 20us FSR - attachments 13 & 14
      ADC data items - attachment 15
      FEE64 temperatures OK - attachment 16
      DSSSD bias & leakage currents OK - attachment 17
      Merger/Tape Server/Merger statistics - attachment 18
       no merger errors reported since previous restart

      analysis of file R18_74
       max dead time 0.05%

10.17 ASIC check
      All histograms, statistics, merger statistics zero'd
Attachment 1: Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png  358 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-24_Statistics_aidas-gsi(4).png  80 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Statistics_aidas-gsi(4).png
Attachment 3: Screenshot_2021-05-24_Statistics_aidas-gsi(3).png  78 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Statistics_aidas-gsi(3).png
Attachment 4: Screenshot_2021-05-24_Statistics_aidas-gsi(2).png  91 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-24_Statistics_aidas-gsi(1).png  83 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png  155 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png
Attachment 7: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png  171 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png
Attachment 8: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png  129 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png
Attachment 9: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png  136 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png
Attachment 10: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png  161 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png
Attachment 11: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png  204 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png
Attachment 12: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png  204 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png
Attachment 13: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png  183 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png
Attachment 14: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png  264 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-05-24_Statistics_aidas-gsi.png  85 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Statistics_aidas-gsi.png
Attachment 16: Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png  130 kB  Uploaded Mon May 24 11:19:59 2021  | Hide | Hide all
Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png
Attachment 17: 900.png  11 kB  Uploaded Mon May 24 11:20:34 2021  | Hide | Hide all
900.png
Attachment 18: 901.png  518 kB  Uploaded Mon May 24 11:20:34 2021  | Hide | Hide all
901.png
Attachment 19: R18_74  11 kB  Uploaded Mon May 24 11:22:13 2021  | Hide | Hide all
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   42982076 (   14466.2 Hz)
        Other data format:  218937925 (   73686.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         59 (       0.0 Hz)
                               RESUME:         59 (       0.0 Hz)
                              SYNC100:      43069 (      14.5 Hz)
                              WR48-63:      43069 (      14.5 Hz)
                           FEE64 disc:    1163669 (     391.6 Hz)
                             MBS info:  217688000 (   73265.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2971.209 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.425                0.000
  3                1.488                0.000
  4                0.000                0.000
  5                0.000             2622.333
  6                0.000                0.000
  7                0.000             2340.476
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         64          0          0          0          0          0          0          0          0          0          0          0
  1       2227          3          0          0          0          0          1          1          1          0          0          0
  2         75  108903724          0          0         25         25      19964      19964          0  108863746          0          0
  3   35357926  110031040          0          0         34         34      21525      21525    1163668  108824254          0          0
  4         89          0          0          0          0          0          0          0          0          0          0          0
  5    3145189       1316          0          0          0          0        658        658          0          0          0          0
  6         64          0          0          0          0          0          0          0          0          0          0          0
  7    4475895       1842          0          0          0          0        921        921          0          0          0          0
  8         56          0          0          0          0          0          0          0          0          0          0          0
  9         67          0          0          0          0          0          0          0          0          0          0          0
 10        115          0          0          0          0          0          0          0          0          0          0          0
 11         65          0          0          0          0          0          0          0          0          0          0          0
 12         65          0          0          0          0          0          0          0          0          0          0          0
 13         77          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        102          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   40.766s (  784.975 blocks/s,  49.061 Mb/s)
ELOG V3.1.4-unknown