ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Sat May 22 23:40:32 2021

Author*:

Subject*:

00.40 Unable to connect to FEE64s 1-7
      DAQ power cycled OK

ASIC settings 2021Apr29-13-16-00
      slow comparator 0x64 *all* FEE64s

BNC PB-5 OFF

file S496/R16

alpha background

07.17 Some hot HEC channels
      ASIC check fixes issue
      System wide checks OK *except*

Base 		Current 	Difference
aida06 fault 	 0x5457 : 	 0x5457 : 	 0  
aida06 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1cd : 	 461  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      3      1      4      3      2      1      4      7   : 40552
aida02 :      3      2      2      1      2      3      3      2      2      4      7   : 41308
aida03 :     14      5      0      1      1      3      2      4      2      3      7   : 40000
aida04 :     14      4      5      1      3      3      1      4      2      3      7   : 39944
aida05 :     12      4      1      3      2      3      3      2      2      4      7   : 41408
aida06 :      8      3      2      1      2      4      2      3      3      4      7   : 42744
aida07 :     10      4      3      2      1      2      2      3      2      4      7   : 41464
aida08 :      7      5      2      0      3      3      3      2      2      4      7   : 41380
aida09 :     10      5      1      1      1      2      2      3      2      4      7   : 41408
aida10 :     11      6      3      2      1      3      2      2      2      4      7   : 41100
aida11 :     12      8      3      2      2      2      3      2      1      4      7   : 40288
aida12 :      9      9      3      1      2      3      3      2      1      4      7   : 40380
aida13 :     10      4      0      1      1      2      1      3      2      4      7   : 41128
aida14 :      9      4      2      1      1      2      1      3      2      4      7   : 41156
aida15 :      5      4      0      3      3      3      3      3      2      4      7   : 41940
aida16 :      9      6      3      2      0      2      1      3      2      4      7   : 41156

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 00:29:48 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

FEE64 temperatures OK - attachment 1
      ADC data item stats - attachment 2
      Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3
      DSSSD bias & leakage currents OK - attachment 4

11.18 Analysis of file S496/R16_33 - attachment 5

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  685M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.0T  867G  88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

867/2 x 3279s = 16.5 days to filesystem full - OK

12.11 aida15 zero data/poll stats - re-loaded DAQ program OK
      aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK

DAQ continues file S496/R18

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1:

Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png Original size: 2108x1310

Attachment 2:

Screenshot_2021-05-23_Statistics_aidas-gsi.png Original size: 1920x927

Attachment 3:

Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png Original size: 1920x927

Attachment 4:

810.png Original size: 502x344

Attachment 5:

R16_33

 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   47385414 (   14450.2 Hz)
        Other data format:  214534594 (   65422.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         61 (       0.0 Hz)
                               RESUME:         61 (       0.0 Hz)
                              SYNC100:      44215 (      13.5 Hz)
                              WR48-63:      44215 (      13.5 Hz)
                           FEE64 disc:    1311957 (     400.1 Hz)
                             MBS info:  213134085 (   64995.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3279.230 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.386                0.000
  3                1.000                0.000
  4                0.000                0.000
  5                0.000             2849.824
  6                0.000                0.000
  7                0.000             2608.689
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         71          0          0          0          0          0          0          0          0          0          0          0
  1       1455          0          0          0          0          0          0          0          0          0          0          0
  2         87  106617422          0          0         35         35      20186      20186          0  106576980          0          0
  3   39184250  107913590          0          0         26         26      22238      22238    1311957  106557105          0          0
  4         63          0          0          0          0          0          0          0          0          0          0          0
  5    3345685       1512          0          0          0          0        756        756          0          0          0          0
  6         60          0          0          0          0          0          0          0          0          0          0          0
  7    4853274       2070          0          0          0          0       1035       1035          0          0          0          0
  8         69          0          0          0          0          0          0          0          0          0          0          0
  9         45          0          0          0          0          0          0          0          0          0          0          0
 10         70          0          0          0          0          0          0          0          0          0          0          0
 11         68          0          0          0          0          0          0          0          0          0          0          0
 12         89          0          0          0          0          0          0          0          0          0          0          0
 13         57          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15         71          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.293s (  722.462 blocks/s,  45.154 Mb/s)

Attachment 6:

Drop attachments here...

ELOG V3.1.4-unknown