AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC  ELOG logo
Message ID: 345     Entry time: Sat May 22 23:40:32 2021
Author: TD 
Subject: Sunday 23 May 
00.40 Unable to connect to FEE64s 1-7
      DAQ power cycled OK

      ASIC settings 2021Apr29-13-16-00
      slow comparator 0x64 *all* FEE64s

      BNC PB-5 OFF

      file S496/R16

      alpha background

07.17 Some hot HEC channels
      ASIC check fixes issue
      System wide checks OK *except*

		 Base 		Current 	Difference
aida06 fault 	 0x5457 : 	 0x5457 : 	 0  
aida06 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1cd : 	 461  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      3      1      4      3      2      1      4      7   : 40552
aida02 :      3      2      2      1      2      3      3      2      2      4      7   : 41308
aida03 :     14      5      0      1      1      3      2      4      2      3      7   : 40000
aida04 :     14      4      5      1      3      3      1      4      2      3      7   : 39944
aida05 :     12      4      1      3      2      3      3      2      2      4      7   : 41408
aida06 :      8      3      2      1      2      4      2      3      3      4      7   : 42744
aida07 :     10      4      3      2      1      2      2      3      2      4      7   : 41464
aida08 :      7      5      2      0      3      3      3      2      2      4      7   : 41380
aida09 :     10      5      1      1      1      2      2      3      2      4      7   : 41408
aida10 :     11      6      3      2      1      3      2      2      2      4      7   : 41100
aida11 :     12      8      3      2      2      2      3      2      1      4      7   : 40288
aida12 :      9      9      3      1      2      3      3      2      1      4      7   : 40380
aida13 :     10      4      0      1      1      2      1      3      2      4      7   : 41128
aida14 :      9      4      2      1      1      2      1      3      2      4      7   : 41156
aida15 :      5      4      0      3      3      3      3      3      2      4      7   : 41940
aida16 :      9      6      3      2      0      2      1      3      2      4      7   : 41156

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 00:29:48 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      FEE64 temperatures OK - attachment 1
      ADC data item stats - attachment 2
      Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3
      DSSSD bias & leakage currents OK - attachment 4

11.18 Analysis of file S496/R16_33 - attachment 5

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  685M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.0T  867G  88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

867/2 x 3279s = 16.5 days to filesystem full - OK

12.11 aida15 zero data/poll stats - re-loaded DAQ program OK
      aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK

      DAQ continues file S496/R18
Attachment 1: Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png  348 kB  Uploaded Sun May 23 07:21:09 2021  | Hide | Hide all | Show all
Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-23_Statistics_aidas-gsi.png  83 kB  Uploaded Sun May 23 07:21:09 2021  | Hide | Hide all | Show all
Screenshot_2021-05-23_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png  130 kB  Uploaded Sun May 23 07:21:09 2021  | Hide | Hide all | Show all
Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: 810.png  11 kB  Uploaded Sun May 23 07:21:40 2021  | Hide | Hide all | Show all
810.png
Attachment 5: R16_33  11 kB  Uploaded Sun May 23 11:49:57 2021  | Show | Hide all | Show all
ELOG V3.1.4-unknown