AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC  ELOG logo
Message ID: 197     Entry time: Sat Mar 13 23:08:08 2021
Author: CA 
Subject: 14th March 00:00-08:00 
00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC
Attachment 1: Screenshot_from_2021-03-13_23-14-33.png  158 kB  | Hide | Hide all
Screenshot_from_2021-03-13_23-14-33.png
Attachment 2: Screenshot_from_2021-03-13_23-14-57.png  119 kB  | Hide | Hide all
Screenshot_from_2021-03-13_23-14-57.png
Attachment 3: Screenshot_from_2021-03-13_23-15-12.png  46 kB  | Hide | Hide all
Screenshot_from_2021-03-13_23-15-12.png
Attachment 4: Screenshot_from_2021-03-14_01-22-56.png  159 kB  Uploaded Sun Mar 14 01:30:24 2021  | Hide | Hide all
Screenshot_from_2021-03-14_01-22-56.png
Attachment 5: Screenshot_from_2021-03-14_01-29-35.png  121 kB  Uploaded Sun Mar 14 01:30:24 2021  | Hide | Hide all
Screenshot_from_2021-03-14_01-29-35.png
Attachment 6: Screenshot_from_2021-03-14_01-29-52.png  46 kB  Uploaded Sun Mar 14 01:30:24 2021  | Hide | Hide all
Screenshot_from_2021-03-14_01-29-52.png
Attachment 7: Screenshot_from_2021-03-14_01-40-57.png  178 kB  Uploaded Sun Mar 14 01:42:03 2021  | Hide | Hide all
Screenshot_from_2021-03-14_01-40-57.png
Attachment 8: Screenshot_from_2021-03-14_02-16-28.png  159 kB  Uploaded Sun Mar 14 02:18:48 2021  | Hide | Hide all
Screenshot_from_2021-03-14_02-16-28.png
Attachment 9: Screenshot_from_2021-03-14_02-17-01.png  118 kB  Uploaded Sun Mar 14 02:18:48 2021  | Hide | Hide all
Screenshot_from_2021-03-14_02-17-01.png
Attachment 10: Screenshot_from_2021-03-14_02-17-32.png  46 kB  Uploaded Sun Mar 14 02:18:48 2021  | Hide | Hide all
Screenshot_from_2021-03-14_02-17-32.png
Attachment 11: Screenshot_from_2021-03-14_02-29-29.png  228 kB  Uploaded Sun Mar 14 02:30:24 2021  | Hide | Hide all
Screenshot_from_2021-03-14_02-29-29.png
Attachment 12: R56_35  10 kB  Uploaded Sun Mar 14 02:50:54 2021  | Hide | Hide all
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261409816 ( 1907248.6 Hz)
        Other data format:     510184 (    3722.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2488 (      18.2 Hz)
                               RESUME:       2487 (      18.1 Hz)
                              SYNC100:        511 (       3.7 Hz)
                              WR48-63:        511 (       3.7 Hz)
                           FEE64 disc:     503410 (    3672.9 Hz)
                             MBS info:        777 (       5.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      60911 (     444.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      137.061 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.725              132.339
  2                3.132              135.291
  3                6.180              132.876
  4               12.844              128.581
  5                8.386              132.070
  6                1.216              132.607
  7                1.093              115.159
  8               50.447              129.654
  9               19.940              134.755
 10                0.568              132.339
 11               51.844              133.681
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22917695      26526          0          0        878        878         65         65      23863        777          0       8590
  1   26593258      35084          0          0         13         13         51         51      34956          0          0      10875
  2   25634005     337148          0          0         16         16         76         76     336964          0          0       6926
  3   28548879       3343          0          0         34         34         27         27       3221          0          0       1214
  4    7607641      18967          0          0        663        663         14         14      17613          0          0       6303
  5   19366719      25239          0          0        305        304         24         24      24582          0          0       8037
  6   18601227      17453          0          0          3          3         18         18      17411          0          0       6563
  7   10076045       2592          0          0          4          4         24         24       2536          0          0        908
  8   30026924       8042          0          0        217        216         34         34       7541          0          0       2274
  9   30858440      18084          0          0        113        113         88         88      17682          0          0       4590
 10    9946552      15820          0          0          3          3         46         46      15722          0          0       4264
 11   31232431       1886          0          0        239        240         44         44       1319          0          0        367
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:13611.407s (    2.351 blocks/s,   0.147 Mb/s)
Attachment 13: Screenshot_from_2021-03-14_04-20-07.png  118 kB  Uploaded Sun Mar 14 04:22:11 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-20-07.png
Attachment 14: Screenshot_from_2021-03-14_04-20-34.png  159 kB  Uploaded Sun Mar 14 04:22:11 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-20-34.png
Attachment 15: Screenshot_from_2021-03-14_04-20-50.png  47 kB  Uploaded Sun Mar 14 04:22:11 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-20-50.png
Attachment 16: Screenshot_from_2021-03-14_04-25-41.png  219 kB  Uploaded Sun Mar 14 04:26:28 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-25-41.png
Attachment 17: Screenshot_from_2021-03-14_04-32-03.png  226 kB  Uploaded Sun Mar 14 04:34:41 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-32-03.png
Attachment 18: Screenshot_from_2021-03-14_04-32-32.png  223 kB  Uploaded Sun Mar 14 04:34:41 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-32-32.png
Attachment 19: Screenshot_from_2021-03-14_04-33-14.png  164 kB  Uploaded Sun Mar 14 04:34:41 2021  | Hide | Hide all
Screenshot_from_2021-03-14_04-33-14.png
Attachment 20: Screenshot_from_2021-03-14_06-10-39.png  158 kB  Uploaded Sun Mar 14 06:11:58 2021  | Hide | Hide all
Screenshot_from_2021-03-14_06-10-39.png
Attachment 21: Screenshot_from_2021-03-14_06-11-19.png  120 kB  Uploaded Sun Mar 14 06:11:58 2021  | Hide | Hide all
Screenshot_from_2021-03-14_06-11-19.png
Attachment 22: Screenshot_from_2021-03-14_06-11-38_-_1.png  46 kB  Uploaded Sun Mar 14 06:11:58 2021  | Hide | Hide all
Screenshot_from_2021-03-14_06-11-38_-_1.png
ELOG V3.1.4-unknown