AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 24 of 37  ELOG logo
ID Date Author Subjectdown
  421   Fri May 6 09:31:41 2022 TD NHFriday 6 May
10:30 DAQ appears to have stopped

      Attachment 2 - DSSSD bias and leakage current OK

      Attachments 1, 3-5 - zero rate adc data items all FEE64s - note aida08 adc data item counter zero

      RESET/SETUP/GO aida08 *only*
      DAQ STOP *all* FEE64s


      Attachment 6 - FEE64 temperatures OK

      Attachments 7-23 - system wide checks - all OK

10:48 DAQ GO *all* FEE64s
      all spectra zero'd

      attachment 29 - ADC data items
       n+n FEE64s aida02, 4, 6 & 8 c. 250k
       p+n FEE64s aida07, 9 & 12 c. 100-150k, all other FEE64s < 100k

      attachments 26 & 27 1.8.L spectra
       aida01 peak width 112ch FWHM
       aida02 peak width 485 ch FWHM

      attachments 24 & 25 1.8.W spectra 20us FSR
      
      attachment 30 - aida05 1*H spectra - aida05 asic#1 is producing HEC data

      attachment 31 - aida05 also producing disc data

      attachments 32 & 33 merger and tape server OK - 14MB/s to disk - no data to MBS currently? 


18.36 Disconnect ribbon cabling from adaptor PCB - continue to observe HEC events from aida05 asic#1
      Disconnect test inputs from adaptor PCB - continue to observe HEC events from aida05 asic#1

      Conclude aida05 asic#1 has probably failed in operation and should be replaced. Other FEE64s are currently in CRYRING hall and measurement is in progress - ML says we should be able to access Monday.
Attachment 1: Screenshot_from_2022-05-06_10-27-27.png
Screenshot_from_2022-05-06_10-27-27.png
Attachment 2: Screenshot_from_2022-05-06_10-26-59.png
Screenshot_from_2022-05-06_10-26-59.png
Attachment 3: Screenshot_from_2022-05-06_10-26-33.png
Screenshot_from_2022-05-06_10-26-33.png
Attachment 4: Screenshot_from_2022-05-06_10-25-44.png
Screenshot_from_2022-05-06_10-25-44.png
Attachment 5: Screenshot_from_2022-05-06_10-24-54.png
Screenshot_from_2022-05-06_10-24-54.png
Attachment 6: Screenshot_from_2022-05-06_10-42-18.png
Screenshot_from_2022-05-06_10-42-18.png
Attachment 7: Screenshot_from_2022-05-06_10-41-06.png
Screenshot_from_2022-05-06_10-41-06.png
Attachment 8: Screenshot_from_2022-05-06_10-41-00.png
Screenshot_from_2022-05-06_10-41-00.png
Attachment 9: Screenshot_from_2022-05-06_10-40-54.png
Screenshot_from_2022-05-06_10-40-54.png
Attachment 10: Screenshot_from_2022-05-06_10-40-09.png
Screenshot_from_2022-05-06_10-40-09.png
Attachment 11: Screenshot_from_2022-05-06_10-39-07.png
Screenshot_from_2022-05-06_10-39-07.png
Attachment 12: Screenshot_from_2022-05-06_10-38-58.png
Screenshot_from_2022-05-06_10-38-58.png
Attachment 13: Screenshot_from_2022-05-06_10-38-35.png
Screenshot_from_2022-05-06_10-38-35.png
Attachment 14: Screenshot_from_2022-05-06_10-38-21.png
Screenshot_from_2022-05-06_10-38-21.png
Attachment 15: Screenshot_from_2022-05-06_10-38-16.png
Screenshot_from_2022-05-06_10-38-16.png
Attachment 16: Screenshot_from_2022-05-06_10-38-10.png
Screenshot_from_2022-05-06_10-38-10.png
Attachment 17: Screenshot_from_2022-05-06_10-38-02.png
Screenshot_from_2022-05-06_10-38-02.png
Attachment 18: Screenshot_from_2022-05-06_10-37-52.png
Screenshot_from_2022-05-06_10-37-52.png
Attachment 19: Screenshot_from_2022-05-06_10-37-46.png
Screenshot_from_2022-05-06_10-37-46.png
Attachment 20: Screenshot_from_2022-05-06_10-37-39.png
Screenshot_from_2022-05-06_10-37-39.png
Attachment 21: Screenshot_from_2022-05-06_10-37-31.png
Screenshot_from_2022-05-06_10-37-31.png
Attachment 22: Screenshot_from_2022-05-06_10-36-24.png
Screenshot_from_2022-05-06_10-36-24.png
Attachment 23: Screenshot_from_2022-05-06_10-35-52.png
Screenshot_from_2022-05-06_10-35-52.png
Attachment 24: Screenshot_from_2022-05-06_10-56-31.png
Screenshot_from_2022-05-06_10-56-31.png
Attachment 25: Screenshot_from_2022-05-06_10-55-56.png
Screenshot_from_2022-05-06_10-55-56.png
Attachment 26: Screenshot_from_2022-05-06_10-55-13.png
Screenshot_from_2022-05-06_10-55-13.png
Attachment 27: Screenshot_from_2022-05-06_10-53-53.png
Screenshot_from_2022-05-06_10-53-53.png
Attachment 28: Screenshot_from_2022-05-06_10-51-00.png
Screenshot_from_2022-05-06_10-51-00.png
Attachment 29: Screenshot_from_2022-05-06_10-50-25.png
Screenshot_from_2022-05-06_10-50-25.png
Attachment 30: Screenshot_from_2022-05-06_11-09-15.png
Screenshot_from_2022-05-06_11-09-15.png
Attachment 31: Screenshot_from_2022-05-06_11-02-33.png
Screenshot_from_2022-05-06_11-02-33.png
Attachment 32: Screenshot_from_2022-05-06_11-19-48.png
Screenshot_from_2022-05-06_11-19-48.png
Attachment 33: Screenshot_from_2022-05-06_11-19-27.png
Screenshot_from_2022-05-06_11-19-27.png
  355   Fri Jun 4 08:40:52 2021 TDFriday 4 June
09.20 DAQ found stopped
      files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight
      aida15 rebooted at 00.02

      regained control of DAQ and issued DAQ STOP by restarting NewMerger *twice*

11.00 DAQ reset and setup
      All system wide checks OK *except*

		 Base 		Current 	Difference
aida05 fault 	 0x4d3c : 	 0x4d3c : 	 0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      4      2      8      5      4      1      2      2      4      7   : 41352
aida02 :      0      3      3      6      6      4      1      2      2      4      7   : 41352
aida03 :      2      4      2     12      1      3      2      2      2      4      7   : 41352
aida04 :      2      2      1      7      4      3      2      2      2      4      7   : 41352
aida05 :      2      3      1      5      3      2      3      2      2      4      7   : 41360
aida06 :      2      4      2      6      4      3      2      2      2      4      7   : 41352
aida07 :      1      5      2      9      4      2      4      3      1      4      7   : 41324
aida08 :      0      3      1      7      4      3      4      3      1      4      7   : 41352
aida09 :      3      3      0      8      2      2      3      2      2      4      7   : 41380
aida10 :      1      2      2     11      3      2      3      3      3      3      7   : 41044
aida11 :      2      3      1      6      6      4      2      3      3      3      7   : 41072
aida12 :      4      2      1      8      3      3      3      3      1      4      7   : 41072
aida13 :      2      4      1      8      4      4      2      3      3      3      7   : 41016
aida14 :      1      4      1      9      4      4      2      3      3      3      7   : 41044
aida15 :      0      5      1      2      2      1      2      4      3      4      7   : 42872
aida16 :      1      3      2      9      4      4      4      4      2      3      7   : 41052

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

11.03 DAQ resumes file S496/R32
      alpha background

      slow comparator 0x64
      BNC PB-5 pulser OFF

11.09 1.8.W spectra - 20us FSR - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

13.07 analysis of file S496/R32_16 - attachment 8
       max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png
Attachment 2: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-04_Statistics_aidas-gsi.png
Screenshot_2021-06-04_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 980.png
980.png
Attachment 7: 981.png
981.png
Attachment 8: R32_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   49166189 (   17465.0 Hz)
        Other data format:  212753815 (   75575.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        137 (       0.0 Hz)
                               RESUME:        137 (       0.0 Hz)
                              SYNC100:      42486 (      15.1 Hz)
                              WR48-63:      42486 (      15.1 Hz)
                           FEE64 disc:    1037793 (     368.6 Hz)
                             MBS info:  211630776 (   75176.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2815.130 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2759.088
  2                0.741                0.000
  3                4.946                0.000
  4                0.000                0.000
  5                0.000             2173.993
  6                0.000                0.000
  7                0.000             2104.432
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        184          0          0          0          0          0          0          0          0          0          0          0
  1     115458         62          0          0          0          0         31         31          0          0          0          0
  2        444  105932150          0          0         36         36      18754      18754          0  105894570          0          0
  3   39683483  106817705          0          0        101        101      21752      21752    1037793  105736206          0          0
  4        154          0          0          0          0          0          0          0          0          0          0          0
  5    4413391       1852          0          0          0          0        926        926          0          0          0          0
  6        586          0          0          0          0          0          0          0          0          0          0          0
  7    4948958       2044          0          0          0          0       1022       1022          0          0          0          0
  8        489          0          0          0          0          0          0          0          0          0          0          0
  9         77          0          0          0          0          0          0          0          0          0          0          0
 10       1172          0          0          0          0          0          0          0          0          0          0          0
 11        163          2          0          0          0          0          1          1          0          0          0          0
 12        754          0          0          0          0          0          0          0          0          0          0          0
 13         94          0          0          0          0          0          0          0          0          0          0          0
 14        605          0          0          0          0          0          0          0          0          0          0          0
 15        177          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.027s ( 1102.409 blocks/s,  68.901 Mb/s)
  697   Fri Jan 31 17:26:47 2025 CC, TD, MPFriday 31 January contd.
18.18 bPlas and BB7 installs complete 
      Restart AIDA DAQ, Merger, Tape Server and re-test

      per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 69 ch FWHM = 42keV FWHM

      per FEE64 Rate Spectra - attachment 2
       BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
       BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK

      per FEE64 1.8.W spectra - 20us FSR - attachments 3-5

      ADC data item stats - attachment 6
       aida01  aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
       BB7 aida10 19k

      WR timestamps OK - attachment 7

      Merger, TapeSever etc - attachments 8-9
       disk directpry /TapeData/FEB25
       working OK
     
      DSSSD bias & leakage current OK - attachment 10

      System wide checks OK - attachment 11-15
       note global clock status 6 errors reported earlier today have now gone

      FEE64 temperatures OK - attachment 16

18.38 Transition to safe state

      DAQ STOP
      disable data transfer 1
      detector bias OFF
      FEE64 power OFF


      Can restart as follows

1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
     sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps

If all OK can re-connect to Merger/TapeServer as follows

1) DAQ STOP
2) enable data transfer 1
3) DAQ GO


Can disconnect from Merger/TapeServer as follows

1) DAQ STOP
2) disable data transfer 1
   
Attachment 1: Screenshot_from_2025-01-31_18-18-17.png
Screenshot_from_2025-01-31_18-18-17.png
Attachment 2: Screenshot_from_2025-01-31_18-19-19.png
Screenshot_from_2025-01-31_18-19-19.png
Attachment 3: Screenshot_from_2025-01-31_18-19-37.png
Screenshot_from_2025-01-31_18-19-37.png
Attachment 4: Screenshot_from_2025-01-31_18-20-04.png
Screenshot_from_2025-01-31_18-20-04.png
Attachment 5: Screenshot_from_2025-01-31_18-20-30.png
Screenshot_from_2025-01-31_18-20-30.png
Attachment 6: Screenshot_from_2025-01-31_18-20-42.png
Screenshot_from_2025-01-31_18-20-42.png
Attachment 7: Screenshot_from_2025-01-31_18-21-01.png
Screenshot_from_2025-01-31_18-21-01.png
Attachment 8: Screenshot_from_2025-01-31_18-25-58.png
Screenshot_from_2025-01-31_18-25-58.png
Attachment 9: Screenshot_from_2025-01-31_18-26-13.png
Screenshot_from_2025-01-31_18-26-13.png
Attachment 10: Screenshot_from_2025-01-31_18-29-38.png
Screenshot_from_2025-01-31_18-29-38.png
Attachment 11: Screenshot_from_2025-01-31_18-29-52.png
Screenshot_from_2025-01-31_18-29-52.png
Attachment 12: Screenshot_from_2025-01-31_18-29-58.png
Screenshot_from_2025-01-31_18-29-58.png
Attachment 13: Screenshot_from_2025-01-31_18-30-06.png
Screenshot_from_2025-01-31_18-30-06.png
Attachment 14: Screenshot_from_2025-01-31_18-30-12.png
Screenshot_from_2025-01-31_18-30-12.png
Attachment 15: Screenshot_from_2025-01-31_18-30-30.png
Screenshot_from_2025-01-31_18-30-30.png
Attachment 16: Screenshot_from_2025-01-31_18-30-48.png
Screenshot_from_2025-01-31_18-30-48.png
  695   Fri Jan 31 09:37:47 2025 CC, TD, MPFriday 31 January
Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000

N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.

n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted

BB7 aida10 (asics #1-2 p+n, #3-4 n+n)

BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream) 

10.30 FEE64 power ON

      DSSSD bias ON (BB18 only) - attachment 1
       leakage current OK

      ASIC settings 2024Dec13-17.02.45 restored
       p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf

      Attachments 2-7 WR timestamp aida05 & aida12, system wide check
       aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
       WR decoder status aida02

      per FEE64 Rate spectra - attachment 8
       aida10 high rate - BB7 not biased
       rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
       BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments

      ADC data item stats - attachment 9
       rates generally OK
       aida10 high - BB7 not biased
       aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
       all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!

      BNC PB-5 settings - attachment 10

      WR timestamps OK - attachment 11

      FEE64 temperatures OK - attachment 12

12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet

      ADC data item stats - attachment 13
       all rates higher cf. attachment 9

      per FEE64 Rate spectra - attachment 14

      per p+n FEE64 1.8.L spectra - attachment 15
       aida09 pulser peak width 69 ch FWHM = 49keV FWHM 
       aida14 pulser peak width 50 ch FWHM = 35keV FWHM

      per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17

      per n+n FEE64 1.8W spectra - 20us FSR - attachment 18

      DSSSD bias & leakage current OK - attachment 19
       ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%

      FEE64 temperatures OK - attachment 20

      DSSSD bias volatge & leakage current OK - attachment 21
       CAEN N1419ET ch#0 BB18, ch#1 BB7

      Install of bPlas drivers, cabling, grounds complete

      ADC data item stats - attachment 22
       
      per FEE64 Rate spectra - attachment 23
       all BB18 p+n FEE64s *except* aida05 show very low rates of noise

      per p+n FEE64 1.8.L spectra - attachment 24
       aida09 pulser peak width 62 ch FWHM = 42keV FWHM
       BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
Attachment 1: Screenshot_from_2025-01-31_10-39-16.png
Screenshot_from_2025-01-31_10-39-16.png
Attachment 2: Screenshot_from_2025-01-31_10-58-48.png
Screenshot_from_2025-01-31_10-58-48.png
Attachment 3: Screenshot_from_2025-01-31_10-58-59.png
Screenshot_from_2025-01-31_10-58-59.png
Attachment 4: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 5: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 6: Screenshot_from_2025-01-31_10-59-52.png
Screenshot_from_2025-01-31_10-59-52.png
Attachment 7: Screenshot_from_2025-01-31_11-00-02.png
Screenshot_from_2025-01-31_11-00-02.png
Attachment 8: Screenshot_from_2025-01-31_11-17-59.png
Screenshot_from_2025-01-31_11-17-59.png
Attachment 9: Screenshot_from_2025-01-31_11-18-43.png
Screenshot_from_2025-01-31_11-18-43.png
Attachment 10: Screenshot_from_2025-01-31_11-18-50.png
Screenshot_from_2025-01-31_11-18-50.png
Attachment 11: Screenshot_from_2025-01-31_11-21-05.png
Screenshot_from_2025-01-31_11-21-05.png
Attachment 12: Screenshot_from_2025-01-31_11-21-29.png
Screenshot_from_2025-01-31_11-21-29.png
Attachment 13: Screenshot_from_2025-01-31_12-21-09.png
Screenshot_from_2025-01-31_12-21-09.png
Attachment 14: Screenshot_from_2025-01-31_12-21-55.png
Screenshot_from_2025-01-31_12-21-55.png
Attachment 15: Screenshot_from_2025-01-31_12-26-26.png
Screenshot_from_2025-01-31_12-26-26.png
Attachment 16: Screenshot_from_2025-01-31_12-30-13.png
Screenshot_from_2025-01-31_12-30-13.png
Attachment 17: Screenshot_from_2025-01-31_12-30-34.png
Screenshot_from_2025-01-31_12-30-34.png
Attachment 18: Screenshot_from_2025-01-31_12-31-25.png
Screenshot_from_2025-01-31_12-31-25.png
Attachment 19: Screenshot_from_2025-01-31_12-41-33.png
Screenshot_from_2025-01-31_12-41-33.png
Attachment 20: Screenshot_from_2025-01-31_12-42-08.png
Screenshot_from_2025-01-31_12-42-08.png
Attachment 21: Screenshot_from_2025-01-31_15-59-26.png
Screenshot_from_2025-01-31_15-59-26.png
Attachment 22: Screenshot_from_2025-01-31_16-07-21.png
Screenshot_from_2025-01-31_16-07-21.png
Attachment 23: Screenshot_from_2025-01-31_16-07-31.png
Screenshot_from_2025-01-31_16-07-31.png
Attachment 24: Screenshot_from_2025-01-31_16-08-39.png
Screenshot_from_2025-01-31_16-08-39.png
Attachment 25: Screenshot_from_2025-01-31_16-10-48.png
Screenshot_from_2025-01-31_16-10-48.png
Attachment 26: Screenshot_from_2025-01-31_16-11-30.png
Screenshot_from_2025-01-31_16-11-30.png
  288   Fri Apr 30 08:38:12 2021 TDFriday 30 April
09.40 All system wide checks OK *except*

FEE64 module aida07 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Thu Apr 29 17:50:06 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.44 DSSSD bias & leakage currents Ok - attachment 1
      Merger time seq errors - attachment 2
       no merger errors since restart yesterday
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      Merger/Tape Server server consoles - attachment 5
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 6

      DAQ continues OK - no data storage
Attachment 1: 310.png
310.png
Attachment 2: 311.png
311.png
Attachment 3: Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-30_Statistics_aidas-gsi.png
Screenshot_2021-04-30_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png
  626   Thu May 2 23:08:43 2024 TDFriday 3 May
00.05 FEE64 temperatures OK
      ADC data item stats OK

09.31 FEE64 temperatures OK
      ADC data item stats OK

13.15 FEE64 temperatures OK
      ADC data item stats OK
       9/16 <20k max aida16 164k

20.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida03 no data - DAQ reset
       9/16 <20k max aida16 158k

23.15 FEE64 temperatures OK
      ADC data item stats
       aida03 no data - DAQ reset
       10/16 <20k max aida16 153k
  39   Mon Apr 1 10:14:09 2019 CA, NH, TDFriday 29th March 2019
16:30 good event statistics ok, with 9,10,11,12 running faster (detector FEE64s) (attachment 1)

      FEE temperatures ok (attachment 2)

      detector bias/leakage currents ok (attachment 3)

      system wide checks ok *except* FEE64 Linux memory information check causes script error,
                                     WR decoder status returns errors

      DAQ & Merger running ok

      slow comparator threshold raised to 0x64

   
Attachment 1: 290319_stats.png
290319_stats.png
Attachment 2: 290319_temp.png
290319_temp.png
Attachment 3: 290913_bias.png
290913_bias.png
  350   Fri May 28 10:29:07 2021 TDFriday 28 May
11.30 DAQ continues - file S496/R18_472
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      7      1      3      2      2      2      2      4      6   : 36980
aida02 :      3      1      5      1      2      4      3      3      2      4      6   : 37892
aida03 :      8      8      6      6      3      3      2      3      3      3      6   : 36800
aida04 :     37     12      5      6      3      4      2      3      3      3      6   : 37060
aida05 :     11      5      6      4      2      2      2      2      2      4      6   : 37044
aida06 :      4      3      3      3      2      2      2      3      3      4      6   : 38456
aida07 :      1     12      3      2      0      3      3      2      2      4      6   : 37204
aida08 :     13      8      1      2      3      4      3      3      2      4      6   : 38020
aida09 :      7      7      5      1      1      3      2      2      2      4      6   : 36996
aida10 :     12      7      9      2      1      3      3      3      1      4      6   : 36856
aida11 :      2      9      3      1      2      3      3      3      1      4      6   : 36768
aida12 :     11      4      3      3      2      4      2      3      2      4      6   : 37724
aida13 :     10      8      6      2      2      3      2      2      2      4      6   : 37128
aida14 :     16      7      7      1      0      4      2      2      2      4      6   : 37128
aida15 :     12      6      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_471 - attachment 9
       max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-28_Statistics_aidas-gsi.png
Screenshot_2021-05-28_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 940.png
940.png
Attachment 8: 941.png
941.png
Attachment 9: R18_471
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   57407666 (   20753.5 Hz)
        Other data format:  204512336 (   73933.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        116 (       0.0 Hz)
                               RESUME:        116 (       0.0 Hz)
                              SYNC100:      42303 (      15.3 Hz)
                              WR48-63:      42303 (      15.3 Hz)
                           FEE64 disc:    1160567 (     419.6 Hz)
                             MBS info:  203266931 (   73483.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        109 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2766.172 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2700.364
  2                0.332                0.000
  3                4.856                0.000
  4                0.000                0.000
  5                0.000             1685.309
  6                0.000                0.000
  7                0.000             1567.327
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        181         23          0          0          0          0          0          0         23          0          0         13
  1     838470        386          0          0          0          0        166        166         54          0          0          0
  2        162  101753181          0          0         19         19      17931      17931         15  101717266          0          5
  3   42392398  102752755          0          0         97         97      21331      21331    1160234  101549665          0         12
  4        211         21          0          0          0          0          0          0         21          0          0         10
  5    6478076       2740          0          0          0          0       1337       1337         66          0          0          0
  6        260         17          0          0          0          0          0          0         17          0          0          4
  7    7695649       3163          0          0          0          0       1537       1537         89          0          0         28
  8        132         14          0          0          0          0          1          1         12          0          0         10
  9        368          5          0          0          0          0          0          0          5          0          0          4
 10        173         18          0          0          0          0          0          0         18          0          0          5
 11        303          5          0          0          0          0          0          0          5          0          0          5
 12        186          1          0          0          0          0          0          0          1          0          0          7
 13        585          2          0          0          0          0          0          0          2          0          0          2
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        512          5          0          0          0          0          0          0          5          0          0          4
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.742s ( 1008.122 blocks/s,  63.008 Mb/s)
  497   Fri Jun 24 15:04:06 2022 MarcFriday 24th June - evening shift

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

 

 

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

 

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

 

 

Attachment 1: Stats-Screenshot_from_2022-06-24_16-53-47.png
Stats-Screenshot_from_2022-06-24_16-53-47.png
Attachment 2: Temp-Screenshot_from_2022-06-24_16-52-58.png
Temp-Screenshot_from_2022-06-24_16-52-58.png
Attachment 3: HV_LC-Screenshot_from_2022-06-24_16-51-39.png
HV_LC-Screenshot_from_2022-06-24_16-51-39.png
Attachment 4: ucesb-Screenshot_from_2022-06-24_16-57-32.png
ucesb-Screenshot_from_2022-06-24_16-57-32.png
Attachment 5: GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
Attachment 6: Stats-Screenshot_from_2022-06-24_19-21-32.png
Stats-Screenshot_from_2022-06-24_19-21-32.png
Attachment 7: Temp-Screenshot_from_2022-06-24_19-20-48.png
Temp-Screenshot_from_2022-06-24_19-20-48.png
Attachment 8: HV-LC-Screenshot_from_2022-06-24_19-19-43.png
HV-LC-Screenshot_from_2022-06-24_19-19-43.png
Attachment 9: Stats-Screenshot_from_2022-06-24_22-01-09.png
Stats-Screenshot_from_2022-06-24_22-01-09.png
Attachment 10: Temp-Screenshot_from_2022-06-24_22-01-50.png
Temp-Screenshot_from_2022-06-24_22-01-50.png
Attachment 11: HV_LC-Screenshot_from_2022-06-24_21-58-33.png
HV_LC-Screenshot_from_2022-06-24_21-58-33.png
  495   Fri Jun 24 00:34:55 2022 AMFriday 24 June 00:00-08:00

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good

Attachment 1: 220624_0130_fpgatime.png
220624_0130_fpgatime.png
Attachment 2: 220624_0130_whiterabbit.png
220624_0130_whiterabbit.png
Attachment 3: 220624_0130_temps.png
220624_0130_temps.png
Attachment 4: 220624_0130_stats.png
220624_0130_stats.png
Attachment 5: 220624_0130_bias.png
220624_0130_bias.png
Attachment 6: 220624_0530_fpgatime.png
220624_0530_fpgatime.png
Attachment 7: 220624_0530_whiterabbit.png
220624_0530_whiterabbit.png
Attachment 8: 220624_0530_temps.png
220624_0530_temps.png
Attachment 9: 220624_0530_stats.png
220624_0530_stats.png
Attachment 10: 220624_0530_bias.png
220624_0530_bias.png
Attachment 11: 220624_0330_fpgatime.png
220624_0330_fpgatime.png
Attachment 12: 220624_0330_whiterabbit.png
220624_0330_whiterabbit.png
Attachment 13: 220624_0330_temps.png
220624_0330_temps.png
Attachment 14: 220624_0330_stats.png
220624_0330_stats.png
Attachment 15: 220624_0330_bias.png
220624_0330_bias.png
Attachment 16: 220624_0730_fpga.png
220624_0730_fpga.png
Attachment 17: 220624_0730_whiterabbit.png
220624_0730_whiterabbit.png
Attachment 18: 220624_0730_temps.png
220624_0730_temps.png
Attachment 19: 220624_0730_stats.png
220624_0730_stats.png
Attachment 20: 220624_0730_bias.png
220624_0730_bias.png
  496   Fri Jun 24 06:56:06 2022 OHFriday 24 June 008:00-16:00
07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16
Attachment 1: 220624_0753_stats.png
220624_0753_stats.png
Attachment 2: 220624_0754_Temp.png
220624_0754_Temp.png
Attachment 3: 220624_0755_Bias.png
220624_0755_Bias.png
Attachment 4: R3_359_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260715255 (  897318.9 Hz)
        Other data format:    1204745 (    4146.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1323 (       4.6 Hz)
                               RESUME:       1323 (       4.6 Hz)
                              SYNC100:      33082 (     113.9 Hz)
                              WR48-63:      33082 (     113.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1135935 (    3909.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     353220 (    1215.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.549 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.171                0.000
  1               46.409                0.000
  2                1.006                0.000
  3                4.601                0.000
  4                0.000               37.574
  5                0.031                0.000
  6                0.191                0.000
  7                0.254                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903987      28078          0          0        236        236       4023       4023          0      19560          0      86851
  1   61208192      17252          0          0        718        718       7908       7908          0          0          0      52897
  2   26785987     291397          0          0         72         72       3425       3425          0     284403          0      56573
  3   43119726     289440          0          0        153        153       5319       5319          0     278496          0      29859
  4    4604412       1194          0          0          0          0        597        597          0          0          0      28791
  5   18336996       4656          0          0         16         16       2312       2312          0          0          0      34152
  6   36059875     277258          0          0         42         42       4514       4514          0     268146          0      48935
  7   39696080     295470          0          0         86         86       4984       4984          0     285330          0      15162
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.877s (  917.511 blocks/s,  57.344 Mb/s)
Attachment 5: 220624_0842_Stats_Stop.png
220624_0842_Stats_Stop.png
Attachment 6: 220624_1111_Stats.png
220624_1111_Stats.png
Attachment 7: 220624_1111_Temp.png
220624_1111_Temp.png
Attachment 8: 220624_1112_Bias.png
220624_1112_Bias.png
Attachment 9: 220624_1303_stats.png
220624_1303_stats.png
Attachment 10: 220624_1303_Temp.png
220624_1303_Temp.png
Attachment 11: 220624_1304_Bias.png
220624_1304_Bias.png
Attachment 12: R5_33_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260717099 (  869523.1 Hz)
        Other data format:    1202901 (    4011.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1344 (       4.5 Hz)
                               RESUME:       1344 (       4.5 Hz)
                              SYNC100:      33117 (     110.4 Hz)
                              WR48-63:      33117 (     110.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1133979 (    3782.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354749 (    1183.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      299.839 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.504                0.000
  1               42.102                0.000
  2                0.275                0.000
  3                6.677                0.000
  4                0.000               31.976
  5                0.051                0.000
  6                0.250                0.000
  7                0.412                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31758241      28867          0          0        238        238       4156       4156          0      20079          0      87480
  1   58453105      16104          0          0        635        635       7417       7417          0          0          0      53128
  2   20104679     289945          0          0         41         41       2565       2565          0     284733          0      56849
  3   41709424     287290          0          0        188        188       5262       5262          0     276390          0      29572
  4    5133297       1264          0          0          0          0        632        632          0          0          0      29278
  5   17857346       4470          0          0         22         22       2213       2213          0          0          0      34539
  6   40732502     278221          0          0         59         59       5175       5175          0     267753          0      48740
  7   44968505     296740          0          0        161        161       5697       5697          0     285024          0      15163
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.742s (  921.070 blocks/s,  57.567 Mb/s)
Attachment 13: 220624_1538_Stats.png
220624_1538_Stats.png
Attachment 14: 220624_1538_Temp.png
220624_1538_Temp.png
Attachment 15: 220624_1538_Bias.png
220624_1538_Bias.png
Attachment 16: R5_63_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260796374 (  834020.0 Hz)
        Other data format:    1123626 (    3593.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1191 (       3.8 Hz)
                               RESUME:       1191 (       3.8 Hz)
                              SYNC100:      33165 (     106.1 Hz)
                              WR48-63:      33165 (     106.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1054914 (    3373.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     350507 (    1120.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.698 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.889                0.000
  1               39.498                0.000
  2                0.306                0.000
  3                5.270                0.000
  4                0.000               26.707
  5                0.036                0.000
  6                0.217                0.000
  7                0.175                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903831      29785          0          0        188        188       4074       4074          0      21261          0      87155
  1   61666403      16932          0          0        646        646       7820       7820          0          0          0      55280
  2   20140383     267008          0          0         42         42       2575       2575          0     261774          0      55290
  3   41652352     266495          0          0        164        164       5306       5306          0     255555          0      28879
  4    4871487       1204          0          0          0          0        602        602          0          0          0      28340
  5   17605547       4430          0          0         11         11       2204       2204          0          0          0      33538
  6   41056242     264471          0          0         55         55       5126       5126          0     254109          0      47278
  7   42900129     273301          0          0         85         85       5458       5458          0     262215          0      14747
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.305s (  774.730 blocks/s,  48.421 Mb/s)
  264   Fri Apr 23 07:09:57 2021 LJWFriday 23rd April 08:00-12:00

08:30

System Checks

 

Clock Status error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status error:


         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

 

Statistics - See attacnment 1

Temperatures - See attachment 2

Bias & Leakage Currents - See attachment 3

 

10:02

System Checks:

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration Error:

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 4

Temperatures - See attachment 5

Bias & Leakage Currents - See attachment 6

 

10:45

System Checks

Clock Status Error:


FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


ADC Calibration check Error:


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 7

Temperatures - See attachment 8

Bias & Leakage Currents - See attachment 9

 

Attachment 1: 2304211_1.png
2304211_1.png
Attachment 2: 230421_2.png
230421_2.png
Attachment 3: 230421_3.png
230421_3.png
Attachment 4: 230421_4.png
230421_4.png
Attachment 5: 230421_5.png
230421_5.png
Attachment 6: 230421_6.png
230421_6.png
Attachment 7: 230421_7.png
230421_7.png
Attachment 8: 230421_8.png
230421_8.png
Attachment 9: 230421_9.png
230421_9.png
  263   Thu Apr 22 23:17:46 2021 DJ Friday 23rd April 00:00-08:00

Took over from TD ... All is well.

00:45 System Wide Checks

Clock: 12 passed, 0 failed.

ADC Calibration: Passed 12, 0 failed.

White Rabbit Decoder: Passed 4,failed 8. - See screenshot.

FPGA Passed 11, failed 1. - See screenshot.

FEE64 Linux: See screenshot.

-----

02:49 run closed. Checking beam.

03:00 File 55 opened.

03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.

File 56 opened at 4:24.

---------

04:30 System Wide Checks (See screenshots)

White Rabbit Decoder: Passed 12, failed 0.

FPGA: page not loading properly.

----

05:37 System wide checks. See screenshots

Clock: Status: 11 passed. 1 fail.

ADC calibration: Passed 9, failed 3.

White Rabbit decoder: Passed 12, failed 0.

-----

07:32 System wide checks (See screenshots).

 

 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi.png
Screenshot_2021-04-23_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot-CAEN_HV_Supply.png
Screenshot-CAEN_HV_Supply.png
Attachment 4: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Attachment 5: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 60.png
60.png
Attachment 10: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Attachment 12: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Attachment 13: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Attachment 17: 61.png
61.png
Attachment 18: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Attachment 19: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Attachment 20: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Attachment 21: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 22: Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Attachment 23: 63.png
63.png
Attachment 24: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Attachment 25: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Attachment 26: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Attachment 27: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
  268   Fri Apr 23 20:19:52 2021 TDFriday 23 April 20.00-00.00
21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378
Attachment 1: 201.png
201.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Attachment 4: Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Attachment 5: 202.png
202.png
Attachment 6: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
  266   Fri Apr 23 13:13:46 2021 MA, TDFriday 23 April 12.00-16.00
14.15 System wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors - reports 'server internal error'

14.16 DSSSD bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachment 2
      good events stats - attachment 3

14.22 DAQ continues file S460/R51_205
      beam intensity 2.2e+09/s


15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6


Clock check

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC check

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



White Rabbit


		 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




FPGA
Loge page error!

Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      6      2      1      2      3      2      4      2      3      6   : 36044
aida02 :     15      3      2      2      0      3      2      3      2      3      4   : 27188
aida03 :     29      4      0      3      2      4      2      2      3      3      6   : 36212
aida04 :      4      6      4      4      2      3      3      3      3      3      6   : 36864
aida05 :     19      5      4      2      3      3      1      3      3      3      6   : 36404
aida06 :     24      9      5      2      3      3      3      2      3      3      6   : 36472
aida07 :     20      4      9      0      2      3      2      2      3      3      6   : 36096
aida08 :     22      9      4      3      1      3      1      3      3      3      6   : 36352
aida09 :     20      7      3      2      2      3      3      2      3      3      6   : 36344
aida10 :     25      9      9      5      2      3      2      3      3      3      6   : 36828
aida11 :     24     11      2      0      1      3      1      3      3      3      6   : 36248
aida12 :     26      5      4      2      1      3      2      2      3      3      6   : 36048
Attachment 1: 200.png
200.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Attachment 4: 210423_1550_Voltages.png
210423_1550_Voltages.png
Attachment 5: 210423_1550_Stats.png
210423_1550_Stats.png
Attachment 6: 210423_1550_Temp.png
210423_1550_Temp.png
  549   Fri Mar 22 08:31:39 2024 TDFriday 22 March
09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39
Attachment 1: Screenshot_from_2024-03-22_09-32-58.png
Screenshot_from_2024-03-22_09-32-58.png
Attachment 2: Screenshot_from_2024-03-22_09-34-54.png
Screenshot_from_2024-03-22_09-34-54.png
Attachment 3: Screenshot_from_2024-03-22_09-39-03.png
Screenshot_from_2024-03-22_09-39-03.png
Attachment 4: Screenshot_from_2024-03-22_09-40-10.png
Screenshot_from_2024-03-22_09-40-10.png
Attachment 5: Screenshot_from_2024-03-22_09-47-26.png
Screenshot_from_2024-03-22_09-47-26.png
Attachment 6: Screenshot_from_2024-03-22_09-49-33.png
Screenshot_from_2024-03-22_09-49-33.png
Attachment 7: Screenshot_from_2024-03-22_09-49-00.png
Screenshot_from_2024-03-22_09-49-00.png
Attachment 8: Screenshot_from_2024-03-22_09-48-25.png
Screenshot_from_2024-03-22_09-48-25.png
Attachment 9: Screenshot_from_2024-03-22_09-47-55.png
Screenshot_from_2024-03-22_09-47-55.png
Attachment 10: Screenshot_from_2024-03-22_10-00-47.png
Screenshot_from_2024-03-22_10-00-47.png
Attachment 11: Screenshot_from_2024-03-22_10-00-01.png
Screenshot_from_2024-03-22_10-00-01.png
Attachment 12: Screenshot_from_2024-03-22_09-58-56.png
Screenshot_from_2024-03-22_09-58-56.png
Attachment 13: Screenshot_from_2024-03-22_09-57-50.png
Screenshot_from_2024-03-22_09-57-50.png
Attachment 14: Screenshot_from_2024-03-22_09-56-45.png
Screenshot_from_2024-03-22_09-56-45.png
Attachment 15: Screenshot_from_2024-03-22_09-55-52.png
Screenshot_from_2024-03-22_09-55-52.png
Attachment 16: Screenshot_from_2024-03-22_09-54-18.png
Screenshot_from_2024-03-22_09-54-18.png
Attachment 17: Screenshot_from_2024-03-22_09-52-45.png
Screenshot_from_2024-03-22_09-52-45.png
Attachment 18: Screenshot_from_2024-03-22_09-52-13.png
Screenshot_from_2024-03-22_09-52-13.png
Attachment 19: Screenshot_from_2024-03-22_09-51-47.png
Screenshot_from_2024-03-22_09-51-47.png
Attachment 20: Screenshot_from_2024-03-22_10-04-22.png
Screenshot_from_2024-03-22_10-04-22.png
Attachment 21: Screenshot_from_2024-03-22_10-03-53.png
Screenshot_from_2024-03-22_10-03-53.png
Attachment 22: Screenshot_from_2024-03-22_14-29-18.png
Screenshot_from_2024-03-22_14-29-18.png
Attachment 23: Screenshot_from_2024-03-22_14-30-00.png
Screenshot_from_2024-03-22_14-30-00.png
Attachment 24: Screenshot_from_2024-03-22_14-31-19.png
Screenshot_from_2024-03-22_14-31-19.png
Attachment 25: Screenshot_from_2024-03-22_14-31-01.png
Screenshot_from_2024-03-22_14-31-01.png
Attachment 26: Screenshot_from_2024-03-22_14-31-57.png
Screenshot_from_2024-03-22_14-31-57.png
Attachment 27: Screenshot_from_2024-03-22_14-32-35.png
Screenshot_from_2024-03-22_14-32-35.png
Attachment 28: Screenshot_from_2024-03-22_14-40-04.png
Screenshot_from_2024-03-22_14-40-04.png
Attachment 29: Screenshot_from_2024-03-22_14-39-30.png
Screenshot_from_2024-03-22_14-39-30.png
Attachment 30: Screenshot_from_2024-03-22_14-38-22.png
Screenshot_from_2024-03-22_14-38-22.png
Attachment 31: Screenshot_from_2024-03-22_14-37-19.png
Screenshot_from_2024-03-22_14-37-19.png
Attachment 32: Screenshot_from_2024-03-22_14-36-28.png
Screenshot_from_2024-03-22_14-36-28.png
Attachment 33: Screenshot_from_2024-03-22_14-35-23.png
Screenshot_from_2024-03-22_14-35-23.png
Attachment 34: Screenshot_from_2024-03-22_14-34-38.png
Screenshot_from_2024-03-22_14-34-38.png
Attachment 35: Screenshot_from_2024-03-22_14-33-55.png
Screenshot_from_2024-03-22_14-33-55.png
Attachment 36: Screenshot_from_2024-03-22_14-41-21.png
Screenshot_from_2024-03-22_14-41-21.png
Attachment 37: Screenshot_from_2024-03-22_14-40-57.png
Screenshot_from_2024-03-22_14-40-57.png
Attachment 38: Capture_2.PNG
Capture_2.PNG
Attachment 39: Capture_3.PNG
Capture_3.PNG
  342   Fri May 21 15:03:48 2021 CAFriday 21st May 16:00 - 00:00 shift
16:00 CA takes over
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

16:14 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x570 : 	 112  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

16:17 temperatures ok - attachment 1
      statistics ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      analysis of R14_1625 - FEE7 deadtime at 4.5%

18:06 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x571 : 	 113  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:07 temperatures ok - attachment 4
      statistics ok - attachment 5
      detector bias / leakage currents ok - attachment 6
      analysis of R14_1662 - FEE7 deadtime at 4.3%

19:07 DAQ continues ok - writing to file R14_1682

20:07 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

20:10 temperatures ok - attachment 7
      statistics ok - attachment 8
      detector bias / leakage currents ok - attachment 9
      analysis of R14_1703 - max deadtime at 4.0% - attachment 10

20:35 rate spectra - attachment 11
      HEC spectra - attachment 12 & 13

20:36 all histograms zeroed

20:37 DAQ continues ok - writing to file R14_1712

20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.

      ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files

21:12 no beam - writing to file R14_1723

21:17 beam is back

22:07 system wide checks all ok - WR decoder fault same as entry at 20:07

22:08 temperatures ok - attachment 14
      statistics ok - attachment 15
      detector bias / leakage currents ok - attachment 16
      analysis of R14_1740 - max deadtime at 4.2% - attachment 17








   
Attachment 1: Screenshot_from_2021-05-21_15-14-59.png
Screenshot_from_2021-05-21_15-14-59.png
Attachment 2: Screenshot_from_2021-05-21_15-15-38.png
Screenshot_from_2021-05-21_15-15-38.png
Attachment 3: Screenshot_from_2021-05-21_15-15-54.png
Screenshot_from_2021-05-21_15-15-54.png
Attachment 4: Screenshot_from_2021-05-21_17-06-56.png
Screenshot_from_2021-05-21_17-06-56.png
Attachment 5: Screenshot_from_2021-05-21_17-07-18.png
Screenshot_from_2021-05-21_17-07-18.png
Attachment 6: Screenshot_from_2021-05-21_17-07-47.png
Screenshot_from_2021-05-21_17-07-47.png
Attachment 7: Screenshot_from_2021-05-21_19-09-19.png
Screenshot_from_2021-05-21_19-09-19.png
Attachment 8: Screenshot_from_2021-05-21_19-09-59.png
Screenshot_from_2021-05-21_19-09-59.png
Attachment 9: Screenshot_from_2021-05-21_19-10-15.png
Screenshot_from_2021-05-21_19-10-15.png
Attachment 10: Screenshot_from_2021-05-21_19-14-45.png
Screenshot_from_2021-05-21_19-14-45.png
Attachment 11: Screenshot_from_2021-05-21_19-25-51.png
Screenshot_from_2021-05-21_19-25-51.png
Attachment 12: Screenshot_from_2021-05-21_19-30-14.png
Screenshot_from_2021-05-21_19-30-14.png
Attachment 13: Screenshot_from_2021-05-21_19-32-13.png
Screenshot_from_2021-05-21_19-32-13.png
Attachment 14: Screenshot_from_2021-05-21_21-08-28.png
Screenshot_from_2021-05-21_21-08-28.png
Attachment 15: Screenshot_from_2021-05-21_21-08-51.png
Screenshot_from_2021-05-21_21-08-51.png
Attachment 16: Screenshot_from_2021-05-21_21-09-06.png
Screenshot_from_2021-05-21_21-09-06.png
Attachment 17: Screenshot_from_2021-05-21_21-12-03.png
Screenshot_from_2021-05-21_21-12-03.png
  341   Fri May 21 07:03:50 2021 OH, TDFriday 21 May 08:00-16:00
08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png
210521_0829_Stats.png
Attachment 2: 210521_0829_Temp.png
210521_0829_Temp.png
Attachment 3: 210521_0830_Bias.png
210521_0830_Bias.png
Attachment 4: 210521_0957.png
210521_0957.png
Attachment 5: 210521_0957_Temp.png
210521_0957_Temp.png
Attachment 6: 210521_0958_Bias.png
210521_0958_Bias.png
Attachment 7: 210521_1232_Stats.png
210521_1232_Stats.png
Attachment 8: 210521_1233_Bias.png
210521_1233_Bias.png
Attachment 9: 210521_1233_Temp.png
210521_1233_Temp.png
  130   Fri Feb 21 09:39:38 2020 TD, NHFriday 21 February
Attachment 1 - layout 2 - stat spectra
Attachment 2 - detector biases & leakage currents - OK
Attachment 3 - FEE64 temperatures - OK
Attachments 4-7 - TapeServer, Merger, Merger Statistics, good event statistics - OK

10:40 ASIC check load x2


13.12 DAQ stopped c. 10.50
      unable to connect messages for multiple FEE64s

      After lunch powwer-cycled FEE64s and restarted DAQ
      aida03 crashed (see https://elog.ph.ed.ac.uk/DESPEC/131 ) and rebooted by issuing reboot command as root

      Data file 20FEB20/R1
      ASIC settings 2019Oct31-13.24.23
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      continue alpha background run
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
  468   Fri May 20 16:22:01 2022 TDFriday 20 May
17.20 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-20_17-19-21.png
Screenshot_from_2022-05-20_17-19-21.png
Attachment 2: Screenshot_from_2022-05-20_17-19-35.png
Screenshot_from_2022-05-20_17-19-35.png
Attachment 3: Screenshot_from_2022-05-20_17-20-22.png
Screenshot_from_2022-05-20_17-20-22.png
ELOG V3.1.3-7933898