AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 6 of 35  ELOG logo
IDdown Date Author Subject
  606   Thu Apr 25 15:02:50 2024 Marc16:00 - 0:00 Thursday 25 April
16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

 

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

Attachment 1: Screenshot_from_2024-04-25_16-19-32.png
Screenshot_from_2024-04-25_16-19-32.png
Attachment 2: Screenshot_from_2024-04-25_16-45-19.png
Screenshot_from_2024-04-25_16-45-19.png
Attachment 3: Screenshot_from_2024-04-25_16-46-12.png
Screenshot_from_2024-04-25_16-46-12.png
Attachment 4: Screenshot_from_2024-04-25_16-46-41.png
Screenshot_from_2024-04-25_16-46-41.png
Attachment 5: Screenshot_from_2024-04-25_16-47-00.png
Screenshot_from_2024-04-25_16-47-00.png
Attachment 6: Screenshot_from_2024-04-25_16-52-18.png
Screenshot_from_2024-04-25_16-52-18.png
Attachment 7: Screenshot_from_2024-04-25_16-52-50.png
Screenshot_from_2024-04-25_16-52-50.png
Attachment 8: Screenshot_from_2024-04-25_17-23-36.png
Screenshot_from_2024-04-25_17-23-36.png
  605   Thu Apr 25 08:26:50 2024 TD08:00-16:00 Thursday 25 April
09.56 DAQ continues data file S100_alpha/R18_387
      FRS switching settings

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK  - attachment 2
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 3-5

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME & Correlation Scaler data time stats - attachments 7-10

      per FEE64 Rate spectra - attachments 11

      per p+n FEE64 1.8.L spectra - attachments 12-13
       aida09 pulser peak width 57 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 14

      per FEE64 1.8.H spectra - attachments 15-16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 19-21

      BNC PB-5 settings - attachment 22

10.07 data file S100_alpha/R20
      new run for new FRS setting

10.08 all histograms, stats, merger stats and data link stats zero'd

10.10 ADC data item stats (beam OFF) - attachment 23
      11/16 < 20k, max 129k (aida08)

10.31 analysis of data file R18_349 (closed 06.00 this morning)
      max deadtime 12% (aida04)
      no timewarps
      HEC data item rate 1.7kHz

11.00 tuning 168dy FRS setting
      ucesb - attachment 24
      spill length 3.5s, extraction time 1.5s

11.30 degraders now set to implant in AIDA DSSSDs
      high rates expected 

      per FEE64 Rate spectra - attachment 25

      ADC data item stats - attachment 26

      Merger etc - attachment 27
      in spill rates to MBS up 30+Mb/s, 2.5M data item/s merged

      DSSSD bias & leakage current - attachment 28
       c. few hundred nA step in leakage current 

      analysis of data file R20_33 - attachment 29
      max deadtime 18% (aida08)
      no timewarps
      HEC data item rate 4.1kHz (c. 2.4x previous FRS setting)

      Correcting for spill cycle => 5.9kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 2.0kHz in spill or 6800 implants per spill
      SC42 L/R currently report c. 4.5kHz

      Note that we currently see a higher fraction of events in DSSSD#2 cf. previous setting so multiplicity assumption may be low
      If we assume multipicity = 6 => implant rate ~ 1.0kHz in spill or 3200 implants per spill

      we are probably somewhere in between

12.50 DSSSD bias & leakage current - Grafana - attachment 30
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 31

      ADC data item stats - attachment 32

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

14.56 analysis of data file R20_106 - attachment 35
      max deadtime c. 18% (aida04 & aida08), c. 12% (aida03), c. 11% (aida06), c. 9% (aida02 & aida13), c. 6% (aida05), all others <2%
      no timewarps
      HEC data item rate 4.2kHz


15.40 DSSSD bias & leakage current - Grafana - attachments 36-37
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41

      ucesb - attachment 42

15.57 all histograms zero'd




      
Attachment 1: Screenshot_from_2024-04-25_09-27-20.png
Screenshot_from_2024-04-25_09-27-20.png
Attachment 2: Screenshot_from_2024-04-25_09-27-44.png
Screenshot_from_2024-04-25_09-27-44.png
Attachment 3: Screenshot_from_2024-04-25_09-28-05.png
Screenshot_from_2024-04-25_09-28-05.png
Attachment 4: Screenshot_from_2024-04-25_09-28-16.png
Screenshot_from_2024-04-25_09-28-16.png
Attachment 5: Screenshot_from_2024-04-25_09-28-23.png
Screenshot_from_2024-04-25_09-28-23.png
Attachment 6: Screenshot_from_2024-04-25_09-29-04.png
Screenshot_from_2024-04-25_09-29-04.png
Attachment 7: Screenshot_from_2024-04-25_09-29-20.png
Screenshot_from_2024-04-25_09-29-20.png
Attachment 8: Screenshot_from_2024-04-25_09-29-46.png
Screenshot_from_2024-04-25_09-29-46.png
Attachment 9: Screenshot_from_2024-04-25_09-29-59.png
Screenshot_from_2024-04-25_09-29-59.png
Attachment 10: Screenshot_from_2024-04-25_09-30-15.png
Screenshot_from_2024-04-25_09-30-15.png
Attachment 11: Screenshot_from_2024-04-25_09-30-56.png
Screenshot_from_2024-04-25_09-30-56.png
Attachment 12: Screenshot_from_2024-04-25_09-31-55.png
Screenshot_from_2024-04-25_09-31-55.png
Attachment 13: Screenshot_from_2024-04-25_09-32-11.png
Screenshot_from_2024-04-25_09-32-11.png
Attachment 14: Screenshot_from_2024-04-25_09-33-15.png
Screenshot_from_2024-04-25_09-33-15.png
Attachment 15: Screenshot_from_2024-04-25_09-34-23.png
Screenshot_from_2024-04-25_09-34-23.png
Attachment 16: Screenshot_from_2024-04-25_09-34-52.png
Screenshot_from_2024-04-25_09-34-52.png
Attachment 17: Screenshot_from_2024-04-25_09-35-38.png
Screenshot_from_2024-04-25_09-35-38.png
Attachment 18: Screenshot_from_2024-04-25_09-36-14.png
Screenshot_from_2024-04-25_09-36-14.png
Attachment 19: Screenshot_from_2024-04-25_09-36-28.png
Screenshot_from_2024-04-25_09-36-28.png
Attachment 20: Screenshot_from_2024-04-25_09-37-02.png
Screenshot_from_2024-04-25_09-37-02.png
Attachment 21: Screenshot_from_2024-04-25_09-37-17.png
Screenshot_from_2024-04-25_09-37-17.png
Attachment 22: Screenshot_from_2024-04-25_10-02-31.png
Screenshot_from_2024-04-25_10-02-31.png
Attachment 23: R18_349
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254055879 (  798496.4 Hz)
        Other data format:    7864121 (   24716.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        850 (       2.7 Hz)
                               RESUME:        851 (       2.7 Hz)
                              SYNC100:      33185 (     104.3 Hz)
                              WR48-63:      33185 (     104.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7796050 (   24502.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     547807 (    1721.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.168 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.011                0.000
  1               15.547                0.000
  2                0.026                0.000
  3               39.441                0.000
  4                0.410                0.000
  5                0.954                0.000
  6                0.000                0.000
  7                0.601                0.000
  8                0.003                0.000
  9                0.000                0.000
 10                0.050                0.000
 11                0.014                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.017                0.000
 15                0.116                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6649451      11230          0          0          3          3        839        839          0       9546          0      49433
  1   33550123     378474          0          0        245        245       4373       4373          0     369238          0     100748
  2    7942952    3356106          0          0          5          5       1429       1429          0    3353238          0      35304
  3   50066846    2782463          0          0        379        380       6547       6547          0    2768610          0      28219
  4   11634489     554162          0          0         19         19       1542       1542          0     551040          0      47658
  5   18478696     749338          0          0         77         77       2403       2403          0     744378          0      37331
  6    6034652       1500          0          0          0          0        750        750          0          0          0      11345
  7   41275133      10854          0          0         71         71       5356       5356          0          0          0      92779
  8    3808950        934          0          0          2          2        465        465          0          0          0      19548
  9    5816078       1448          0          0          0          0        724        724          0          0          0       8443
 10   16595017       4342          0          0         16         16       2155       2155          0          0          0       3675
 11    5732285       1360          0          0          2          2        678        678          0          0          0      65221
 12    7170732       1770          0          0          0          0        885        885          0          0          0      12515
 13    5117877       1350          0          0          0          0        675        675          0          0          0      13502
 14    5096608       1234          0          0          2          2        615        615          0          0          0       7339
 15   29085990       7556          0          0         29         29       3749       3749          0          0          0      14747
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.840s ( 1005.030 blocks/s,  62.814 Mb/s)
Attachment 24: Screenshot_from_2024-04-25_11-27-43.png
Screenshot_from_2024-04-25_11-27-43.png
Attachment 25: Screenshot_from_2024-04-25_11-39-59.png
Screenshot_from_2024-04-25_11-39-59.png
Attachment 26: Screenshot_from_2024-04-25_11-40-11.png
Screenshot_from_2024-04-25_11-40-11.png
Attachment 27: Screenshot_from_2024-04-25_11-40-28.png
Screenshot_from_2024-04-25_11-40-28.png
Attachment 28: Screenshot_from_2024-04-25_11-43-22.png
Screenshot_from_2024-04-25_11-43-22.png
Attachment 29: R20_33
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257467881 ( 1603138.0 Hz)
        Other data format:    4452119 (   27721.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1792 (      11.2 Hz)
                               RESUME:       1788 (      11.1 Hz)
                              SYNC100:      32598 (     203.0 Hz)
                              WR48-63:      32598 (     203.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4383343 (   27293.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     659812 (    4108.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.602 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                4.032                0.000
  1               14.271                0.000
  2               17.643                0.000
  3               26.583                0.000
  4               10.476                0.000
  5               15.929                0.000
  6                1.815                0.000
  7               29.387                0.000
  8                0.158                0.000
  9                0.036                0.000
 10                0.592                0.000
 11                0.531                0.000
 12               14.310                0.000
 13                0.165                0.000
 14                0.189                0.000
 15                0.995                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14126134       8441          0          0         70         70       1791       1791          0       4719          0      78615
  1   16495785     288035          0          0        163        162       2094       2094          0     283522          0     105734
  2   19988207    1651337          0          0        161        161       2732       2732          0    1645551          0      49977
  3   24594867    1476087          0          0        215        214       3183       3183          0    1469292          0      26591
  4   15580694     468076          0          0        119        119       1941       1941          0     463956          0      27262
  5   22750595     522722          0          0        300        299       2910       2910          0     516303          0      62407
  6   11629769       3045          0          0         37         36       1486       1486          0          0          0      11991
  7   37179788       9710          0          0        273        273       4582       4582          0          0          0      86342
  8    7628214       1994          0          0          9          9        988        988          0          0          0       1179
  9    5725517       1418          0          0          5          5        704        704          0          0          0       1622
 10   13190745       3550          0          0         69         69       1706       1706          0          0          0         70
 11    9285908       2326          0          0         29         29       1134       1134          0          0          0      74999
 12   16713447       4700          0          0        252        252       2098       2098          0          0          0      21646
 13    6115493       1498          0          0         25         25        724        724          0          0          0      35370
 14    8494444       2072          0          0         11         11       1025       1025          0          0          0         92
 15   27968274       7108          0          0         54         54       3500       3500          0          0          0      75915
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.629s (  924.083 blocks/s,  57.755 Mb/s)
Attachment 30: Screenshot_from_2024-04-25_12-50-09.png
Screenshot_from_2024-04-25_12-50-09.png
Attachment 31: Screenshot_from_2024-04-25_12-50-16.png
Screenshot_from_2024-04-25_12-50-16.png
Attachment 32: Screenshot_from_2024-04-25_12-50-28.png
Screenshot_from_2024-04-25_12-50-28.png
Attachment 33: Screenshot_from_2024-04-25_12-50-35.png
Screenshot_from_2024-04-25_12-50-35.png
Attachment 34: Screenshot_from_2024-04-25_12-51-02.png
Screenshot_from_2024-04-25_12-51-02.png
Attachment 35: R20_106
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257522900 ( 1655989.1 Hz)
        Other data format:    4397100 (   28275.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1842 (      11.8 Hz)
                               RESUME:       1844 (      11.9 Hz)
                              SYNC100:      32579 (     209.5 Hz)
                              WR48-63:      32579 (     209.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4328256 (   27832.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     659185 (    4238.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      155.510 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                3.085                0.000
  1               13.568                0.000
  2               18.124                0.000
  3               27.697                0.000
  4                9.792                0.000
  5               16.474                0.000
  6                2.671                0.000
  7               28.543                0.000
  8                0.065                0.000
  9                0.015                0.000
 10                0.643                0.000
 11                0.597                0.000
 12               14.389                0.000
 13                0.083                0.000
 14                0.439                0.000
 15                1.481                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13689199       8238          0          0         65         65       1771       1771          0       4566          0      77953
  1   16947527     287988          0          0        165        166       2170       2170          0     283317          0     104868
  2   20356284    1630731          0          0        174        175       2797       2797          0    1624788          0      51128
  3   23896908    1435934          0          0        213        214       3077       3077          0    1429353          0      26122
  4   15647459     464207          0          0        117        117       2008       2008          0     459957          0      27024
  5   22696175     532714          0          0        306        305       2914       2914          0     526275          0      64287
  6   12236208       3082          0          0         42         42       1499       1499          0          0          0      11994
  7   36272075       9624          0          0        259        259       4553       4553          0          0          0      84854
  8    7567267       1906          0          0          4          4        949        949          0          0          0       1286
  9    5993294       1442          0          0          3          3        718        718          0          0          0       1730
 10   13223300       3544          0          0         82         82       1690       1690          0          0          0         53
 11    9630981       2436          0          0         36         36       1182       1182          0          0          0      75223
 12   16792007       4646          0          0        270        270       2053       2053          0          0          0      21202
 13    6155506       1514          0          0         19         19        738        738          0          0          0      35333
 14    8544527       2006          0          0         19         19        984        984          0          0          0         68
 15   27874183       7088          0          0         68         68       3476       3476          0          0          0      76060
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.426s ( 1051.740 blocks/s,  65.734 Mb/s)
Attachment 36: Screenshot_from_2024-04-25_15-39-07.png
Screenshot_from_2024-04-25_15-39-07.png
Attachment 37: Screenshot_from_2024-04-25_15-39-14.png
Screenshot_from_2024-04-25_15-39-14.png
Attachment 38: Screenshot_from_2024-04-25_15-39-36.png
Screenshot_from_2024-04-25_15-39-36.png
Attachment 39: Screenshot_from_2024-04-25_15-39-53.png
Screenshot_from_2024-04-25_15-39-53.png
Attachment 40: Screenshot_from_2024-04-25_15-40-30.png
Screenshot_from_2024-04-25_15-40-30.png
Attachment 41: Screenshot_from_2024-04-25_15-40-48.png
Screenshot_from_2024-04-25_15-40-48.png
Attachment 42: Screenshot_from_2024-04-25_15-43-08.png
Screenshot_from_2024-04-25_15-43-08.png
  604   Wed Apr 24 21:39:51 2024 TD00:00-08:00 Thursday 25 April
22.43 24.4.24

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

      analysis of data file R18_266 - attachment 6
       max dead time 12% (aida04)
       no timewarps
       HEC data item rate 1.7kHz

03.37 DSSSD bias & leakage current OK - attachments 7-8

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10

      per FEE64 rate spectra - attachments 11

      Merger etc - attachment 12
Attachment 1: Screenshot_from_2024-04-24_22-43-23.png
Screenshot_from_2024-04-24_22-43-23.png
Attachment 2: Screenshot_from_2024-04-24_22-43-58.png
Screenshot_from_2024-04-24_22-43-58.png
Attachment 3: Screenshot_from_2024-04-24_22-44-50.png
Screenshot_from_2024-04-24_22-44-50.png
Attachment 4: Screenshot_from_2024-04-24_22-45-35.png
Screenshot_from_2024-04-24_22-45-35.png
Attachment 5: Screenshot_from_2024-04-24_22-46-21.png
Screenshot_from_2024-04-24_22-46-21.png
Attachment 6: R18_266
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254416606 (  820824.6 Hz)
        Other data format:    7503396 (   24208.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        846 (       2.7 Hz)
                               RESUME:        846 (       2.7 Hz)
                              SYNC100:      33155 (     107.0 Hz)
                              WR48-63:      33155 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7435394 (   23988.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     524404 (    1691.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.952 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.014                0.000
  1               16.009                0.000
  2                0.003                0.000
  3               37.388                0.000
  4                0.176                0.000
  5                0.544                0.000
  6                0.000                0.000
  7                0.784                0.000
  8                0.015                0.000
  9                0.000                0.000
 10                0.083                0.000
 11                0.004                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.017                0.000
 15                0.227                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6529814      10932          0          0          4          4        812        812          0       9300          0      46302
  1   32238460     350302          0          0        231        231       4129       4129          0     341582          0      92776
  2    8209450    3208628          0          0          2          2       1421       1421          0    3205782          0      32990
  3   48353626    2676061          0          0        374        374       6369       6369          0    2662575          0      27015
  4   10911439     520528          0          0          9          9       1454       1454          0     517602          0      45169
  5   18550590     703657          0          0         64         64       2488       2488          0     698553          0      35506
  6    6051492       1578          0          0          0          0        789        789          0          0          0      10690
  7   41812278      11146          0          0         85         85       5488       5488          0          0          0      97961
  8    3594476        900          0          0          4          4        446        446          0          0          0      18281
  9    5838926       1490          0          0          0          0        745        745          0          0          0       8125
 10   18621964       4722          0          0         27         27       2334       2334          0          0          0       3637
 11    5447469       1386          0          0          1          1        692        692          0          0          0      60590
 12    7038926       1642          0          0          0          0        821        821          0          0          0      12101
 13    5011795       1240          0          0          0          0        620        620          0          0          0      12487
 14    5028304       1286          0          0          2          2        641        641          0          0          0       7019
 15   31177597       7898          0          0         43         43       3906       3906          0          0          0      13755
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.508s (  984.379 blocks/s,  61.524 Mb/s)
Attachment 7: Screenshot_from_2024-04-25_03-37-40.png
Screenshot_from_2024-04-25_03-37-40.png
Attachment 8: Screenshot_from_2024-04-25_03-38-57.png
Screenshot_from_2024-04-25_03-38-57.png
Attachment 9: Screenshot_from_2024-04-25_03-39-38.png
Screenshot_from_2024-04-25_03-39-38.png
Attachment 10: Screenshot_from_2024-04-25_03-40-20.png
Screenshot_from_2024-04-25_03-40-20.png
Attachment 11: Screenshot_from_2024-04-25_03-40-52.png
Screenshot_from_2024-04-25_03-40-52.png
Attachment 12: Screenshot_from_2024-04-25_03-41-31.png
Screenshot_from_2024-04-25_03-41-31.png
  603   Wed Apr 24 18:44:05 2024 PP19:30 checks

All seems smooth.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_19-34-58.png
Screenshot_from_2024-04-24_19-34-58.png
Attachment 2: Screenshot_from_2024-04-24_19-37-12.png
Screenshot_from_2024-04-24_19-37-12.png
Attachment 3: Screenshot_from_2024-04-24_19-40-36.png
Screenshot_from_2024-04-24_19-40-36.png
Attachment 4: Screenshot_from_2024-04-24_19-41-39.png
Screenshot_from_2024-04-24_19-41-39.png
Attachment 5: Screenshot_from_2024-04-24_19-42-20.png
Screenshot_from_2024-04-24_19-42-20.png
Attachment 6: Screenshot_from_2024-04-24_19-42-53.png
Screenshot_from_2024-04-24_19-42-53.png
Attachment 7: Screenshot_from_2024-04-24_19-43-21.png
Screenshot_from_2024-04-24_19-43-21.png
Attachment 8: Screenshot_from_2024-04-24_19-43-41.png
Screenshot_from_2024-04-24_19-43-41.png
  602   Wed Apr 24 15:37:59 2024 PP16:00-00:00 shift Wednesday 24 April

All seems OK.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_16-38-45.png
Screenshot_from_2024-04-24_16-38-45.png
Attachment 2: Screenshot_from_2024-04-24_16-39-09.png
Screenshot_from_2024-04-24_16-39-09.png
Attachment 3: Screenshot_from_2024-04-24_16-39-38.png
Screenshot_from_2024-04-24_16-39-38.png
Attachment 4: Screenshot_from_2024-04-24_16-40-24.png
Screenshot_from_2024-04-24_16-40-24.png
Attachment 5: Screenshot_from_2024-04-24_16-41-15.png
Screenshot_from_2024-04-24_16-41-15.png
Attachment 6: Screenshot_from_2024-04-24_16-43-41.png
Screenshot_from_2024-04-24_16-43-41.png
Attachment 7: Screenshot_from_2024-04-24_16-44-15.png
Screenshot_from_2024-04-24_16-44-15.png
Attachment 8: Screenshot_from_2024-04-24_16-44-32.png
Screenshot_from_2024-04-24_16-44-32.png
  601   Wed Apr 24 09:04:55 2024 TD08:00-16:00 Wednesday 24 April
09.56 DAQ continues data file S100_alpha/R18_122
      FRS setting 162Eu

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK  - attachment 3
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 4-6

      WR timestamps OK - attachment 7

      ASIC settings aida01 & aida02 - attachments 8-9

      ADC, DISC, PAUSE, RESUME, WR & Correlation Scaler data time stats - attachments 10-15

      per FEE64 Rate spectra - attachments 16-18
       unusually can observe HEC Rates esp. in DSSSD #1 

      per p+n FEE64 1.8.L spectra - attachments 19-21
       aida09 pulser peak width 61 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 22

      per FEE64 1.8.H spectra - attachments 23-24

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 27-28

      ucesb - attachments 29-30
       spill cycle reported as 3.5s, extraction 1.5s

10.36 All histograms & stats zero'd

      BNC PB-5 settings - attachment 31

      DSSSD bias & leakage current - Grafana Alerting - attachment 32
       can observe bias & leakage current in greater detail
       DSSSD #1 bias stable, c. 10s nA micro structure observed in leakage current
       DSSSD #2 bias c. 10mV variations, micro structure observed in leakage current but smaller variations cf. DSSSD#1

10.46 analysis of data file R18_131
      max dead time 12% aida04, 6% aida02, all others 0.1% or less
      no timewarps
      HEC data item rates 1.6kHz - correcting for spill cycle => 2.3kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 760Hz in spill or 2700 implants per spill
      SC42 L/R currently report c. 2kHz





   

      
Attachment 1: Screenshot_from_2024-04-24_09-56-46.png
Screenshot_from_2024-04-24_09-56-46.png
Attachment 2: Screenshot_from_2024-04-24_09-57-04.png
Screenshot_from_2024-04-24_09-57-04.png
Attachment 3: Screenshot_from_2024-04-24_09-57-42.png
Screenshot_from_2024-04-24_09-57-42.png
Attachment 4: Screenshot_from_2024-04-24_09-57-57.png
Screenshot_from_2024-04-24_09-57-57.png
Attachment 5: Screenshot_from_2024-04-24_09-58-05.png
Screenshot_from_2024-04-24_09-58-05.png
Attachment 6: Screenshot_from_2024-04-24_09-58-19.png
Screenshot_from_2024-04-24_09-58-19.png
Attachment 7: Screenshot_from_2024-04-24_09-58-57.png
Screenshot_from_2024-04-24_09-58-57.png
Attachment 8: Screenshot_from_2024-04-24_09-59-09.png
Screenshot_from_2024-04-24_09-59-09.png
Attachment 9: Screenshot_from_2024-04-24_09-59-20.png
Screenshot_from_2024-04-24_09-59-20.png
Attachment 10: Screenshot_from_2024-04-24_09-59-47.png
Screenshot_from_2024-04-24_09-59-47.png
Attachment 11: Screenshot_from_2024-04-24_09-59-59.png
Screenshot_from_2024-04-24_09-59-59.png
Attachment 12: Screenshot_from_2024-04-24_10-00-21.png
Screenshot_from_2024-04-24_10-00-21.png
Attachment 13: Screenshot_from_2024-04-24_10-00-37.png
Screenshot_from_2024-04-24_10-00-37.png
Attachment 14: Screenshot_from_2024-04-24_10-00-51.png
Screenshot_from_2024-04-24_10-00-51.png
Attachment 15: Screenshot_from_2024-04-24_10-01-06.png
Screenshot_from_2024-04-24_10-01-06.png
Attachment 16: Screenshot_from_2024-04-24_10-01-31.png
Screenshot_from_2024-04-24_10-01-31.png
Attachment 17: Screenshot_from_2024-04-24_10-01-39.png
Screenshot_from_2024-04-24_10-01-39.png
Attachment 18: Screenshot_from_2024-04-24_10-02-48.png
Screenshot_from_2024-04-24_10-02-48.png
Attachment 19: Screenshot_from_2024-04-24_10-03-43.png
Screenshot_from_2024-04-24_10-03-43.png
Attachment 20: Screenshot_from_2024-04-24_10-04-03.png
Screenshot_from_2024-04-24_10-04-03.png
Attachment 21: Screenshot_from_2024-04-24_10-04-24.png
Screenshot_from_2024-04-24_10-04-24.png
Attachment 22: Screenshot_from_2024-04-24_10-06-03.png
Screenshot_from_2024-04-24_10-06-03.png
Attachment 23: Screenshot_from_2024-04-24_10-07-09.png
Screenshot_from_2024-04-24_10-07-09.png
Attachment 24: Screenshot_from_2024-04-24_10-08-01.png
Screenshot_from_2024-04-24_10-08-01.png
Attachment 25: Screenshot_from_2024-04-24_10-08-56.png
Screenshot_from_2024-04-24_10-08-56.png
Attachment 26: Screenshot_from_2024-04-24_10-09-21.png
Screenshot_from_2024-04-24_10-09-21.png
Attachment 27: Screenshot_from_2024-04-24_10-10-39.png
Screenshot_from_2024-04-24_10-10-39.png
Attachment 28: Screenshot_from_2024-04-24_10-10-47.png
Screenshot_from_2024-04-24_10-10-47.png
Attachment 29: Screenshot_from_2024-04-24_10-11-34.png
Screenshot_from_2024-04-24_10-11-34.png
Attachment 30: Screenshot_from_2024-04-24_10-11-40.png
Screenshot_from_2024-04-24_10-11-40.png
Attachment 31: Screenshot_from_2024-04-24_10-36-43.png
Screenshot_from_2024-04-24_10-36-43.png
Attachment 32: Screenshot_from_2024-04-24_10-38-18.png
Screenshot_from_2024-04-24_10-38-18.png
Attachment 33: R18_131
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254316594 (  804922.0 Hz)
        Other data format:    7603408 (   24065.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        827 (       2.6 Hz)
                               RESUME:        827 (       2.6 Hz)
                              SYNC100:      33177 (     105.0 Hz)
                              WR48-63:      33177 (     105.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7535400 (   23849.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     512028 (    1620.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      315.952 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               17.612                0.000
  2                0.021                0.000
  3               36.829                0.000
  4                0.071                0.000
  5                0.375                0.000
  6                0.000                0.000
  7                0.653                0.000
  8                0.004                0.000
  9                0.000                0.000
 10                0.011                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.010                0.000
 15                0.281                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5990261      11033          0          0          0          0        778        778          0       9477          0      48652
  1   32634474     342633          0          0        250        249       4180       4180          0     333774          0      88686
  2    8233792    3266215          0          0          3          3       1445       1445          0    3263319          0      33897
  3   49889631    2758758          0          0        373        374       6684       6684          0    2744643          0      26463
  4    7710704     511890          0          0          7          7       1031       1031          0     509814          0      33167
  5   17424106     678891          0          0         57         57       2202       2202          0     674373          0      32013
  6    6276423       1654          0          0          0          0        827        827          0          0          0       9743
  7   42348474      10910          0          0         79         79       5376       5376          0          0          0     105118
  8    3355535        852          0          0          1          1        425        425          0          0          0      18387
  9    5876369       1434          0          0          0          0        717        717          0          0          0       7335
 10   19688353       5108          0          0          5          5       2549       2549          0          0          0       3256
 11    5133774       1300          0          0          1          1        649        649          0          0          0      64128
 12    6809941       1758          0          0          0          0        879        879          0          0          0       9510
 13    5093601       1298          0          0          0          0        649        649          0          0          0      11354
 14    4801692       1232          0          0          1          1        615        615          0          0          0       7416
 15   33049464       8442          0          0         50         50       4171       4171          0          0          0      12903
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.441s ( 1086.905 blocks/s,  67.932 Mb/s)
  600   Tue Apr 23 22:37:46 2024 TD00:00-08:00 Wednesday 24 April
23.34 23.4.24

      DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

03.41 fDSSSD bias & leakage current OK - attachment 6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8

      per FEE64 rate spectra (off and on spill) - attachments 9-10

      Merger etc - attachment 11

03.54 analysis data file R18_67 - attachment 12
      max deadtime 11% (aida04)
      HEC data item rate c. 1.5kHz
      no timewarps

0.35 fDSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data item stats - attachment 15

      per FEE64 rate spectra - attachments 16

      Merger etc - attachment 17

03.54 analysis data file R18_96 - attachment 18
      max deadtime 12% (aida04)
      HEC data item rate c. 1.6kHz
      no timewarps
Attachment 1: Screenshot_from_2024-04-23_23-34-18.png
Screenshot_from_2024-04-23_23-34-18.png
Attachment 2: Screenshot_from_2024-04-23_23-34-54.png
Screenshot_from_2024-04-23_23-34-54.png
Attachment 3: Screenshot_from_2024-04-23_23-35-45.png
Screenshot_from_2024-04-23_23-35-45.png
Attachment 4: Screenshot_from_2024-04-23_23-36-31.png
Screenshot_from_2024-04-23_23-36-31.png
Attachment 5: Screenshot_from_2024-04-23_23-37-21.png
Screenshot_from_2024-04-23_23-37-21.png
Attachment 6: Screenshot_from_2024-04-24_03-41-40.png
Screenshot_from_2024-04-24_03-41-40.png
Attachment 7: Screenshot_from_2024-04-24_03-42-32.png
Screenshot_from_2024-04-24_03-42-32.png
Attachment 8: Screenshot_from_2024-04-24_03-42-58.png
Screenshot_from_2024-04-24_03-42-58.png
Attachment 9: Screenshot_from_2024-04-24_03-43-35.png
Screenshot_from_2024-04-24_03-43-35.png
Attachment 10: Screenshot_from_2024-04-24_03-44-11.png
Screenshot_from_2024-04-24_03-44-11.png
Attachment 11: Screenshot_from_2024-04-24_03-44-35.png
Screenshot_from_2024-04-24_03-44-35.png
Attachment 12: R18_67
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  250287937 (  518744.3 Hz)
        Other data format:   11632065 (   24108.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1106 (       2.3 Hz)
                               RESUME:       1109 (       2.3 Hz)
                              SYNC100:      33796 (      70.0 Hz)
                              WR48-63:      33796 (      70.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   11562258 (   23963.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     752941 (    1560.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      482.488 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               23.222                0.000
  2                0.000                0.000
  3               56.625                0.000
  4                0.000                0.000
  5                0.704                0.000
  6                0.000                0.000
  7                0.939                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.006                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3375676      15359          0          0          0          0        442        442          0      14475          0      75815
  1   47916692     537101          0          0        366        367       6254       6254          0     523860          0     139250
  2    3817331    4998323          0          0          0          0       1132       1132          0    4996059          0      51826
  3   75749746    4208495          0          0        548        550      10285      10285          0    4186827          0      40621
  4    3336977     792495          0          0          1          1        545        545          0     791403          0      53742
  5   23132211    1055976          0          0         83         83       3088       3088          0    1049634          0      49833
  6    3422365        890          0          0          0          0        445        445          0          0          0      14690
  7   56668346      14890          0          0        106        106       7339       7339          0          0          0     118883
  8    2824304        672          0          0          0          0        336        336          0          0          0      28810
  9    3844587        966          0          0          0          0        483        483          0          0          0      11512
 10    4370286       1248          0          0          0          0        624        624          0          0          0       5026
 11    3088851        802          0          0          0          0        401        401          0          0          0      98441
 12    5014612       1352          0          0          0          0        676        676          0          0          0      15213
 13    3462527        896          0          0          0          0        448        448          0          0          0      18001
 14    3185286        772          0          0          2          2        384        384          0          0          0      11128
 15    7078140       1828          0          0          0          0        914        914          0          0          0      20150
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.998s ( 1032.323 blocks/s,  64.520 Mb/s)
Attachment 13: Screenshot_from_2024-04-24_07-35-18.png
Screenshot_from_2024-04-24_07-35-18.png
Attachment 14: Screenshot_from_2024-04-24_07-36-00.png
Screenshot_from_2024-04-24_07-36-00.png
Attachment 15: Screenshot_from_2024-04-24_07-36-29.png
Screenshot_from_2024-04-24_07-36-29.png
Attachment 16: Screenshot_from_2024-04-24_07-37-03.png
Screenshot_from_2024-04-24_07-37-03.png
Attachment 17: Screenshot_from_2024-04-24_07-37-45.png
Screenshot_from_2024-04-24_07-37-45.png
Attachment 18: R18_96
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253809078 (  751826.0 Hz)
        Other data format:    8110924 (   24025.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        844 (       2.5 Hz)
                               RESUME:        844 (       2.5 Hz)
                              SYNC100:      33258 (      98.5 Hz)
                              WR48-63:      33258 (      98.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8042720 (   23823.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     529514 (    1568.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      337.590 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               19.161                0.000
  2                0.007                0.000
  3               41.205                0.000
  4                0.172                0.000
  5                0.406                0.000
  6                0.000                0.000
  7                0.656                0.000
  8                0.010                0.000
  9                0.000                0.000
 10                0.193                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.023                0.000
 15                0.121                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5871650      11620          0          0          0          0        746        746          0      10128          0      52136
  1   34432367     363819          0          0        263        263       4482       4482          0     354329          0      94194
  2    7680769    3497668          0          0          2          2       1401       1401          0    3494862          0      35954
  3   53024609    2927317          0          0        398        398       7008       7008          0    2912505          0      27672
  4    8502384     548708          0          0          9          9       1159       1159          0     546372          0      36407
  5   16750131     728930          0          0         53         53       2150       2150          0     724524          0      33989
  6    6256772       1674          0          0          0          0        837        837          0          0          0      10191
  7   41893078      10740          0          0         71         71       5299       5299          0          0          0      94711
  8    3653651        920          0          0          2          2        458        458          0          0          0      20092
  9    5991221       1564          0          0          0          0        782        782          0          0          0       8149
 10   16753048       4406          0          0          5          5       2198       2198          0          0          0       3402
 11    5222175       1254          0          0          1          1        626        626          0          0          0      68583
 12    7072961       1726          0          0          0          0        863        863          0          0          0      10498
 13    5337727       1396          0          0          0          0        698        698          0          0          0      12368
 14    4940099       1278          0          0          4          4        635        635          0          0          0       7688
 15   30426436       7904          0          0         36         36       3916       3916          0          0          0      13480
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.307s ( 1055.874 blocks/s,  65.992 Mb/s)
  599   Tue Apr 23 19:23:42 2024 PP20:30 checks

All looks good.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-23_20-24-09.png
Screenshot_from_2024-04-23_20-24-09.png
Attachment 2: Screenshot_from_2024-04-23_20-24-25.png
Screenshot_from_2024-04-23_20-24-25.png
Attachment 3: Screenshot_from_2024-04-23_20-24-52.png
Screenshot_from_2024-04-23_20-24-52.png
Attachment 4: Screenshot_from_2024-04-23_20-25-35.png
Screenshot_from_2024-04-23_20-25-35.png
Attachment 5: Screenshot_from_2024-04-23_20-26-16.png
Screenshot_from_2024-04-23_20-26-16.png
Attachment 6: Screenshot_from_2024-04-23_20-27-01.png
Screenshot_from_2024-04-23_20-27-01.png
Attachment 7: Screenshot_from_2024-04-23_20-27-26.png
Screenshot_from_2024-04-23_20-27-26.png
Attachment 8: Screenshot_from_2024-04-23_20-27-53.png
Screenshot_from_2024-04-23_20-27-53.png
  598   Tue Apr 23 15:11:50 2024 TDTuesday 23 April 16:00-00:00
16.12 Analysis of data file R17_101 - attachment 1

      max deadtime c. 2% (aida02)
      no timewarps observed


16.15 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 3

      per FEE64 rate spectra (on and off spill) - attachments 4-5

      Merger etc - attachment 6


16.53 aida04 stops producing data

      DAQ STOP fails for aida04
      Restart Merger to enable aida04 to complete DAQ STOP

      aida04 rebooted and DAQ restarted OK

      data file S100_alpha/R18
Attachment 1: R17_101
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  244392363 (  344543.7 Hz)
        Other data format:   17527637 (   24710.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        632 (       0.9 Hz)
                               RESUME:        632 (       0.9 Hz)
                              SYNC100:      34642 (      48.8 Hz)
                              WR48-63:      34642 (      48.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   17457089 (   24611.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     976870 (    1377.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      709.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               16.367                0.000
  2                0.000                0.000
  3                3.744                0.000
  4                0.000                0.000
  5                0.028                0.000
  6                0.000                0.000
  7                0.009                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.004                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3247904      22147          0          0          0          0        434        434          0      21279          0     100052
  1   78164199     878648          0          0        419        419      10303      10303          0     857204          0     222655
  2    3679894    7126519          0          0          0          0       1427       1427          0    7123665          0      69753
  3   82858158    7034348          0          0        202        202      11907      11907          0    7010130          0      82407
  4    2430024    1045502          0          0          0          0        415        415          0    1044672          0      72940
  5   14446252    1404299          0          0          6          6       2074       2074          0    1400139          0      68087
  6    2997919        898          0          0          0          0        449        449          0          0          0      20036
  7   28730414       7980          0          0          4          4       3986       3986          0          0          0      64499
  8    2617862        676          0          0          0          0        338        338          0          0          0      38168
  9    3117608        872          0          0          0          0        436        436          0          0          0      15569
 10    3647866        954          0          0          0          0        477        477          0          0          0       7030
 11    3150764        810          0          0          0          0        405        405          0          0          0     129790
 12    4169396       1050          0          0          0          0        525        525          0          0          0      20235
 13    2725572        706          0          0          0          0        353        353          0          0          0      23658
 14    2606679        622          0          0          1          1        310        310          0          0          0      15170
 15    5801852       1606          0          0          0          0        803        803          0          0          0      26821
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.402s ( 1052.550 blocks/s,  65.784 Mb/s)
Attachment 2: Screenshot_from_2024-04-23_16-15-38.png
Screenshot_from_2024-04-23_16-15-38.png
Attachment 3: Screenshot_from_2024-04-23_16-16-52.png
Screenshot_from_2024-04-23_16-16-52.png
Attachment 4: Screenshot_from_2024-04-23_16-17-40.png
Screenshot_from_2024-04-23_16-17-40.png
Attachment 5: Screenshot_from_2024-04-23_16-17-25.png
Screenshot_from_2024-04-23_16-17-25.png
Attachment 6: Screenshot_from_2024-04-23_16-18-04.png
Screenshot_from_2024-04-23_16-18-04.png
Attachment 7: Screenshot_from_2024-04-23_16-22-04.png
Screenshot_from_2024-04-23_16-22-04.png
Attachment 8: Screenshot_from_2024-04-23_17-18-08.png
Screenshot_from_2024-04-23_17-18-08.png
Attachment 9: Screenshot_from_2024-04-23_17-18-34.png
Screenshot_from_2024-04-23_17-18-34.png
Attachment 10: Screenshot_from_2024-04-23_17-19-01.png
Screenshot_from_2024-04-23_17-19-01.png
Attachment 11: Screenshot_from_2024-04-23_17-19-18.png
Screenshot_from_2024-04-23_17-19-18.png
Attachment 12: Screenshot_from_2024-04-23_17-20-08.png
Screenshot_from_2024-04-23_17-20-08.png
Attachment 13: Screenshot_from_2024-04-23_17-20-47.png
Screenshot_from_2024-04-23_17-20-47.png
Attachment 14: Screenshot_from_2024-04-23_17-22-09.png
Screenshot_from_2024-04-23_17-22-09.png
Attachment 15: Screenshot_from_2024-04-23_17-22-09.png
Screenshot_from_2024-04-23_17-22-09.png
Attachment 16: Screenshot_from_2024-04-23_17-25-10.png
Screenshot_from_2024-04-23_17-25-10.png
Attachment 17: Screenshot_from_2024-04-23_17-29-21.png
Screenshot_from_2024-04-23_17-29-21.png
Attachment 18: Screenshot_from_2024-04-23_17-29-21.png
Screenshot_from_2024-04-23_17-29-21.png
Attachment 19: Screenshot_from_2024-04-23_17-29-40.png
Screenshot_from_2024-04-23_17-29-40.png
Attachment 20: Screenshot_from_2024-04-23_17-30-04.png
Screenshot_from_2024-04-23_17-30-04.png
  597   Tue Apr 23 13:15:43 2024 PP14:00 checks

All looks good.

Screenshots attached

Attachment 1: Screenshot_from_2024-04-23_14-11-33.png
Screenshot_from_2024-04-23_14-11-33.png
Attachment 2: Screenshot_from_2024-04-23_14-04-26.png
Screenshot_from_2024-04-23_14-04-26.png
Attachment 3: Screenshot_from_2024-04-23_14-04-44.png
Screenshot_from_2024-04-23_14-04-44.png
Attachment 4: Screenshot_from_2024-04-23_14-06-02.png
Screenshot_from_2024-04-23_14-06-02.png
Attachment 5: Screenshot_from_2024-04-23_14-07-07.png
Screenshot_from_2024-04-23_14-07-07.png
Attachment 6: Screenshot_from_2024-04-23_14-09-26.png
Screenshot_from_2024-04-23_14-09-26.png
Attachment 7: Screenshot_from_2024-04-23_14-10-55.png
Screenshot_from_2024-04-23_14-10-55.png
Attachment 8: Screenshot_from_2024-04-23_14-11-16.png
Screenshot_from_2024-04-23_14-11-16.png
  595   Tue Apr 23 13:08:28 2024 TDSummary of AIDA FEE64 stops
We have observed a number of AIDA FEE64s stop producing data.

Summary of dates & (approximate) times

aida03	21.4.24	08.20
aida04	22.4.24	09.14             https://elog.ph.ed.ac.uk/DESPEC/586
aida01	22.4.24	16.00
aida02	22.4.24	22.00
aida02	23.4.24	10.21
aida02	23.4.24 11.25
aida04  23.4.24 16.53?            attachment 2
aida04/aida05 25.4.24 22.28       no messages in aida04 system console
aida04  27.8.24 c.23:00
aida04  28.8.24 c. 12-13:00       power cycle
aida04  28.8.24 c. 16.00          reset per Elog 618
aida02  28.4.24 c. 22.20          reset per Elog 618, no error messages in system console
aida04  29.4.24 01.15
aida04  30.4.24 <03.30
aida02  30.4.24 <10.00


aida02 2.5.24 c. 19.30 reset


In future, it would be useful to copy system console reports to the Elog to check whether we can identify common error messages etc.


ssh pi@nnrpi1
<password>

cd logs
cat /ttyUSBx 

The current mapping of USB port # to AIDA FEE64s - attachment 1


aida02 system console c. 19.30 2.5.24

02:05:24/14:40:51|executing generic doStop
02:05:24/19:27:28|Halt (1)
02:05:24/19:27:29|Halt (2)
02:05:24/19:27:29|Halt (3)
02:05:24/19:27:30|Halt (4)
02:05:24/19:27:30|Halt (5)
02:05:24/19:27:31|Halt (6)
02:05:24/19:27:31|Halt (7)
02:05:24/19:27:32|Halt (8)
02:05:24/19:27:32|Halt (9)
02:05:24/19:27:33|Halt (10)
02:05:24/19:27:33|Halt (11)
02:05:24/19:27:34|Halt (12)
02:05:24/19:27:34|Halt (13)
02:05:24/19:27:35|Halt (14)
02:05:24/19:27:35|Halt (15)
02:05:24/19:27:36|Halt (16)
02:05:24/19:27:36|Halt (17)
02:05:24/19:27:37|Halt (18)
02:05:24/19:27:38|Halt (19)
02:05:24/19:27:38|Action has not completed
02:05:24/19:27:39|Giving up waiting
02:05:24/19:27:39|disconnect xfer stream #0 (1)
02:05:24/19:27:39|disconnect xfer stream #0 (2)
02:05:24/19:27:39|disconnect xfer stream #0 (3)
02:05:24/19:27:40|disconnect xfer stream #0 (4)
02:05:24/19:27:40|disconnect xfer stream #0 (5)
02:05:24/19:27:41|disconnect xfer stream #0 (6)
02:05:24/19:27:41|disconnect xfer stream #0 (7)
02:05:24/19:27:42|disconnect xfer stream #0 (8)
02:05:24/19:27:42|disconnect xfer stream #0 (9)
02:05:24/19:27:43|disconnect xfer stream #0 (10)
02:05:24/19:27:43|disconnect xfer stream #0 (11)
02:05:24/19:27:44|disconnect xfer stream #0 (12)
02:05:24/19:27:44|disconnect xfer stream #0 (13)
02:05:24/19:27:45|disconnect xfer stream #0 (14)
02:05:24/19:27:46|disconnect xfer stream #0 (15)
02:05:24/19:27:46|disconnect xfer stream #0 (16)
02:05:24/19:27:47|disconnect xfer stream #0 (17)
02:05:24/19:27:47|disconnect xfer stream #0 (18)
02:05:24/19:27:48|disconnect xfer stream #0 (19)
02:05:24/19:27:48|Action has not completed
02:05:24/19:27:49|Giving up waiting
02:05:24/19:27:49|executing generic Stop_Merger
02:05:24/19:27:49|executing generic Stop_DataStorageServer
02:05:24/19:27:49|completed generic doStop
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:28:01|Transfer Error - : Connection reset by peer
02:05:24/19:28:08|send() failed: 
02:05:24/19:28:08|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
02:05:24/19:28:08|do_GetState returned z=0 and 0
02:05:24/19:28:13|executing generic doGo
02:05:24/19:28:45|Initialising communication with storage server 192.168.11.99 on port 11002
02:05:24/19:28:45|Setting Transfer Block Size 65536
02:05:24/19:28:45|Setting Transfer Mode 3
02:05:24/19:28:45|Setting Overlap Mode 0
02:05:24/19:28:45|Setting Blocking Mode 0
02:05:24/19:28:45|Setting nice 0
02:05:24/19:28:45|TCP transfer library version 3.12
02:05:24/19:28:45|TCP socket send buffer was 16384 - now 221184
02:05:24/19:28:45|TCP socket receive buffer was 87380 - now 221184
02:05:24/19:28:45|TCP socket created OK - now connecting to 192.168.11.99 port 11002
02:05:24/19:28:45|Connected to 192.168.11.99 port 11002
02:05:24/19:28:45|connect xfer stream #0 (1)
02:05:24/19:28:45|Data Acquisition Statistics counters now cleared
02:05:24/19:28:46|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
02:05:24/19:28:46|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
02:05:24/19:28:46|Go (1)
02:05:24/19:28:46|Running Go_Electronics for aida02 with TSMaster aida01
02:05:24/19:28:46|ADCs Calibrated. 
02:05:24/19:28:46|Finished Go_Electronics for AIDA
02:05:24/19:28:46|executing generic Go_Merger
02:05:24/19:28:46|executing generic Go_DataStorageServer
02:05:24/19:28:46|completed generic doGo
02:05:24/19:28:46|do_GetState returned z=0 and 1
02:05:24/19:28:46|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
02:05:24/19:28:51|Also in RDOGo_Operate: Enabled waveform readout
02:05:24/19:28:51|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0201
02:05:24/19:28:51|WV:0:Buffer info. Length = 64 : Offset = 0 
02:05:24/19:28:51|WV:1:Buffer info. Length = 62 : Offset = 2 
02:05:24/19:28:51|WV:2:Buffer info. Length = 60 : Offset = 4 
02:05:24/19:28:52|WV:3:Buffer info. Length = 58 : Offset = 6 
02:05:24/19:28:52|WV:4:Buffer info. Length = 56 : Offset = 8 
02:05:24/19:28:52|WV:5:Buffer info. Length = 54 : Offset = 10 
02:05:24/19:28:52|WV:6:Buffer info. Length = 52 : Offset = 12 
02:05:24/19:28:52|WV:7:Buffer info. Length = 50 : Offset = 14 
02:05:24/19:28:52|WV:8:Buffer info. Length = 48 : Offset = 16 
02:05:24/19:28:52|WV:9:Buffer info. Length = 46 : Offset = 18 
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 1048, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|
Attachment 1: Screenshot_from_2024-04-23_14-22-04.png
Screenshot_from_2024-04-23_14-22-04.png
Attachment 2: aida04
23:04:24/11:48:07|TCP transfer library version 3.12
23:04:24/11:48:07|TCP socket send buffer was 16384 - now 221184
23:04:24/11:48:07|TCP socket receive buffer was 87380 - now 221184
23:04:24/11:48:07|TCP socket created OK - now connecting to 192.168.11.99 port 11004
23:04:24/11:48:07|Connected to 192.168.11.99 port 11004
23:04:24/11:48:07|connect xfer stream #0 (1)
23:04:24/11:48:07|Data Acquisition Statistics counters now cleared
23:04:24/11:48:07|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
23:04:24/11:48:07|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
23:04:24/11:48:07|Go (1)
23:04:24/11:48:07|Running Go_Electronics for aida04 with TSMaster aida01
23:04:24/11:48:07|ADCs Calibrated. 
23:04:24/11:48:07|Finished Go_Electronics for AIDA
23:04:24/11:48:08|executing generic Go_Merger
23:04:24/11:48:08|executing generic Go_DataStorageServer
23:04:24/11:48:08|completed generic doGo
23:04:24/11:48:08|do_GetState returned z=0 and 1
23:04:24/11:48:08|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
23:04:24/11:48:13|Also in RDOGo_Operate: Enabled waveform readout
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 104A, State machines = 0201
23:04:24/11:48:13|WV:0:Buffer info. Length = 64 : Offset = 0 
23:04:24/11:48:13|WV:1:Buffer info. Length = 62 : Offset = 2 
23:04:24/11:48:13|WV:2:Buffer info. Length = 60 : Offset = 4 
23:04:24/11:48:13|WV:3:Buffer info. Length = 58 : Offset = 6 
23:04:24/11:48:13|WV:4:Buffer info. Length = 56 : Offset = 8 
23:04:24/11:48:13|WV:5:Buffer info. Length = 54 : Offset = 10 
23:04:24/11:48:13|WV:6:Buffer info. Length = 52 : Offset = 12 
23:04:24/11:48:13|WV:7:Buffer info. Length = 50 : Offset = 14 
23:04:24/11:48:13|WV:8:Buffer info. Length = 48 : Offset = 16 
23:04:24/11:48:13|WV:9:Buffer info. Length = 46 : Offset = 18 
23:04:24/11:48:13|get_WAVEBlk (A) which = 1 ---- status = 0188, State machines = 0201
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:13|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|executing generic doStop
23:04:24/16:58:30|Halt (1)
23:04:24/16:58:31|Halt (2)
23:04:24/16:58:32|Halt (3)
23:04:24/16:58:32|Halt (4)
23:04:24/16:58:33|Halt (5)
23:04:24/16:58:33|Halt (6)
23:04:24/16:58:34|Halt (7)
23:04:24/16:58:35|Halt (8)
23:04:24/16:58:35|Halt (9)
23:04:24/16:58:36|Halt (10)
23:04:24/16:58:36|Halt (11)
23:04:24/16:58:37|Halt (12)
23:04:24/16:58:37|Halt (13)
23:04:24/16:58:38|Halt (14)
23:04:24/16:58:39|Halt (15)
23:04:24/16:58:39|Halt (16)
23:04:24/16:58:40|Halt (17)
23:04:24/16:58:40|Halt (18)
23:04:24/16:58:41|Halt (19)
23:04:24/16:58:41|Action has not completed
23:04:24/16:58:42|Giving up waiting
23:04:24/16:58:42|disconnect xfer stream #0 (1)
23:04:24/16:58:42|disconnect xfer stream #0 (2)
23:04:24/16:58:43|disconnect xfer stream #0 (3)
23:04:24/16:58:43|disconnect xfer stream #0 (4)
23:04:24/16:58:44|disconnect xfer stream #0 (5)
23:04:24/16:58:44|disconnect xfer stream #0 (6)
23:04:24/16:58:45|disconnect xfer stream #0 (7)
23:04:24/16:58:45|disconnect xfer stream #0 (8)
23:04:24/16:58:46|disconnect xfer stream #0 (9)
23:04:24/16:58:47|disconnect xfer stream #0 (10)
23:04:24/16:58:47|disconnect xfer stream #0 (11)
23:04:24/16:58:48|disconnect xfer stream #0 (12)
23:04:24/16:58:48|disconnect xfer stream #0 (13)
23:04:24/16:58:49|disconnect xfer stream #0 (14)
23:04:24/16:58:49|disconnect xfer stream #0 (15)
23:04:24/16:58:50|disconnect xfer stream #0 (16)
23:04:24/16:58:51|disconnect xfer stream #0 (17)
23:04:24/16:58:51|disconnect xfer stream #0 (18)
23:04:24/16:58:52|disconnect xfer stream #0 (19)
23:04:24/16:58:52|Action has not completed
23:04:24/16:58:53|Giving up waiting
23:04:24/16:58:53|executing generic Stop_Merger
23:04:24/16:58:53|executing generic Stop_DataStorageServer
23:04:24/16:58:53|completed generic doStop
23:04:24/16:58:53|do_GetState returned z=0 and 1
23:04:24/16:58:53|do_GetState returned z=0 and 1
23:04:24/16:59:06|Transfer Error - : Connection reset by peer
23:04:24/16:59:15|send() failed: 
23:04:24/16:59:15|TCP transfer library version 3.12
23:04:24/16:59:15|TCP socket send buffer was 16384 - now 221184
23:04:24/16:59:15|TCP socket receive buffer was 87380 - now 221184
23:04:24/16:59:15|TCP socket created OK - now connecting to 192.168.11.99 port 11004
23:04:24/16:59:15|Transfer Error - : Connection refused
23:04:24/16:59:15|connect() failed:  
23:04:24/16:59:15|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
23:04:24/16:59:15|do_GetState returned z=0 and 0
23:04:24/16:59:18|do_GetState returned z=0 and 0
23:04:24/17:00:09|do_GetState returned z=0 and 0
23:04:24/17:00:11|do_GetState returned z=0 and 0
23:04:24/17:00:19|do_GetState returned z=0 and 0
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
23:04:24/17:00:43|[  OK  ]
23:04:24/17:00:43|Stopping rpcbind: [  OK  ]
23:04:24/17:00:45|Shutting down kernel logger: [  OK  ]
23:04:24/17:00:47|Shutting down system logger: [  OK  ]
23:04:24/17:00:47|Stopping xinetd: [  OK  ]
23:04:24/17:00:47|[  OK  ]
23:04:24/17:00:47|Sending all processes the TERM signal... xaida: release:
23:04:24/17:00:48|
23:04:24/17:00:48|AIDAMEM: aidamem_release:
23:04:24/17:00:48|xaida: release:
23:04:24/17:00:48|Sending all processes the KILL signal... 
23:04:24/17:00:53|Saving random seed:  
23:04:24/17:00:53|Syncing hardware clock to systxaida: open:
23:04:24/17:00:53|emxaida: release:
23:04:24/17:00:53| time xaida: open:
23:04:24/17:00:53|xaida: ioctl: called with command 536899587
23:04:24/17:00:53|xaida: ioctl: called with command 1076129801
23:04:24/17:00:53|RTCxaida: release:
23:04:24/17:00:53|_RD_TIME: Inappropriate ioctl for device
23:04:24/17:00:53|ioctl() to /dev/rtc to read the time failed.
23:04:24/17:00:53|
23:04:24/17:00:53|Please stand by while rebooting the system...
Sj5õ24/17:00:54|Restarting system.
23:04:24/17:00:56|U×®­Ë
                        )	$T,W¤']Z¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:00:56|
Finished copying zImage to RAM 
23:04:24/17:00:57|
Found 0 errors checking kernel image
23:04:24/17:00:58|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
23:04:24/17:00:58|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
23:04:24/17:00:58|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
23:04:24/17:00:58|SPIconfig done now to set up the LMK3200 registers
23:04:24/17:00:58|LMK #0 : regInit[0]=0x80000000
23:04:24/17:00:58|LMK #0 : regInit[1]=0x10070600
23:04:24/17:00:58|LMK #0 : regInit[2]=0x60601
23:04:24/17:00:59|LMK #0 : regInit[3]=0x60602
23:04:24/17:00:59|LMK #0 : regInit[4]=0x60603
23:04:24/17:00:59|LMK #0 : regInit[5]=0x70624
23:04:24/17:00:59|LMK #0 : regInit[6]=0x70605
23:04:24/17:00:59|LMK #0 : regInit[7]=0x70606
23:04:24/17:00:59|LMK #0 : regInit[8]=0x70627
23:04:24/17:00:59|LMK #0 : regInit[9]=0x10000908
23:04:24/17:00:59|LMK #0 : regInit[10]=0xA0022A09
23:04:24/17:00:59|LMK #0 : regInit[11]=0x82800B
23:04:24/17:00:59|LMK #0 : regInit[12]=0x28C800D
23:04:24/17:00:59|LMK #0 : regInit[13]=0x830020E
23:04:24/17:00:59|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
23:04:24/17:00:59|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
23:04:24/17:00:59|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
23:04:24/17:01:00|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
23:04:24/17:01:00|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
23:04:24/17:01:03|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xa0 0x71 
23:04:24/17:01:09|Using Xilinx Virtex440 machine description
23:04:24/17:01:10|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
23:04:24/17:01:10|Zone PFN ranges:
23:04:24/17:01:10|  DMA      0x00000000 -> 0x00007000
23:04:24/17:01:10|  Normal   0x00007000 -> 0x00007000
23:04:24/17:01:10|Movable zone start PFN for each node
23:04:24/17:01:10|early_node_map[1] active PFN ranges
23:04:24/17:01:10|    0: 0x00000000 -> 0x00007000
23:04:24/17:01:10|MMU: Allocated 1088 bytes of context maps for 255 contexts
23:04:24/17:01:10|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
23:04:24/17:01:10|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
23:04:24/17:01:10|PID hash table entries: 512 (order: 9, 2048 bytes)
23:04:24/17:01:10|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
23:04:24/17:01:11|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
23:04:24/17:01:11|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
23:04:24/17:01:11|Kernel virtual memory layout:
23:04:24/17:01:11|  * 0xffffe000..0xfffff000  : fixmap
23:04:24/17:01:11|  * 0xfde00000..0xfe000000  : consistent mem
23:04:24/17:01:11|  * 0xfde00000..0xfde00000  : early ioremap
23:04:24/17:01:11|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
23:04:24/17:01:11|NR_IRQS:512
23:04:24/17:01:11|clocksource: timebase mult[a00000] shift[22] registered
23:04:24/17:01:11|Console: colour dummy device 80x25
23:04:24/17:01:11|Mount-cache hash table entries: 512
23:04:24/17:01:11|NET: Registered protocol family 16
23:04:24/17:01:11|PCI: Probing PCI hardware
23:04:24/17:01:11|bio: create slab <bio-0> at 0
23:04:24/17:01:11|NET: Registered protocol family 2
23:04:24/17:01:11|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
23:04:24/17:01:11|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
23:04:24/17:01:11|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
23:04:24/17:01:11|TCP: Hash tables configured (established 4096 bind 4096)
23:04:24/17:01:11|TCP reno registered
23:04:24/17:01:12|NET: Registered protocol family 1
23:04:24/17:01:12|ROMFS MTD (C) 2007 Red Hat, Inc.
23:04:24/17:01:12|msgmni has been set to 214
23:04:24/17:01:12|io scheduler noop registered
23:04:24/17:01:12|io scheduler anticipatory registered
23:04:24/17:01:12|io scheduler deadline registered
23:04:24/17:01:12|io scheduler cfq registered (default)
23:04:24/17:01:12|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
23:04:24/17:01:12|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
23:04:24/17:01:12|console [ttyS0] enabled
23:04:24/17:01:12|brd: module loaded
23:04:24/17:01:12|loop: module loaded
23:04:24/17:01:12|Device Tree Probing 'ethernet'
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:a0:71
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
23:04:24/17:01:12|XLlTemac: DCR address: 0x80
23:04:24/17:01:12|XLlTemac: buffer descriptor size: 32768 (0x8000)
23:04:24/17:01:12|XLlTemac: Allocating DMA descriptors with kmalloc
23:04:24/17:01:12|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
23:04:24/17:01:12|XTemac: PHY detected at address 7.
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
23:04:24/17:01:13|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13|Using buffer write method
23:04:24/17:01:13|cfi_cmdset_0001: Erase suspend on write enabled
23:04:24/17:01:13|cmdlinepart partition parsing not available
23:04:24/17:01:13|RedBoot partition parsing not available
23:04:24/17:01:13|Creating 5 MTD partitions on "fc000000.flash":
23:04:24/17:01:13|0x000000000000-0x000000500000 : "golden_firmware"
23:04:24/17:01:13|0x000000500000-0x000000800000 : "golden_kernel"
23:04:24/17:01:13|0x000000800000-0x000000d00000 : "user_firmware"
23:04:24/17:01:13|0x000000d00000-0x000000fe0000 : "user_kernel"
23:04:24/17:01:13|0x000000fe0000-0x000001000000 : "env_variables"
23:04:24/17:01:13|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
23:04:24/17:01:13|SPI: XIlinx spi: bus number now 32766
23:04:24/17:01:13|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
23:04:24/17:01:14|SPI: XIlinx spi: bus number now 32765
23:04:24/17:01:14|mice: PS/2 mouse device common for all mice
23:04:24/17:01:14|Device Tree Probing 'i2c'
23:04:24/17:01:14| #0 at 0x81600000 mapped to 0xD1030000, irq=22
23:04:24/17:01:14|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
23:04:24/17:01:14|TCP cubic registered
23:04:24/17:01:14|NET: Registered protocol family 17
23:04:24/17:01:14|RPC: Registered udp transport module.
23:04:24/17:01:14|RPC: Registered tcp transport module.
23:04:24/17:01:14|eth0: XLlTemac: Options: 0x3fa
23:04:24/17:01:14|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
23:04:24/17:01:14|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
23:04:24/17:01:15|eth0: XLlTemac: speed set to 1000Mb/s
23:04:24/17:01:17|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
23:04:24/17:01:17|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
23:04:24/17:01:17|Sending DHCP requests ., OK
23:04:24/17:01:18|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
23:04:24/17:01:18|IP-Config: Complete:
23:04:24/17:01:18|     device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
23:04:24/17:01:18|     host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
23:04:24/17:01:18|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
23:04:24/17:01:18|Looking up port of RPC 100003/2 on 192.168.11.99
23:04:24/17:01:18|Looking up port of RPC 100005/1 on 192.168.11.99
23:04:24/17:01:18|VFS: Mounted root (nfs filesystem) on device 0:12.
23:04:24/17:01:18|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
23:04:24/17:01:19|Starting sysinit...
23:04:24/17:01:19|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
23:04:24/17:01:20|		Press 'I' to enter interactive startup.
23:04:24/17:01:20|Setting clock  (utc): Tue Apr 23 16:01:20 BST 2024 [  OK  ]
23:04:24/17:01:20|Building the cache [  OK  ]
23:04:24/17:01:20|Setting hostname aida04:  [  OK  ]
23:04:24/17:01:21|Mounting local filesystems:  [  OK  ]
23:04:24/17:01:22|Enabling /etc/fstab swaps:  [  OK  ]
23:04:24/17:01:24|Finishing sysinit...
INIT: Entering runlevel: 3
... 257 more lines ...
  594   Tue Apr 23 11:09:00 2024 PP12:00 checks

Everything is OK now.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-23_12-01-49.png
Screenshot_from_2024-04-23_12-01-49.png
Attachment 2: Screenshot_from_2024-04-23_12-02-24.png
Screenshot_from_2024-04-23_12-02-24.png
Attachment 3: Screenshot_from_2024-04-23_12-03-36.png
Screenshot_from_2024-04-23_12-03-36.png
Attachment 4: Screenshot_from_2024-04-23_12-04-05.png
Screenshot_from_2024-04-23_12-04-05.png
Attachment 5: Screenshot_from_2024-04-23_12-05-52.png
Screenshot_from_2024-04-23_12-05-52.png
Attachment 6: Screenshot_from_2024-04-23_12-06-42.png
Screenshot_from_2024-04-23_12-06-42.png
Attachment 7: Screenshot_from_2024-04-23_12-07-41.png
Screenshot_from_2024-04-23_12-07-41.png
Attachment 8: Screenshot_from_2024-04-23_12-08-32.png
Screenshot_from_2024-04-23_12-08-32.png
  593   Tue Apr 23 10:52:39 2024 PPAida02 down and back

We lost aida02 at about 11:25 again. Nick fixed it about 11:50.

It is now back in the DAQ.

  592   Tue Apr 23 10:10:31 2024 PPaida02 is back

Nick rebooted AIDA and aida02 came back at around 10:55.

AIDA was out of the DAQ from about 10:30 until 10:55

  591   Tue Apr 23 09:21:01 2024 PP10:20 checks - aida02 is down

Aida02 is down. Contacted Nick.

The rest looks stable, see screenshots.

10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14

Go > All 16 writing to merger, looks OK

 

aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again

Attachment 1: Screenshot_from_2024-04-23_10-08-54.png
Screenshot_from_2024-04-23_10-08-54.png
Attachment 2: Screenshot_from_2024-04-23_10-13-23.png
Screenshot_from_2024-04-23_10-13-23.png
Attachment 3: Screenshot_from_2024-04-23_10-15-28.png
Screenshot_from_2024-04-23_10-15-28.png
Attachment 4: Screenshot_from_2024-04-23_10-16-23.png
Screenshot_from_2024-04-23_10-16-23.png
Attachment 5: Screenshot_from_2024-04-23_10-16-47.png
Screenshot_from_2024-04-23_10-16-47.png
Attachment 6: Screenshot_from_2024-04-23_10-17-47.png
Screenshot_from_2024-04-23_10-17-47.png
Attachment 7: Screenshot_from_2024-04-23_10-19-30.png
Screenshot_from_2024-04-23_10-19-30.png
Attachment 8: Screenshot_from_2024-04-23_10-20-20.png
Screenshot_from_2024-04-23_10-20-20.png
  590   Tue Apr 23 07:27:18 2024 PP08:00-16:00 Tuesday 24 April shift

Checks 08:00-08:30

Screenshots are attached and everything looks OK.

Attachment 1: Screenshot_from_2024-04-23_08-10-21.png
Screenshot_from_2024-04-23_08-10-21.png
Attachment 2: Screenshot_from_2024-04-23_08-13-20.png
Screenshot_from_2024-04-23_08-13-20.png
Attachment 3: Screenshot_from_2024-04-23_08-15-30.png
Screenshot_from_2024-04-23_08-15-30.png
Attachment 4: Screenshot_from_2024-04-23_08-18-28.png
Screenshot_from_2024-04-23_08-18-28.png
Attachment 5: Screenshot_from_2024-04-23_08-23-07.png
Screenshot_from_2024-04-23_08-23-07.png
Attachment 6: Screenshot_from_2024-04-23_08-26-31.png
Screenshot_from_2024-04-23_08-26-31.png
Attachment 7: Screenshot_from_2024-04-23_08-35-31.png
Screenshot_from_2024-04-23_08-35-31.png
Attachment 8: Screenshot_from_2024-04-23_08-39-43.png
Screenshot_from_2024-04-23_08-39-43.png
  589   Mon Apr 22 22:35:59 2024 TD00.00-08.00 Tuesday 22 April
23.36 Analysis of data file R17_34 - attachment 1
      max deadtime c. 1% (aida02)
      no timewarps

23.38 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 3
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

01.03 DSSSD bias & leakage current OK - attachment 7

      FEE64 temperatures OK - attachment 8
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 9

      per FEE64 Rate spectra - attachment 10

      Merger etc - attachment 11

03.06 DSSSD bias & leakage current OK - attachment 12

      FEE64 temperatures OK - attachment 13
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 14

      per FEE64 Rate spectra - attachment 15

      Merger etc - attachment 16

05.04 DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Merger etc - attachment 21

07.29 DSSSD bias & leakage current OK - attachment 22

      FEE64 temperatures OK - attachment 23
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 24

      per FEE64 Rate spectra - attachment 25

      Merger etc - attachment 26
Attachment 1: R17_34
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  242086746 (  241481.1 Hz)
        Other data format:   19833254 (   19783.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        347 (       0.3 Hz)
                               RESUME:        347 (       0.3 Hz)
                              SYNC100:      35735 (      35.6 Hz)
                              WR48-63:      35735 (      35.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   19761090 (   19711.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     676877 (     675.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     1002.508 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                7.276                0.000
  2                0.000                0.000
  3                3.384                0.000
  4                0.000                0.000
  5                0.005                0.000
  6                0.000                0.000
  7                0.131                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.020                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2611636      30762          0          0          0          0        342        342          0      30078          0      64523
  1   15105593     133663          0          0        117        117       1949       1949          0     129531          0      31773
  2    4737354    9053995          0          0          0          0       1811       1811          0    9050373          0      45719
  3   93081574    8973436          0          0        187        187      14170      14170          0    8944722          0      56135
  4    2149658     687339          0          0          0          0        339        339          0     686661          0      47171
  5   16270251     924233          0          0          3          3       2251       2251          0     919725          0      42612
  6    4771409       1270          0          0          0          0        635        635          0          0          0      12858
  7   64665607      18074          0          0         39         39       8998       8998          0          0          0     197755
  8    1902284        428          0          0          0          0        214        214          0          0          0      25268
  9    4387089       1362          0          0          0          0        681        681          0          0          0      10256
 10    8523582       2388          0          0          0          0       1194       1194          0          0          0       4627
 11    2287700        576          0          0          0          0        288        288          0          0          0      83305
 12    4391198       1112          0          0          0          0        556        556          0          0          0      12633
 13    3103598        860          0          0          0          0        430        430          0          0          0      15208
 14    1965500        506          0          0          0          0        253        253          0          0          0      10217
 15   12132713       3250          0          0          1          1       1624       1624          0          0          0      16817
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.539s ( 1047.838 blocks/s,  65.490 Mb/s)
Attachment 2: Screenshot_from_2024-04-22_23-38-08.png
Screenshot_from_2024-04-22_23-38-08.png
Attachment 3: Screenshot_from_2024-04-22_23-39-17.png
Screenshot_from_2024-04-22_23-39-17.png
Attachment 4: Screenshot_from_2024-04-22_23-39-44.png
Screenshot_from_2024-04-22_23-39-44.png
Attachment 5: Screenshot_from_2024-04-22_23-40-27.png
Screenshot_from_2024-04-22_23-40-27.png
Attachment 6: Screenshot_from_2024-04-22_23-40-54.png
Screenshot_from_2024-04-22_23-40-54.png
Attachment 7: Screenshot_from_2024-04-23_01-04-32.png
Screenshot_from_2024-04-23_01-04-32.png
Attachment 8: Screenshot_from_2024-04-23_01-05-40.png
Screenshot_from_2024-04-23_01-05-40.png
Attachment 9: Screenshot_from_2024-04-23_01-06-12.png
Screenshot_from_2024-04-23_01-06-12.png
Attachment 10: Screenshot_from_2024-04-23_01-06-40.png
Screenshot_from_2024-04-23_01-06-40.png
Attachment 11: Screenshot_from_2024-04-23_01-07-18.png
Screenshot_from_2024-04-23_01-07-18.png
Attachment 12: Screenshot_from_2024-04-23_03-06-28.png
Screenshot_from_2024-04-23_03-06-28.png
Attachment 13: Screenshot_from_2024-04-23_03-07-02.png
Screenshot_from_2024-04-23_03-07-02.png
Attachment 14: Screenshot_from_2024-04-23_03-07-35.png
Screenshot_from_2024-04-23_03-07-35.png
Attachment 15: Screenshot_from_2024-04-23_03-08-09.png
Screenshot_from_2024-04-23_03-08-09.png
Attachment 16: Screenshot_from_2024-04-23_03-08-43.png
Screenshot_from_2024-04-23_03-08-43.png
Attachment 17: Screenshot_from_2024-04-23_05-02-11.png
Screenshot_from_2024-04-23_05-02-11.png
Attachment 18: Screenshot_from_2024-04-23_05-03-10.png
Screenshot_from_2024-04-23_05-03-10.png
Attachment 19: Screenshot_from_2024-04-23_05-03-57.png
Screenshot_from_2024-04-23_05-03-57.png
Attachment 20: Screenshot_from_2024-04-23_05-04-37.png
Screenshot_from_2024-04-23_05-04-37.png
Attachment 21: Screenshot_from_2024-04-23_05-05-13.png
Screenshot_from_2024-04-23_05-05-13.png
Attachment 22: Screenshot_from_2024-04-23_07-29-15.png
Screenshot_from_2024-04-23_07-29-15.png
Attachment 23: Screenshot_from_2024-04-23_07-29-59.png
Screenshot_from_2024-04-23_07-29-59.png
Attachment 24: Screenshot_from_2024-04-23_07-30-27.png
Screenshot_from_2024-04-23_07-30-27.png
Attachment 25: Screenshot_from_2024-04-23_07-30-54.png
Screenshot_from_2024-04-23_07-30-54.png
Attachment 26: Screenshot_from_2024-04-23_07-31-16.png
Screenshot_from_2024-04-23_07-31-16.png
  588   Mon Apr 22 15:17:07 2024 DSJ16.00-0.00 22/04/24

First checks of shift. All looks ok.

Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.

Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)

 

checks at 16.00 - merger has crashed - aida01 dropped out  - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09 

Reset 01, restarted DAQ, and restarted as R16.

16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.

Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms

RUN17 STARTED 16.51|

 HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.

 

20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift

21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps

22.48 - things apear ok. (Attachments 23-26) 

Attachment 1: Screenshot_from_2024-04-22_16-16-30.png
Screenshot_from_2024-04-22_16-16-30.png
Attachment 2: Screenshot_from_2024-04-22_16-14-31.png
Screenshot_from_2024-04-22_16-14-31.png
Attachment 3: Screenshot_from_2024-04-22_16-09-16.png
Screenshot_from_2024-04-22_16-09-16.png
Attachment 4: Screenshot_from_2024-04-22_16-05-22.png
Screenshot_from_2024-04-22_16-05-22.png
Attachment 5: Screenshot_from_2024-04-22_16-33-34.png
Screenshot_from_2024-04-22_16-33-34.png
Attachment 6: Screenshot_from_2024-04-22_16-32-54.png
Screenshot_from_2024-04-22_16-32-54.png
Attachment 7: Screenshot_from_2024-04-22_16-32-04.png
Screenshot_from_2024-04-22_16-32-04.png
Attachment 8: Screenshot_from_2024-04-22_16-31-03.png
Screenshot_from_2024-04-22_16-31-03.png
Attachment 9: Screenshot_from_2024-04-22_16-30-00.png
Screenshot_from_2024-04-22_16-30-00.png
Attachment 10: Screenshot_from_2024-04-22_16-29-03.png
Screenshot_from_2024-04-22_16-29-03.png
Attachment 11: Screenshot_from_2024-04-22_16-27-01.png
Screenshot_from_2024-04-22_16-27-01.png
Attachment 12: Screenshot_from_2024-04-22_16-25-30.png
Screenshot_from_2024-04-22_16-25-30.png
Attachment 13: Screenshot_from_2024-04-22_19-00-04.png
Screenshot_from_2024-04-22_19-00-04.png
Attachment 14: Screenshot_from_2024-04-22_18-59-25.png
Screenshot_from_2024-04-22_18-59-25.png
Attachment 15: Screenshot_from_2024-04-22_18-58-21.png
Screenshot_from_2024-04-22_18-58-21.png
Attachment 16: Screenshot_from_2024-04-22_18-56-34.png
Screenshot_from_2024-04-22_18-56-34.png
Attachment 17: Screenshot_from_2024-04-22_20-46-50.png
Screenshot_from_2024-04-22_20-46-50.png
Attachment 18: Screenshot_from_2024-04-22_20-45-57.png
Screenshot_from_2024-04-22_20-45-57.png
Attachment 19: Screenshot_from_2024-04-22_20-45-10.png
Screenshot_from_2024-04-22_20-45-10.png
Attachment 20: Screenshot_from_2024-04-22_20-44-41.png
Screenshot_from_2024-04-22_20-44-41.png
Attachment 21: Screenshot_from_2024-04-22_22-00-30.png
Screenshot_from_2024-04-22_22-00-30.png
Attachment 22: Screenshot_from_2024-04-22_22-53-11.png
Screenshot_from_2024-04-22_22-53-11.png
Attachment 23: Screenshot_from_2024-04-22_22-52-19.png
Screenshot_from_2024-04-22_22-52-19.png
Attachment 24: Screenshot_from_2024-04-22_22-50-52.png
Screenshot_from_2024-04-22_22-50-52.png
Attachment 25: Screenshot_from_2024-04-22_22-49-07.png
Screenshot_from_2024-04-22_22-49-07.png
Attachment 26: Screenshot_from_2024-04-22_22-48-31.png
Screenshot_from_2024-04-22_22-48-31.png
  587   Mon Apr 22 11:43:55 2024 TD21-22.4.24 overnight MIDASsort online sort
Attachment 1 - per DSSSD LEC e_p versus e_n (20keV/channel)

Attachment 2 - per DSSSD HEC m_p versus m_n 

Attachment 3 - per DSSSD HEC x versus y (I really do need to sort out the mapping!)

Attachment 4 - per DSSSD HEC e_p versus e_n (20MeV/channel)

Attachment 5 - per DSSSD per pixel HEC-HEC time (4.096us/channel)

Attachment 6 - per DSSSD implant/decay event rates (262.144us/channel)
               showing beam spill cycle

Attachment 7 - colours/contour levels for all 2D spectra
Attachment 1: Screenshot_from_2024-04-22_07-58-12.png
Screenshot_from_2024-04-22_07-58-12.png
Attachment 2: Screenshot_from_2024-04-22_07-59-15.png
Screenshot_from_2024-04-22_07-59-15.png
Attachment 3: Screenshot_from_2024-04-22_08-00-37.png
Screenshot_from_2024-04-22_08-00-37.png
Attachment 4: Screenshot_from_2024-04-22_08-01-08.png
Screenshot_from_2024-04-22_08-01-08.png
Attachment 5: Screenshot_from_2024-04-22_08-02-03.png
Screenshot_from_2024-04-22_08-02-03.png
Attachment 6: Screenshot_from_2024-04-22_08-03-25.png
Screenshot_from_2024-04-22_08-03-25.png
Attachment 7: Screenshot_from_2024-04-22_12-48-18.png
Screenshot_from_2024-04-22_12-48-18.png
  586   Mon Apr 22 09:03:54 2024 TDReport: aida04 stops producing data
09.14 aida04 stopped producing data - system console output - attachment 1

It is not clear (to me at least) whether the cause is radiation-induced single event upsets or
the start/stop of LN2 fills.

Nor is it clear why the FEE64 stops
- WR timestamp error (data link to Merger blocks)
- AIDAExecV10 fails
- ?

rebooted aida04

DAQ RESET/SETUP etc cycle for all FEE64s

10.03 successful restart 
      current data file R15
Attachment 1: Screenshot_from_2024-04-22_09-49-30.png
Screenshot_from_2024-04-22_09-49-30.png
ELOG V3.1.4-unknown