AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 13 of 35  ELOG logo
  ID Date Author Subject
  457   Sun May 15 15:51:49 2022 MLSunday 15th May 16:00-0:00

Status at 16:45 (CET)

The expreiment continues to run smoothly.

AIDA Stats look ok - Attachment 1

AIDA Temperature ok - Attachment 2

AIDA Leakage current: ok - Attachment 3

System wide check:

Clock: 13 passed, 1 failed (aida09)

ADC Calibration: 9 passed, 5 fialed (aida2,6,9,10,13)

White rabbit decoder: 14 passed, 0 failed.

FPGA timestamp: 14 passed, 0 failed.

 

Status at 18:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 4

AIDA Temperature ok - Attachment 5

AIDA Leakage current: increase slightly - Attachment 6

System wide check: No changes, same as above

 

Status at 20:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 7

AIDA Temperature ok - Attachment 8

AIDA Leakage current: increase slightly - Attachment 9

System wide check: No changes, same as above

 

Status at 22:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 10

AIDA Temperature ok - Attachment 11

AIDA Leakage current: increase slightly - Attachment 12

System wide check: No changes, same as above

Attachment 1: Stats_Screenshot_from_2022-05-15_16-49-06.png
Stats_Screenshot_from_2022-05-15_16-49-06.png
Attachment 2: Temp_Screenshot_from_2022-05-15_16-48-31.png
Temp_Screenshot_from_2022-05-15_16-48-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-15_16-47-44.png
HV_LC_Screenshot_from_2022-05-15_16-47-44.png
Attachment 4: Stats_Screenshot_from_2022-05-15_18-43-16.png
Stats_Screenshot_from_2022-05-15_18-43-16.png
Attachment 5: Temp_Screenshot_from_2022-05-15_18-44-06.png
Temp_Screenshot_from_2022-05-15_18-44-06.png
Attachment 6: HV_LC_Screenshot_from_2022-05-15_18-42-23.png
HV_LC_Screenshot_from_2022-05-15_18-42-23.png
Attachment 7: Stats_Screenshot_from_2022-05-15_20-42-03.png
Stats_Screenshot_from_2022-05-15_20-42-03.png
Attachment 8: Temp_Screenshot_from_2022-05-15_20-42-42.png
Temp_Screenshot_from_2022-05-15_20-42-42.png
Attachment 9: HV_LC_Screenshot_from_2022-05-15_20-36-28.png
HV_LC_Screenshot_from_2022-05-15_20-36-28.png
Attachment 10: Stats_Screenshot_from_2022-05-15_22-43-37.png
Stats_Screenshot_from_2022-05-15_22-43-37.png
Attachment 11: Temp_Screenshot_from_2022-05-15_22-44-20.png
Temp_Screenshot_from_2022-05-15_22-44-20.png
Attachment 12: HV_LC_Screenshot_from_2022-05-15_22-42-36.png
HV_LC_Screenshot_from_2022-05-15_22-42-36.png
  456   Sun May 15 07:40:52 2022 OHSunday 15th May 08:00-16:00
08:40 FEE Temperatures ok - attachment 1
      FEE statistics - Attachment 2
      Bias and leakage currents ok - attachment 3
      ASIC check ok
      All system wide checks ok - Except aida09 fails clock check with bit 6

11:00 FEE statistics - attachment 4
      FEE temperatures all ok - attachment 5
      BIAS and leakage currents - attachment 6
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1030
      13.1 MB/s to disk
      4.8 million items per second
      
13:13 FEE statistics - attachment 7
      FEE Temperatures all ok - attahcment 8
      Bias and leakage currents ok - attachment 9
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1078
      13.1 MB/s to disk
      4.8 million items per second

15:35 Analysis of R4_1128 - attachment 10
Attachment 1: Screenshot_from_2022-05-15_08-38-04.png
Screenshot_from_2022-05-15_08-38-04.png
Attachment 2: Screenshot_from_2022-05-15_08-38-34.png
Screenshot_from_2022-05-15_08-38-34.png
Attachment 3: Screenshot_from_2022-05-15_08-39-12.png
Screenshot_from_2022-05-15_08-39-12.png
Attachment 4: Screenshot_from_2022-05-15_11-00-32.png
Screenshot_from_2022-05-15_11-00-32.png
Attachment 5: Screenshot_from_2022-05-15_11-00-50.png
Screenshot_from_2022-05-15_11-00-50.png
Attachment 6: Screenshot_from_2022-05-15_11-01-02.png
Screenshot_from_2022-05-15_11-01-02.png
Attachment 7: Screenshot_from_2022-05-15_13-13-50.png
Screenshot_from_2022-05-15_13-13-50.png
Attachment 8: Screenshot_from_2022-05-15_13-14-12.png
Screenshot_from_2022-05-15_13-14-12.png
Attachment 9: Screenshot_from_2022-05-15_13-14-31.png
Screenshot_from_2022-05-15_13-14-31.png
Attachment 10: R4_1128
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259195371 ( 1637875.8 Hz)
        Other data format:    2724629 (   17217.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2455 (      15.5 Hz)
                               RESUME:       2456 (      15.5 Hz)
                              SYNC100:      32590 (     205.9 Hz)
                              WR48-63:      32590 (     205.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2654538 (   16774.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      38026 (     240.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      158.251 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                4.735                0.000
  2                0.000                0.000
  3               16.674                0.000
  4                0.000                0.000
  5               77.034                0.000
  6                0.000                0.000
  7               33.677                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000               12.872
 11               17.247                0.000
 12                0.000                0.000
 13                0.000                8.200
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7464928       2802          0          0          0          0        927        927          0        948          0        661
  1   38695355      10162          0          0        254        254       4827       4827          0          0          0      32843
  2    6890959    1397581          0          0          0          0       1028       1028          0    1395525          0        633
  3   41303989    1261446          0          0        398        398       5227       5227          0    1250196          0        331
  4    4066993       1070          0          0          0          0        535        535          0          0          0        242
  5   37291979      10883          0          0        737        738       4704       4704          0          0          0        222
  6   12390385       7719          0          0          0          0       1626       1626          0       4467          0        394
  7   42901955      15088          0          0        626        626       5217       5217          0       3402          0        167
  8   10872099       2752          0          0          0          0       1376       1376          0          0          0        592
  9    4061050       1014          0          0          0          0        507        507          0          0          0        432
 10    2113468        548          0          0          0          0        274        274          0          0          0        291
 11   41561869      11172          0          0        440        440       5146       5146          0          0          0        468
 12    6798748       1722          0          0          0          0        861        861          0          0          0        436
 13    2781594        670          0          0          0          0        335        335          0          0          0        314
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.348s (  959.588 blocks/s,  59.974 Mb/s)
  455   Sat May 14 23:02:33 2022 TDSunday 15 May 00:00-08:00
Pb beam c. 1e+9/spill

00.02 zero all histograms
      system wide checks - baseline counters

00.08 ASIC check control - all FEE64s, all ASICs

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2 & 3 - ucesb - AIDA DSSSD #1 c. 10-20Hz peak rate on spill, c. 0Hz off spill

Attachment 4 - DSSSD bias & Leakage current - OK

Attachment 5 - FEE64 temps OK

Attachment 6-13 - adc, pause, resume & correlation scaler data items, push, flush, aida01, aida06

Attachments 14-18 - system wide checks - all OK *except* aida09 clock fail status 6

Attachments 19-25 - iptraf, TapeServer, NewMerger, NewMerger stats



00.44 file S450/R4_792

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

00.51 disk space OK - c. 3' per data file

[npg@aidas-gsi S450]$ ls -l /
total 40
lrwxrwxrwx.   1 root root     7 Oct 18  2021 bin -> usr/bin
dr-xr-xr-x.   5 root root  4096 May 12 18:31 boot
drwxr-xr-x.  23 root root  4100 May 12 18:31 dev
drwxr-xr-x. 145 root root  8192 May 12 18:31 etc
drwxr-xr-x.   4 root root    56 Jan 28 14:56 home
lrwxrwxrwx.   1 root root     7 Oct 18  2021 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 Oct 18  2021 lib64 -> usr/lib64
drwxr-xr-x.   5 root root   113 Apr 28 11:58 media
lrwxrwxrwx.   1 root root    45 Oct 18  2021 MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119
drwxr-xr-x.   2 root root     6 Apr 11  2018 mnt
drwxr-xr-x.   3 root root    16 Oct 18  2021 opt
dr-xr-xr-x. 464 root root     0 May 12 18:30 proc
dr-xr-x---.  10 root root  4096 May 13 11:47 root
drwxr-xr-x.  44 root root  1360 May 14 06:34 run
lrwxrwxrwx.   1 root root     8 Oct 18  2021 sbin -> usr/sbin
drwxr-xr-x.   2 root root     6 Apr 11  2018 srv
dr-xr-xr-x.  13 root root     0 May 12 18:31 sys
lrwxrwxrwx.   1 root root    28 Apr 28 14:19 TapeData -> /media/SecondDrive/TapeData/
drwxrwxrwt.  40 root root 12288 May 15 00:49 tmp
drwxr-xr-x.  13 root root   155 Oct 18  2021 usr
drwxr-xr-x.  21 root root  4096 Oct 18  2021 var
[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  373M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  3.1T  3.8T  46% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

Attachment 26 - analysis data file R4_792

Attachments 27-28 - per FEE64 rate & stat spectra - shows distro HEC events

Attachments 29-30 - per FEE64 1.8.L spectra 
 pulser peak widths aida01 134 ch FWHM, aida04 393 ch FWHM

Attachments 31-32 - per FEE64 1.8.H spectra

Attachments 33-34 - aida02 & aida04 1.*.H spectra

04.13

Attachment 35 - DSSSD bias & Leakage current - OK

Attachment 36 - FEE64 temps OK

Attachment 37 - adc data item stats

Attachments 38 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6


07.05

Attachment 39 - DSSSD bias & Leakage current - OK

Attachment 40 - FEE64 temps OK

Attachment 41 - adc data item stats

Attachments 42 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

Attachments 43-48 - aida02 & aida04 & aida06 & aida08 & 1.*.H spectra
Attachment 1: Screenshot_from_2022-05-15_00-03-33.png
Screenshot_from_2022-05-15_00-03-33.png
Attachment 2: Screenshot_from_2022-05-15_00-03-58.png
Screenshot_from_2022-05-15_00-03-58.png
Attachment 3: Screenshot_from_2022-05-15_00-04-18.png
Screenshot_from_2022-05-15_00-04-18.png
Attachment 4: Screenshot_from_2022-05-15_00-04-39.png
Screenshot_from_2022-05-15_00-04-39.png
Attachment 5: Screenshot_from_2022-05-15_00-05-10.png
Screenshot_from_2022-05-15_00-05-10.png
Attachment 6: Screenshot_from_2022-05-15_00-07-25.png
Screenshot_from_2022-05-15_00-07-25.png
Attachment 7: Screenshot_from_2022-05-15_00-16-59.png
Screenshot_from_2022-05-15_00-16-59.png
Attachment 8: Screenshot_from_2022-05-15_00-17-52.png
Screenshot_from_2022-05-15_00-17-52.png
Attachment 9: Screenshot_from_2022-05-15_00-18-55.png
Screenshot_from_2022-05-15_00-18-55.png
Attachment 10: Screenshot_from_2022-05-15_00-19-35.png
Screenshot_from_2022-05-15_00-19-35.png
Attachment 11: Screenshot_from_2022-05-15_00-20-14.png
Screenshot_from_2022-05-15_00-20-14.png
Attachment 12: Screenshot_from_2022-05-15_00-20-57.png
Screenshot_from_2022-05-15_00-20-57.png
Attachment 13: Screenshot_from_2022-05-15_00-21-32.png
Screenshot_from_2022-05-15_00-21-32.png
Attachment 14: Screenshot_from_2022-05-15_00-26-13.png
Screenshot_from_2022-05-15_00-26-13.png
Attachment 15: Screenshot_from_2022-05-15_00-26-49.png
Screenshot_from_2022-05-15_00-26-49.png
Attachment 16: Screenshot_from_2022-05-15_00-27-19.png
Screenshot_from_2022-05-15_00-27-19.png
Attachment 17: Screenshot_from_2022-05-15_00-28-00.png
Screenshot_from_2022-05-15_00-28-00.png
Attachment 18: Screenshot_from_2022-05-15_00-28-45.png
Screenshot_from_2022-05-15_00-28-45.png
Attachment 19: Screenshot_from_2022-05-15_00-33-46.png
Screenshot_from_2022-05-15_00-33-46.png
Attachment 20: Screenshot_from_2022-05-15_00-35-33.png
Screenshot_from_2022-05-15_00-35-33.png
Attachment 21: Screenshot_from_2022-05-15_00-36-00.png
Screenshot_from_2022-05-15_00-36-00.png
Attachment 22: Screenshot_from_2022-05-15_00-36-26.png
Screenshot_from_2022-05-15_00-36-26.png
Attachment 23: Screenshot_from_2022-05-15_00-36-58.png
Screenshot_from_2022-05-15_00-36-58.png
Attachment 24: Screenshot_from_2022-05-15_00-37-25.png
Screenshot_from_2022-05-15_00-37-25.png
Attachment 25: Screenshot_from_2022-05-15_00-37-46.png
Screenshot_from_2022-05-15_00-37-46.png
Attachment 26: R4_792
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  250102849 ( 1597022.2 Hz)
        Other data format:   11817151 (   75458.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2692 (      17.2 Hz)
                               RESUME:       2692 (      17.2 Hz)
                              SYNC100:      32584 (     208.1 Hz)
                              WR48-63:      32584 (     208.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   11746599 (   75007.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      48170 (     307.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      156.606 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                6.802                0.000
  2                0.000                0.000
  3               28.710                0.000
  4                0.000                0.000
  5               76.548                0.000
  6                0.000                0.000
  7               36.670                0.000
  8                0.002                0.000
  9                0.000                0.000
 10                0.000               14.137
 11               20.418                0.000
 12                0.000                0.000
 13                0.000                3.327
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7219269       2805          0          0          0          0        933        933          0        939          0        589
  1   37816150      10006          0          0        269        269       4734       4734          0          0          0      43094
  2    6973157    6465448          0          0          0          0       1676       1676          0    6462096          0        570
  3   37723722    5287878          0          0        579        580       5326       5326          0    5276067          0        352
  4    3982759       1044          0          0          0          0        522        522          0          0          0        233
  5   36626106      10369          0          0        747        746       4438       4438          0          0          0        211
  6   11915365       7321          0          0          0          0       1544       1544          0       4233          0        403
  7   42135728      15066          0          0        653        653       5248       5248          0       3264          0        174
  8   10616770       2532          0          0          2          2       1264       1264          0          0          0        570
  9    3975235        966          0          0          0          0        483        483          0          0          0        497
 10    2072466        538          0          0          0          0        269        269          0          0          0        289
 11   40106886      10836          0          0        442        442       4976       4976          0          0          0        472
 12    6251481       1636          0          0          0          0        818        818          0          0          0        413
 13    2687755        706          0          0          0          0        353        353          0          0          0        303
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.205s (  883.854 blocks/s,  55.241 Mb/s)
Attachment 27: Screenshot_from_2022-05-15_00-56-27.png
Screenshot_from_2022-05-15_00-56-27.png
Attachment 28: Screenshot_from_2022-05-15_00-57-44.png
Screenshot_from_2022-05-15_00-57-44.png
Attachment 29: Screenshot_from_2022-05-15_00-59-12.png
Screenshot_from_2022-05-15_00-59-12.png
Attachment 30: Screenshot_from_2022-05-15_01-02-36.png
Screenshot_from_2022-05-15_01-02-36.png
Attachment 31: Screenshot_from_2022-05-15_01-04-44.png
Screenshot_from_2022-05-15_01-04-44.png
Attachment 32: Screenshot_from_2022-05-15_01-05-49.png
Screenshot_from_2022-05-15_01-05-49.png
Attachment 33: Screenshot_from_2022-05-15_01-13-22.png
Screenshot_from_2022-05-15_01-13-22.png
Attachment 34: Screenshot_from_2022-05-15_01-15-01.png
Screenshot_from_2022-05-15_01-15-01.png
Attachment 35: Screenshot_from_2022-05-15_04-13-03.png
Screenshot_from_2022-05-15_04-13-03.png
Attachment 36: Screenshot_from_2022-05-15_04-13-31.png
Screenshot_from_2022-05-15_04-13-31.png
Attachment 37: Screenshot_from_2022-05-15_04-13-52.png
Screenshot_from_2022-05-15_04-13-52.png
Attachment 38: Screenshot_from_2022-05-15_04-14-31.png
Screenshot_from_2022-05-15_04-14-31.png
Attachment 39: Screenshot_from_2022-05-15_07-05-19.png
Screenshot_from_2022-05-15_07-05-19.png
Attachment 40: Screenshot_from_2022-05-15_07-05-50.png
Screenshot_from_2022-05-15_07-05-50.png
Attachment 41: Screenshot_from_2022-05-15_07-06-15.png
Screenshot_from_2022-05-15_07-06-15.png
Attachment 42: Screenshot_from_2022-05-15_07-06-42.png
Screenshot_from_2022-05-15_07-06-42.png
Attachment 43: Screenshot_from_2022-05-15_07-06-42.png
Screenshot_from_2022-05-15_07-06-42.png
Attachment 44: Screenshot_from_2022-05-15_07-12-37.png
Screenshot_from_2022-05-15_07-12-37.png
Attachment 45: Screenshot_from_2022-05-15_07-14-49.png
Screenshot_from_2022-05-15_07-14-49.png
Attachment 46: Screenshot_from_2022-05-15_07-16-18.png
Screenshot_from_2022-05-15_07-16-18.png
Attachment 47: Screenshot_from_2022-05-15_07-20-47.png
Screenshot_from_2022-05-15_07-20-47.png
Attachment 48: Screenshot_from_2022-05-15_07-22-02.png
Screenshot_from_2022-05-15_07-22-02.png
  454   Sat May 14 21:34:11 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Attachment 1: Rates_2022-05-14_22-32-44.png
Rates_2022-05-14_22-32-44.png
Attachment 2: Temp_2022-05-14_22-31-37.png
Temp_2022-05-14_22-31-37.png
Attachment 3: Leakg_2022-05-14_22-30-28.png
Leakg_2022-05-14_22-30-28.png
  453   Sat May 14 19:39:13 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Attachment 1: Stat_2022-05-14_20-36-15.png
Stat_2022-05-14_20-36-15.png
Attachment 2: Temp_2022-05-14_20-35-19.png
Temp_2022-05-14_20-35-19.png
Attachment 3: Leakage_2022-05-14_20-33-32.png
Leakage_2022-05-14_20-33-32.png
  452   Sat May 14 17:34:40 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: Stat_2022-05-14_18-33-16.png
Stat_2022-05-14_18-33-16.png
Attachment 2: Temp_2022-05-14_18-32-07.png
Temp_2022-05-14_18-32-07.png
Attachment 3: Leakage_2022-05-14_18-30-52.png
Leakage_2022-05-14_18-30-52.png
  451   Sat May 14 15:35:24 2022 BA, AASaturday 14 May
Attachment 1: Stais_from_2022-05-14_16-30-53.png
Stais_from_2022-05-14_16-30-53.png
Attachment 2: Temp_2022-05-14_16-32-42.png
Temp_2022-05-14_16-32-42.png
Attachment 3: Leakage_2022-05-14_16-33-45.png
Leakage_2022-05-14_16-33-45.png
  450   Sat May 14 13:01:58 2022 MS, OHSaturday 14 May

14:00

Attachment 1: Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_14-03-54.png
Screenshot_from_2022-05-14_14-03-54.png
  449   Sat May 14 10:59:52 2022 MS, OHSaturday 14 May

12:00

Attachment 1: Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_12-01-54.png
Screenshot_from_2022-05-14_12-01-54.png
  448   Sat May 14 09:00:21 2022 MS, OHSaturday 14 May

10:00

Attachment 1: Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_10-02-51.png
Screenshot_from_2022-05-14_10-02-51.png
  447   Sat May 14 06:57:21 2022 MS, OHSaturday 14 May
08:00 Took over the shift from Philippos
      In system wide checks aida09 fails ASIC clock check but it is bit 6 which is ok
  446   Sat May 14 03:14:00 2022 PPShift Checks

4:13

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

 

Screenshot 1: statistics

Screenshot 2: temperatures

Screenshot 3: scalers

Screenshot 4: Bias and leakage current

 

6:36

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR


Screenshot 5: statistics

Screenshot 6: temperatures

Screenshot 7: scalers

Screenshot 8: Bias and leakage current

 

Attachment 1: stats-220514-0428.png
stats-220514-0428.png
Attachment 2: temps-220514-0417.png
temps-220514-0417.png
Attachment 3: scalers-220514-0426.png
scalers-220514-0426.png
Attachment 4: I-V-220514-0431.png
I-V-220514-0431.png
Attachment 5: stats-220514-0639.png
stats-220514-0639.png
Attachment 6: temps-220514-0639.png
temps-220514-0639.png
Attachment 7: scalers-220514-0641.png
scalers-220514-0641.png
Attachment 8: I-V-220514-0640.png
I-V-220514-0640.png
  445   Sat May 14 01:52:03 2022 ML2h-Shift Checks

AIDA stats ok

Leakage current ok

Temperatires ok

grafana ok

ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.

System wide check done and same results as earlier:

aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_02-48-14.png
Stats_Screenshot_from_2022-05-14_02-48-14.png
Attachment 2: Temp_Screenshot_from_2022-05-14_02-47-31.png
Temp_Screenshot_from_2022-05-14_02-47-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-14_02-46-42.png
HV_LC_Screenshot_from_2022-05-14_02-46-42.png
Attachment 4: ucesb_rates_Screenshot_from_2022-05-14_02-50-36.png
ucesb_rates_Screenshot_from_2022-05-14_02-50-36.png
Attachment 5: Grafana_Screenshot_from_2022-05-14_03-50-00.png
Grafana_Screenshot_from_2022-05-14_03-50-00.png
  444   Fri May 13 23:37:59 2022 ML2h-Shift Checks
Took over remotely from CB  about an hour ago

At 0h40 (CET):
      Stats still ok - attach 1
      Leakage still OK - Attach 2
      Temps - attach 3
      Grafana OK
      UCESB rates ok - attach 5
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_00-40-56.png
Stats_Screenshot_from_2022-05-14_00-40-56.png
Attachment 2: HV_LC_Screenshot_from_2022-05-14_00-39-51.png
HV_LC_Screenshot_from_2022-05-14_00-39-51.png
Attachment 3: temp_Screenshot_from_2022-05-14_00-41-44.png
temp_Screenshot_from_2022-05-14_00-41-44.png
Attachment 4: UCESB_Screenshot_from_2022-05-14_01-00-00.png
UCESB_Screenshot_from_2022-05-14_01-00-00.png
  443   Fri May 13 15:00:23 2022 CB13 March - 16:00-24:00 shift
16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before
Attachment 1: 1305_1600_ucesb.png
1305_1600_ucesb.png
Attachment 2: 1305_1600_StatsTemps.png
1305_1600_StatsTemps.png
Attachment 3: 1305_1600_Grafana.png
1305_1600_Grafana.png
Attachment 4: 1305_1800_StatsLeakage.png
1305_1800_StatsLeakage.png
Attachment 5: 1305_2030_Stats.png
1305_2030_Stats.png
Attachment 6: 1305_2030_ucesb.png
1305_2030_ucesb.png
Attachment 7: 1305_2030_Grafana.png
1305_2030_Grafana.png
Attachment 8: 1305_2300_StatsLeakage.png
1305_2300_StatsLeakage.png
Attachment 9: 1305_2300_ucesb.png
1305_2300_ucesb.png
Attachment 10: 1305_2300_Grafana.png
1305_2300_Grafana.png
  442   Fri May 13 07:36:18 2022 OH, TDFriday 13 May
08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)
Attachment 1: 220513_0839_R1_327_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259232212 ( 1719602.4 Hz)
        Other data format:    2687788 (   17829.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        986 (       6.5 Hz)
                               RESUME:        986 (       6.5 Hz)
                              SYNC100:      32562 (     216.0 Hz)
                              WR48-63:      32562 (     216.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2620692 (   17384.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     130683 (     866.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      150.751 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1                9.978                0.000
  2                0.059                0.000
  3               21.640                0.000
  4                0.158                0.000
  5               66.897                0.000
  6                1.778                0.000
  7               36.972                0.000
  8                0.172                0.000
  9                0.018                0.000
 10                0.000                7.315
 11               28.135                0.000
 12                0.007                0.000
 13                0.000                3.034
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7933827       2887          0          0          5          5        987        987          0        903          0       1141
  1   37526918       9508          0          0         50         50       4704       4704          0          0          0     122502
  2    8368319    1400718          0          0          2          2       1222       1222          0    1398270          0        647
  3   39185673    1207805          0          0        156        156       5116       5116          0    1197261          0        703
  4    4609566       1178          0          0          5          5        584        584          0          0          0       1170
  5   40049357      10540          0          0        309        309       4961       4961          0          0          0        297
  6   13162553      17502          0          0         13         13       1685       1685          0      14106          0        302
  7   41199857      20850          0          0        248        248       5101       5101          0      10152          0        241
  8   12021437       2864          0          0          5          5       1427       1427          0          0          0        467
  9    4748197       1188          0          0          2          2        592        592          0          0          0       1165
 10    3089949        744          0          0          0          0        372        372          0          0          0        247
 11   38258323       9856          0          0        190        190       4738       4738          0          0          0       1052
 12    6307578       1518          0          0          1          1        758        758          0          0          0        311
 13    2770658        630          0          0          0          0        315        315          0          0          0        438
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.600s (  952.392 blocks/s,  59.525 Mb/s)
Attachment 2: Screenshot_from_2022-05-13_14-28-00.png
Screenshot_from_2022-05-13_14-28-00.png
Attachment 3: Screenshot_from_2022-05-13_14-27-51.png
Screenshot_from_2022-05-13_14-27-51.png
Attachment 4: Screenshot_from_2022-05-13_14-27-31.png
Screenshot_from_2022-05-13_14-27-31.png
Attachment 5: Screenshot_from_2022-05-13_14-27-17.png
Screenshot_from_2022-05-13_14-27-17.png
Attachment 6: Screenshot_from_2022-05-13_14-26-32.png
Screenshot_from_2022-05-13_14-26-32.png
Attachment 7: Screenshot_from_2022-05-13_14-26-26.png
Screenshot_from_2022-05-13_14-26-26.png
Attachment 8: Screenshot_from_2022-05-13_14-26-21.png
Screenshot_from_2022-05-13_14-26-21.png
Attachment 9: Screenshot_from_2022-05-13_14-26-08.png
Screenshot_from_2022-05-13_14-26-08.png
Attachment 10: Screenshot_from_2022-05-13_14-25-46.png
Screenshot_from_2022-05-13_14-25-46.png
Attachment 11: Screenshot_from_2022-05-13_14-25-38.png
Screenshot_from_2022-05-13_14-25-38.png
Attachment 12: Screenshot_from_2022-05-13_14-25-22.png
Screenshot_from_2022-05-13_14-25-22.png
Attachment 13: Screenshot_from_2022-05-13_14-24-07.png
Screenshot_from_2022-05-13_14-24-07.png
Attachment 14: Screenshot_from_2022-05-13_14-22-44.png
Screenshot_from_2022-05-13_14-22-44.png
Attachment 15: Screenshot_from_2022-05-13_14-21-16.png
Screenshot_from_2022-05-13_14-21-16.png
Attachment 16: Screenshot_from_2022-05-13_14-21-04.png
Screenshot_from_2022-05-13_14-21-04.png
Attachment 17: Screenshot_from_2022-05-13_14-20-34.png
Screenshot_from_2022-05-13_14-20-34.png
Attachment 18: Screenshot_from_2022-05-13_14-20-14.png
Screenshot_from_2022-05-13_14-20-14.png
Attachment 19: Screenshot_from_2022-05-13_14-19-10.png
Screenshot_from_2022-05-13_14-19-10.png
Attachment 20: Screenshot_from_2022-05-13_14-18-40.png
Screenshot_from_2022-05-13_14-18-40.png
Attachment 21: Screenshot_from_2022-05-13_14-18-31.png
Screenshot_from_2022-05-13_14-18-31.png
Attachment 22: Screenshot_from_2022-05-13_14-18-14.png
Screenshot_from_2022-05-13_14-18-14.png
Attachment 23: Screenshot_from_2022-05-13_14-17-38.png
Screenshot_from_2022-05-13_14-17-38.png
Attachment 24: Screenshot_from_2022-05-13_14-17-23.png
Screenshot_from_2022-05-13_14-17-23.png
Attachment 25: Screenshot_from_2022-05-13_14-17-09.png
Screenshot_from_2022-05-13_14-17-09.png
Attachment 26: Screenshot_from_2022-05-13_14-16-56.png
Screenshot_from_2022-05-13_14-16-56.png
Attachment 27: Screenshot_from_2022-05-13_14-16-30.png
Screenshot_from_2022-05-13_14-16-30.png
Attachment 28: Screenshot_from_2022-05-13_14-16-11.png
Screenshot_from_2022-05-13_14-16-11.png
Attachment 29: R4_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259102731 ( 1572653.5 Hz)
        Other data format:    2817271 (   17099.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2412 (      14.6 Hz)
                               RESUME:       2414 (      14.7 Hz)
                              SYNC100:      32614 (     198.0 Hz)
                              WR48-63:      32614 (     198.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2747217 (   16674.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      30594 (     185.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      164.755 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                5.297                0.000
  2                0.000                0.000
  3               18.287                0.000
  4                0.000                0.000
  5               77.894                0.000
  6                0.000                0.000
  7               25.402                0.000
  8                0.001                0.000
  9                0.000                0.000
 10                0.000                2.705
 11               15.617                0.000
 12                0.000                0.000
 13                0.000               29.550
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6844719       2775          0          0          0          0        894        894          0        987          0        375
  1   39920437      10378          0          0        277        277       4912       4912          0          0          0      27546
  2    7063505    1453891          0          0          0          0       1085       1085          0    1451721          0        328
  3   42633132    1301928          0          0        410        410       5590       5590          0    1289928          0        251
  4    3625899        900          0          0          0          0        450        450          0          0          0        125
  5   38867835      10853          0          0        768        769       4658       4658          0          0          0        129
  6    7590830       4334          0          0          0          0        934        934          0       2466          0        227
  7   44498336      14120          0          0        564        565       5438       5438          0       2115          0        113
  8   10949373       2716          0          0          1          1       1357       1357          0          0          0        345
  9    3887105        996          0          0          0          0        498        498          0          0          0        273
 10    1999345        532          0          0          0          0        266        266          0          0          0        160
 11   43113077      11754          0          0        392        392       5485       5485          0          0          0        288
 12    6030564       1596          0          0          0          0        798        798          0          0          0        252
 13    2078574        498          0          0          0          0        249        249          0          0          0        182
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
Attachment 30: Screenshot_from_2022-05-13_15-11-36.png
Screenshot_from_2022-05-13_15-11-36.png
Attachment 31: Screenshot_from_2022-05-13_15-11-29.png
Screenshot_from_2022-05-13_15-11-29.png
Attachment 32: Screenshot_from_2022-05-13_15-11-17.png
Screenshot_from_2022-05-13_15-11-17.png
Attachment 33: Screenshot_from_2022-05-13_15-10-55.png
Screenshot_from_2022-05-13_15-10-55.png
Attachment 34: Screenshot_from_2022-05-13_15-10-41.png
Screenshot_from_2022-05-13_15-10-41.png
Attachment 35: Screenshot_from_2022-05-13_15-10-30.png
Screenshot_from_2022-05-13_15-10-30.png
  441   Fri May 13 06:56:14 2022 OH, BA, MALow Rates

 

Quote:

The rate is very low in all aida, not sure .

This was because the statistics page had been changed to transfer buffers. The rates were ok

  440   Fri May 13 06:47:33 2022 ML2h-Shift Checks

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

Attachment 1: Stats_Screenshot_from_2022-05-13_07-43-54.png
Stats_Screenshot_from_2022-05-13_07-43-54.png
Attachment 2: Temp_Screenshot_from_2022-05-13_07-43-13.png
Temp_Screenshot_from_2022-05-13_07-43-13.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_07-42-38.png
HV_LC_Screenshot_from_2022-05-13_07-42-38.png
  439   Fri May 13 04:45:23 2022 ML2h-Shift Checks

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

Attachment 1: Stats_Screenshot_from_2022-05-13_05-43-14.png
Stats_Screenshot_from_2022-05-13_05-43-14.png
Attachment 2: Temp_Screenshot_from_2022-05-13_05-42-10.png
Temp_Screenshot_from_2022-05-13_05-42-10.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_05-41-23.png
HV_LC_Screenshot_from_2022-05-13_05-41-23.png
  438   Fri May 13 02:56:36 2022 ML2h-Shift Checks

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

Attachment 1: Stats_Screenshot_from_2022-05-13_03-43-34.png
Stats_Screenshot_from_2022-05-13_03-43-34.png
Attachment 2: Temp_Screenshot_from_2022-05-13_03-44-41.png
Temp_Screenshot_from_2022-05-13_03-44-41.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_03-41-28.png
HV_LC_Screenshot_from_2022-05-13_03-41-28.png
ELOG V3.1.4-unknown