AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40 MONNET
  DESPEC, Page 29 of 37  ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
ID Datedown Author Subject
  180   Mon Mar 8 09:45:20 2021 CA, TDAnalysis R7_20 (new version of NewMerger with min info code 4 & 5 data items)
An analysis of file R6_70 can be found at https://elog.ph.ed.ac.uk/DESPEC/177 attachment 19
We observe c. 80M ADC data items per 2Gb file and c. 160M info code 4 & 5 data items.

This am NH switched NewMerger for a new version which minimises the number of info code 4 & 5
data items. The analysis of one of these data files (R7_20) is appended - attachment 1. We
observe c. 2M ADC data items per 2Gb file. The number of info code 4 data items is significantly
reduced as expected but info code 5 data items are not observed.

Info code 4 & 5 data items will increment every (2^28 * 1e-9) 0.268s and (2^48 * 1e-9) 3.26d.

If we examine the verbose output (attachment 2) we observe that each data block only contains
c. 60 ADC data items - a small fraction of the capacity of each data block.  
Attachment 1: R7_20
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:    1822115 ( 2111176.3 Hz)
        Other data format:       1885 (    2184.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          8 (       9.3 Hz)
                               RESUME:         10 (      11.6 Hz)
                              SYNC100:          4 (       4.6 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:       1857 (    2151.6 Hz)
                             MBS info:          6 (       7.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:        0.863 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.206                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.060                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.201                0.000
  9                0.166                0.805
 10                0.000                0.000
 11                0.592                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     150071         10          0          0          2          2          0          0          0          6          0          0
  1     194013          1          0          0          0          0          1          0          0          0          0          0
  2     150373       1859          0          0          1          1          0          0       1857          0          0          0
  3     230996          0          0          0          0          0          0          0          0          0          0          0
  4      53452          0          0          0          0          0          0          0          0          0          0          0
  5     142131          5          0          0          2          3          0          0          0          0          0          0
  6     178856          0          0          0          0          0          0          0          0          0          0          0
  7      74144          0          0          0          0          0          0          0          0          0          0          0
  8     183174          3          0          0          1          2          0          0          0          0          0          0
  9     218137          5          0          0          1          1          3          0          0          0          0          0
 10      83482          0          0          0          0          0          0          0          0          0          0          0
 11     163286          2          0          0          1          1          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.008s ( 1390.832 blocks/s,  86.927 Mb/s)
Attachment 2: R7_20_verbose
[npg@aidas-gsi analyser]$ ./analyser v  /TapeData/S452/R7_20 | more
 *** TDR format 3.3.0 analyser - TD - January 2019
  verbose
 ***            ADC data: block:     1 ptr:     7 data: 0xC1177F65 module:  4 fail: 0 range: 0 id:   279 channel: 23 adc: 32613  ts: 0x00000000078DEB0E
 ***            ADC data: block:     1 ptr:     9 data: 0xC0478143 module:  1 fail: 0 range: 0 id:    71 channel:  7 adc: 33091  ts: 0x00000000078DEB4A
 ***            ADC data: block:     1 ptr:    11 data: 0xC0D182FC module:  3 fail: 0 range: 0 id:   209 channel: 17 adc: 33532  ts: 0x00000000078DEB68
 ***            ADC data: block:     1 ptr:    13 data: 0xC2C081B9 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 33209  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    15 data: 0xC2D7820D module: 11 fail: 0 range: 0 id:   727 channel: 23 adc: 33293  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    17 data: 0xC2E4835E module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 33630  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    19 data: 0xC2F681F4 module: 11 fail: 0 range: 0 id:   758 channel: 54 adc: 33268  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    21 data: 0xC00B7FDF module:  0 fail: 0 range: 0 id:    11 channel: 11 adc: 32735  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    23 data: 0xC0147DDE module:  0 fail: 0 range: 0 id:    20 channel: 20 adc: 32222  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    25 data: 0xC034803A module:  0 fail: 0 range: 0 id:    52 channel: 52 adc: 32826  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    27 data: 0xC272814E module:  9 fail: 0 range: 0 id:   626 channel: 50 adc: 33102  ts: 0x00000000078DF2A2
 ***            ADC data: block:     1 ptr:    29 data: 0xC11B7E13 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32275  ts: 0x00000000078DF2DE
 ***            ADC data: block:     1 ptr:    31 data: 0xC1327D8E module:  4 fail: 0 range: 0 id:   306 channel: 50 adc: 32142  ts: 0x00000000078DF2DE
 ***            ADC data: block:     1 ptr:    33 data: 0xC05F7FB9 module:  1 fail: 0 range: 0 id:    95 channel: 31 adc: 32697  ts: 0x00000000078DF31A
 ***            ADC data: block:     1 ptr:    35 data: 0xC0C28087 module:  3 fail: 0 range: 0 id:   194 channel:  2 adc: 32903  ts: 0x00000000078DF338
 ***            ADC data: block:     1 ptr:    37 data: 0xC0DB8075 module:  3 fail: 0 range: 0 id:   219 channel: 27 adc: 32885  ts: 0x00000000078DF338
 ***            ADC data: block:     1 ptr:    39 data: 0xC1BA7ECE module:  6 fail: 0 range: 0 id:   442 channel: 58 adc: 32462  ts: 0x00000000078DF644
 ***            ADC data: block:     1 ptr:    41 data: 0xC2C381FB module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 33275  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    43 data: 0xC2D981FD module: 11 fail: 0 range: 0 id:   729 channel: 25 adc: 33277  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    45 data: 0xC2E682A5 module: 11 fail: 0 range: 0 id:   742 channel: 38 adc: 33445  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    47 data: 0xC2F782E0 module: 11 fail: 0 range: 0 id:   759 channel: 55 adc: 33504  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    49 data: 0xC0177CE4 module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 31972  ts: 0x00000000078DF982
 ***            ADC data: block:     1 ptr:    51 data: 0xC27A7F8E module:  9 fail: 0 range: 0 id:   634 channel: 58 adc: 32654  ts: 0x00000000078DFA72
 ***            ADC data: block:     1 ptr:    53 data: 0xC0C3816A module:  3 fail: 0 range: 0 id:   195 channel:  3 adc: 33130  ts: 0x00000000078DFB08
 ***            ADC data: block:     1 ptr:    55 data: 0xC0FD81FD module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 33277  ts: 0x00000000078DFB08
 ***            ADC data: block:     1 ptr:    57 data: 0xC0847F9A module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 32666  ts: 0x00000000078DFCB6
 ***            ADC data: block:     1 ptr:    59 data: 0xC0B77EED module:  2 fail: 0 range: 0 id:   183 channel: 55 adc: 32493  ts: 0x00000000078DFCB6
 ***            ADC data: block:     1 ptr:    61 data: 0xC1AD8002 module:  6 fail: 0 range: 0 id:   429 channel: 45 adc: 32770  ts: 0x00000000078DFE14
 ***            ADC data: block:     1 ptr:    63 data: 0xC2C4837A module: 11 fail: 0 range: 0 id:   708 channel:  4 adc: 33658  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    65 data: 0xC2DC8250 module: 11 fail: 0 range: 0 id:   732 channel: 28 adc: 33360  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    67 data: 0xC2EA8214 module: 11 fail: 0 range: 0 id:   746 channel: 42 adc: 33300  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    69 data: 0xC27B812C module:  9 fail: 0 range: 0 id:   635 channel: 59 adc: 33068  ts: 0x00000000078E0242
 ***            ADC data: block:     1 ptr:    71 data: 0xC0C5811F module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 33055  ts: 0x00000000078E02D8
 ***            ADC data: block:     1 ptr:    73 data: 0xC0877F4D module:  2 fail: 0 range: 0 id:   135 channel:  7 adc: 32589  ts: 0x00000000078E0486
 ***            ADC data: block:     1 ptr:    75 data: 0xC2C782D6 module: 11 fail: 0 range: 0 id:   711 channel:  7 adc: 33494  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    77 data: 0xC2DE8261 module: 11 fail: 0 range: 0 id:   734 channel: 30 adc: 33377  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    79 data: 0xC2EB8115 module: 11 fail: 0 range: 0 id:   747 channel: 43 adc: 33045  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    81 data: 0xC0027C33 module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 31795  ts: 0x00000000078E0922
 ***            ADC data: block:     1 ptr:    83 data: 0xC0C68218 module:  3 fail: 0 range: 0 id:   198 channel:  6 adc: 33304  ts: 0x00000000078E0AA8
 ***            ADC data: block:     1 ptr:    85 data: 0xC0D58287 module:  3 fail: 0 range: 0 id:   213 channel: 21 adc: 33415  ts: 0x00000000078E0AA8
 ***            ADC data: block:     1 ptr:    87 data: 0xC08C4EFE module:  2 fail: 0 range: 0 id:   140 channel: 12 adc: 20222  ts: 0x00000000078E0C56
 ***            ADC data: block:     1 ptr:    89 data: 0xC2F48203 module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 33283  ts: 0x00000000078E0FA8
 ***            ADC data: block:     1 ptr:    91 data: 0xC267809D module:  9 fail: 0 range: 0 id:   615 channel: 39 adc: 32925  ts: 0x00000000078E11E2
 ***            ADC data: block:     1 ptr:    93 data: 0xC044810C module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 33036  ts: 0x00000000078E125A
 ***            ADC data: block:     1 ptr:    95 data: 0xC0C9815B module:  3 fail: 0 range: 0 id:   201 channel:  9 adc: 33115  ts: 0x00000000078E1278
 ***            ADC data: block:     1 ptr:    97 data: 0xC0DA8174 module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 33140  ts: 0x00000000078E1278
 ***            ADC data: block:     1 ptr:    99 data: 0xC1827F8D module:  6 fail: 0 range: 0 id:   386 channel:  2 adc: 32653  ts: 0x00000000078E1584
 ***            ADC data: block:     1 ptr:   101 data: 0xC1A77E56 module:  6 fail: 0 range: 0 id:   423 channel: 39 adc: 32342  ts: 0x00000000078E1584
 ***            ADC data: block:     1 ptr:   103 data: 0xC2FD846E module: 11 fail: 0 range: 0 id:   765 channel: 61 adc: 33902  ts: 0x00000000078E1778
 ***            ADC data: block:     1 ptr:   105 data: 0xC27081A7 module:  9 fail: 0 range: 0 id:   624 channel: 48 adc: 33191  ts: 0x00000000078E19B2
 ***            ADC data: block:     1 ptr:   107 data: 0xC0CA8171 module:  3 fail: 0 range: 0 id:   202 channel: 10 adc: 33137  ts: 0x00000000078E1A48
 ***            ADC data: block:     1 ptr:   109 data: 0xC0DF80F0 module:  3 fail: 0 range: 0 id:   223 channel: 31 adc: 33008  ts: 0x00000000078E1A48
 ***            ADC data: block:     1 ptr:   111 data: 0xC1F28069 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 32873  ts: 0x00000000078E1A7A
 ***            ADC data: block:     1 ptr:   113 data: 0xC1837E89 module:  6 fail: 0 range: 0 id:   387 channel:  3 adc: 32393  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   115 data: 0xC1987EC6 module:  6 fail: 0 range: 0 id:   408 channel: 24 adc: 32454  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   117 data: 0xC1B37F93 module:  6 fail: 0 range: 0 id:   435 channel: 51 adc: 32659  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   119 data: 0xC2C28169 module: 11 fail: 0 range: 0 id:   706 channel:  2 adc: 33129  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:     7 data: 0xC2D5830E module: 11 fail: 0 range: 0 id:   725 channel: 21 adc: 33550  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:     9 data: 0xC2E18150 module: 11 fail: 0 range: 0 id:   737 channel: 33 adc: 33104  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:    11 data: 0xC0117F5D module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 32605  ts: 0x00000000078E2092
 ***            ADC data: block:     2 ptr:    13 data: 0xC27E81BD module:  9 fail: 0 range: 0 id:   638 channel: 62 adc: 33213  ts: 0x00000000078E2182
 ***            ADC data: block:     2 ptr:    15 data: 0xC07D817F module:  1 fail: 0 range: 0 id:   125 channel: 61 adc: 33151  ts: 0x00000000078E21FA
 ***            ADC data: block:     2 ptr:    17 data: 0xC0CB8125 module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 33061  ts: 0x00000000078E2218
 ***            ADC data: block:     2 ptr:    19 data: 0xC1C28219 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33305  ts: 0x00000000078E224A
 ***            ADC data: block:     2 ptr:    21 data: 0xC1F77FD8 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32728  ts: 0x00000000078E224A
 ***            ADC data: block:     2 ptr:    23 data: 0xC08B7FBA module:  2 fail: 0 range: 0 id:   139 channel: 11 adc: 32698  ts: 0x00000000078E23C6
 ***            ADC data: block:     2 ptr:    25 data: 0xC09F7E74 module:  2 fail: 0 range: 0 id:   159 channel: 31 adc: 32372  ts: 0x00000000078E23C6
 ***            ADC data: block:     2 ptr:    27 data: 0xC1867FE8 module:  6 fail: 0 range: 0 id:   390 channel:  6 adc: 32744  ts: 0x00000000078E2524
 ***            ADC data: block:     2 ptr:    29 data: 0xC19A7EC3 module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 32451  ts: 0x00000000078E2524
 ***            ADC data: block:     2 ptr:    31 data: 0xC2C98098 module: 11 fail: 0 range: 0 id:   713 channel:  9 adc: 32920  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    33 data: 0xC2DA837D module: 11 fail: 0 range: 0 id:   730 channel: 26 adc: 33661  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    35 data: 0xC2E582C0 module: 11 fail: 0 range: 0 id:   741 channel: 37 adc: 33472  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    37 data: 0xC0CC81D9 module:  3 fail: 0 range: 0 id:   204 channel: 12 adc: 33241  ts: 0x00000000078E29E8
 ***            ADC data: block:     2 ptr:    39 data: 0xC187804E module:  6 fail: 0 range: 0 id:   391 channel:  7 adc: 32846  ts: 0x00000000078E2CF4
 ***            ADC data: block:     2 ptr:    41 data: 0xC2DB81B9 module: 11 fail: 0 range: 0 id:   731 channel: 27 adc: 33209  ts: 0x00000000078E2EE8
 ***            ADC data: block:     2 ptr:    43 data: 0xC2E782E0 module: 11 fail: 0 range: 0 id:   743 channel: 39 adc: 33504  ts: 0x00000000078E2EE8
 ***            ADC data: block:     2 ptr:    45 data: 0xC0CD80DF module:  3 fail: 0 range: 0 id:   205 channel: 13 adc: 32991  ts: 0x00000000078E31B8
 ***            ADC data: block:     2 ptr:    47 data: 0xC0D380C6 module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 32966  ts: 0x00000000078E31B8
 ***            ADC data: block:     2 ptr:    49 data: 0xC1897EEF module:  6 fail: 0 range: 0 id:   393 channel:  9 adc: 32495  ts: 0x00000000078E34C4
 ***            ADC data: block:     2 ptr:    51 data: 0xC2E98401 module: 11 fail: 0 range: 0 id:   745 channel: 41 adc: 33793  ts: 0x00000000078E36B8
 ***            ADC data: block:     2 ptr:    53 data: 0xC0E08147 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 33095  ts: 0x00000000078E3988
 ***            ADC data: block:     2 ptr:    55 data: 0xC2C881AC module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 33196  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    57 data: 0xC2ED8349 module: 11 fail: 0 range: 0 id:   749 channel: 45 adc: 33609  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    59 data: 0xC2F380C2 module: 11 fail: 0 range: 0 id:   755 channel: 51 adc: 32962  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    61 data: 0xC1BD7C77 module:  6 fail: 0 range: 0 id:   445 channel: 61 adc: 31863  ts: 0x00000000078E4464
 ***            ADC data: block:     2 ptr:    63 data: 0xC2D3837F module: 11 fail: 0 range: 0 id:   723 channel: 19 adc: 33663  ts: 0x00000000078E4658
 ***            ADC data: block:     2 ptr:    65 data: 0xC24F80FC module:  9 fail: 0 range: 0 id:   591 channel: 15 adc: 33020  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    67 data: 0xC26C80FA module:  9 fail: 0 range: 0 id:   620 channel: 44 adc: 33018  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    69 data: 0xC27782CE module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 33486  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    71 data: 0xC0C08052 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 32850  ts: 0x00000000078E4928
 ***            ADC data: block:     2 ptr:    73 data: 0xC08E4C1D module:  2 fail: 0 range: 0 id:   142 channel: 14 adc: 19485  ts: 0x00000000078E4AD6
 ***            ADC data: block:     2 ptr:    75 data: 0xC1817D36 module:  6 fail: 0 range: 0 id:   385 channel:  1 adc: 32054  ts: 0x00000000078E4C34
 ***            ADC data: block:     2 ptr:    77 data: 0xC1BF7FAF module:  6 fail: 0 range: 0 id:   447 channel: 63 adc: 32687  ts: 0x00000000078E4C34
 ***            ADC data: block:     2 ptr:    79 data: 0xC2E2826E module: 11 fail: 0 range: 0 id:   738 channel: 34 adc: 33390  ts: 0x00000000078E55F8
 ***            ADC data: block:     2 ptr:    81 data: 0xC1C380E1 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 32993  ts: 0x00000000078E58FA
 ***            ADC data: block:     2 ptr:    83 data: 0xC0F78257 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 33367  ts: 0x00000000078E6098
 ***            ADC data: block:     2 ptr:    85 data: 0xC2C681D1 module: 11 fail: 0 range: 0 id:   710 channel:  6 adc: 33233  ts: 0x00000000078E6598
 ***            ADC data: block:     2 ptr:    87 data: 0xC04E8051 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 32849  ts: 0x00000000078E684A
 ***            ADC data: block:     2 ptr:    89 data: 0xC0758189 module:  1 fail: 0 range: 0 id:   117 channel: 53 adc: 33161  ts: 0x00000000078E684A
 ***            ADC data: block:     2 ptr:    91 data: 0xC0948066 module:  2 fail: 0 range: 0 id:   148 channel: 20 adc: 32870  ts: 0x00000000078E6A16
 ***            ADC data: block:     2 ptr:    93 data: 0xC0057DFD module:  0 fail: 0 range: 0 id:     5 channel:  5 adc: 32253  ts: 0x00000000078E6EB2
 ***            ADC data: block:     2 ptr:    95 data: 0xC2418102 module:  9 fail: 0 range: 0 id:   577 channel:  1 adc: 33026  ts: 0x00000000078E6FA2
 ***            ADC data: block:     2 ptr:    97 data: 0xC26B8030 module:  9 fail: 0 range: 0 id:   619 channel: 43 adc: 32816  ts: 0x00000000078E6FA2
 ***            ADC data: block:     2 ptr:    99 data: 0xC0077C68 module:  0 fail: 0 range: 0 id:     7 channel:  7 adc: 31848  ts: 0x00000000078E7682
 ***            ADC data: block:     2 ptr:   101 data: 0xC0107DBC module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 32188  ts: 0x00000000078E7682
 ***            ADC data: block:     2 ptr:   103 data: 0xC26F8145 module:  9 fail: 0 range: 0 id:   623 channel: 47 adc: 33093  ts: 0x00000000078E7772
 ***            ADC data: block:     2 ptr:   105 data: 0xC0D18120 module:  3 fail: 0 range: 0 id:   209 channel: 17 adc: 33056  ts: 0x00000000078E7808
 ***            ADC data: block:     2 ptr:   107 data: 0xC2C081F4 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 33268  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   109 data: 0xC2D78240 module: 11 fail: 0 range: 0 id:   727 channel: 23 adc: 33344  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   111 data: 0xC2E48241 module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 33345  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   113 data: 0xC0137D35 module:  0 fail: 0 range: 0 id:    19 channel: 19 adc: 32053  ts: 0x00000000078E7E52
 ***            ADC data: block:     2 ptr:   115 data: 0xC2728164 module:  9 fail: 0 range: 0 id:   626 channel: 50 adc: 33124  ts: 0x00000000078E7F42
 ***            ADC data: block:     2 ptr:   117 data: 0xC0C18052 module:  3 fail: 0 range: 0 id:   193 channel:  1 adc: 32850  ts: 0x00000000078E7FD8
 ***            ADC data: block:     2 ptr:   119 data: 0xC2C38244 module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 33348  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:     7 data: 0xC2D982DC module: 11 fail: 0 range: 0 id:   729 channel: 25 adc: 33500  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:     9 data: 0xC2E683C3 module: 11 fail: 0 range: 0 id:   742 channel: 38 adc: 33731  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:    11 data: 0xC2F68233 module: 11 fail: 0 range: 0 id:   758 channel: 54 adc: 33331  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:    13 data: 0xC0177D5C module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 32092  ts: 0x00000000078E8622
 ***            ADC data: block:     3 ptr:    15 data: 0xC0C2803E module:  3 fail: 0 range: 0 id:   194 channel:  2 adc: 32830  ts: 0x00000000078E87A8
 ***            ADC data: block:     3 ptr:    17 data: 0xC0FD81FC module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 33276  ts: 0x00000000078E87A8
 ***            ADC data: block:     3 ptr:    19 data: 0xC1AD7F36 module:  6 fail: 0 range: 0 id:   429 channel: 45 adc: 32566  ts: 0x00000000078E8AB4
 ***            ADC data: block:     3 ptr:    21 data: 0xC2F78642 module: 11 fail: 0 range: 0 id:   759 channel: 55 adc: 34370  ts: 0x00000000078E8CA8
 ***            ADC data: block:     3 ptr:    23 data: 0xC0C3815B module:  3 fail: 0 range: 0 id:   195 channel:  3 adc: 33115  ts: 0x00000000078E8F78
 ***            ADC data: block:     3 ptr:    25 data: 0xC0847EE9 module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 32489  ts: 0x00000000078E9126
 ***            ADC data: block:     3 ptr:    27 data: 0xC0B77F35 module:  2 fail: 0 range: 0 id:   183 channel: 55 adc: 32565  ts: 0x00000000078E9126
 ***            ADC data: block:     3 ptr:    29 data: 0xC27B8125 module:  9 fail: 0 range: 0 id:   635 channel: 59 adc: 33061  ts: 0x00000000078E96B2
 ***            ADC data: block:     3 ptr:    31 data: 0xC08C3881 module:  2 fail: 0 range: 0 id:   140 channel: 12 adc: 14465  ts: 0x00000000078E98F6
 ***            ADC data: block:     3 ptr:    33 data: 0xC2C181D1 module: 11 fail: 0 range: 0 id:   705 channel:  1 adc: 33233  ts: 0x00000000078E9C48
 ***            ADC data: block:     3 ptr:    35 data: 0xC2D480BF module: 11 fail: 0 range: 0 id:   724 channel: 20 adc: 32959  ts: 0x00000000078E9C48
 ***            ADC data: block:     3 ptr:    37 data: 0xC2648191 module:  9 fail: 0 range: 0 id:   612 channel: 36 adc: 33169  ts: 0x00000000078E9E82
 ***            ADC data: block:     3 ptr:    39 data: 0xC04780C3 module:  1 fail: 0 range: 0 id:    71 channel:  7 adc: 32963  ts: 0x00000000078E9EFA
 ***            ADC data: block:     3 ptr:    41 data: 0xC0E6809B module:  3 fail: 0 range: 0 id:   230 channel: 38 adc: 32923  ts: 0x00000000078E9F18
 ***            ADC data: block:     3 ptr:    43 data: 0xC2C4834F module: 11 fail: 0 range: 0 id:   708 channel:  4 adc: 33615  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    45 data: 0xC2DC8132 module: 11 fail: 0 range: 0 id:   732 channel: 28 adc: 33074  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    47 data: 0xC2FD8325 module: 11 fail: 0 range: 0 id:   765 channel: 61 adc: 33573  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    49 data: 0xC0C581D9 module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 33241  ts: 0x00000000078EA6E8
 ***            ADC data: block:     3 ptr:    51 data: 0xC2E183D2 module: 11 fail: 0 range: 0 id:   737 channel: 33 adc: 33746  ts: 0x00000000078EABE8
 ***            ADC data: block:     3 ptr:    53 data: 0xC0C9806C module:  3 fail: 0 range: 0 id:   201 channel:  9 adc: 32876  ts: 0x00000000078EAEB8
 ***            ADC data: block:     3 ptr:    55 data: 0xC0DA814C module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 33100  ts: 0x00000000078EAEB8
 ***            ADC data: block:     3 ptr:    57 data: 0xC1C28260 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33376  ts: 0x00000000078EAEEA
 ***            ADC data: block:     3 ptr:    59 data: 0xC08B7FEC module:  2 fail: 0 range: 0 id:   139 channel: 11 adc: 32748  ts: 0x00000000078EB066
 ***            ADC data: block:     3 ptr:    61 data: 0xC09F7EF1 module:  2 fail: 0 range: 0 id:   159 channel: 31 adc: 32497  ts: 0x00000000078EB066
 ***            ADC data: block:     3 ptr:    63 data: 0xC2668171 module:  9 fail: 0 range: 0 id:   614 channel: 38 adc: 33137  ts: 0x00000000078EB5F2
 ***            ADC data: block:     3 ptr:    65 data: 0xC0CA8094 module:  3 fail: 0 range: 0 id:   202 channel: 10 adc: 32916  ts: 0x00000000078EB688
 ***            ADC data: block:     3 ptr:    67 data: 0xC0DF806C module:  3 fail: 0 range: 0 id:   223 channel: 31 adc: 32876  ts: 0x00000000078EB688
 ***            ADC data: block:     3 ptr:    69 data: 0xC1F782AB module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33451  ts: 0x00000000078EB6BA
 ***            ADC data: block:     3 ptr:    71 data: 0xC2C282CB module: 11 fail: 0 range: 0 id:   706 channel:  2 adc: 33483  ts: 0x00000000078EBB88
 ***            ADC data: block:     3 ptr:    73 data: 0xC07D8111 module:  1 fail: 0 range: 0 id:   125 channel: 61 adc: 33041  ts: 0x00000000078EBE3A
 ***            ADC data: block:     3 ptr:    75 data: 0xC0CB81B8 module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 33208  ts: 0x00000000078EBE58
 ***            ADC data: block:     3 ptr:    77 data: 0xC2E98262 module: 11 fail: 0 range: 0 id:   745 channel: 41 adc: 33378  ts: 0x00000000078EC358
 ***            ADC data: block:     3 ptr:    79 data: 0xC044807F module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 32895  ts: 0x00000000078EC60A
 ***            ADC data: block:     3 ptr:    81 data: 0xC19A7DA3 module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 32163  ts: 0x00000000078EC934
 ***            ADC data: block:     3 ptr:    83 data: 0xC2ED8207 module: 11 fail: 0 range: 0 id:   749 channel: 45 adc: 33287  ts: 0x00000000078ECB28
 ***            ADC data: block:     3 ptr:    85 data: 0xC2F380FC module: 11 fail: 0 range: 0 id:   755 channel: 51 adc: 33020  ts: 0x00000000078ECB28
 ***            ADC data: block:     3 ptr:    87 data: 0xC2708152 module:  9 fail: 0 range: 0 id:   624 channel: 48 adc: 33106  ts: 0x00000000078ECD62
 ***            ADC data: block:     3 ptr:    89 data: 0xC1B47F47 module:  6 fail: 0 range: 0 id:   436 channel: 52 adc: 32583  ts: 0x00000000078ED104
 ***            ADC data: block:     3 ptr:    91 data: 0xC2C88150 module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 33104  ts: 0x00000000078ED2F8
 ***            ADC data: block:     3 ptr:    93 data: 0xC27781B4 module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 33204  ts: 0x00000000078ED532
 ***            ADC data: block:     3 ptr:    95 data: 0xC0C081FF module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 33279  ts: 0x00000000078ED5C8
 ***            ADC data: block:     3 ptr:    97 data: 0xC1C780DD module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32989  ts: 0x00000000078ED5FA
 ***            ADC data: block:     3 ptr:    99 data: 0xC1BD7D53 module:  6 fail: 0 range: 0 id:   445 channel: 61 adc: 32083  ts: 0x00000000078ED8D4
 ***            ADC data: block:     3 ptr:   101 data: 0xC2D38300 module: 11 fail: 0 range: 0 id:   723 channel: 19 adc: 33536  ts: 0x00000000078EDAC8
 ***            ADC data: block:     3 ptr:   103 data: 0xC0027F2E module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 32558  ts: 0x00000000078EDC12
 ***            ADC data: block:     3 ptr:   105 data: 0xC0117FD8 module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 32728  ts: 0x00000000078EDC12
 ***            ADC data: block:     3 ptr:   107 data: 0xC2588009 module:  9 fail: 0 range: 0 id:   600 channel: 24 adc: 32777  ts: 0x00000000078EDD02
 ***            ADC data: block:     3 ptr:   109 data: 0xC1CD810E module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33038  ts: 0x00000000078EDDCA
 ***            ADC data: block:     3 ptr:   111 data: 0xC1927F0C module:  6 fail: 0 range: 0 id:   402 channel: 18 adc: 32524  ts: 0x00000000078EE0A4
 ***            ADC data: block:     3 ptr:   113 data: 0xC2DB8197 module: 11 fail: 0 range: 0 id:   731 channel: 27 adc: 33175  ts: 0x00000000078EE298
 ***            ADC data: block:     3 ptr:   115 data: 0xC2E7821D module: 11 fail: 0 range: 0 id:   743 channel: 39 adc: 33309  ts: 0x00000000078EE298
 ***            ADC data: block:     3 ptr:   117 data: 0xC0097F87 module:  0 fail: 0 range: 0 id:     9 channel:  9 adc: 32647  ts: 0x00000000078EE3E2
 ***            ADC data: block:     3 ptr:   119 data: 0xC2598015 module:  9 fail: 0 range: 0 id:   601 channel: 25 adc: 32789  ts: 0x00000000078EE4D2
 ***            ADC data: block:     4 ptr:     7 data: 0xC0E080A6 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 32934  ts: 0x00000000078EE568
 ***            ADC data: block:     4 ptr:     9 data: 0xC08E4C7E module:  2 fail: 0 range: 0 id:   142 channel: 14 adc: 19582  ts: 0x00000000078EE716
 ***            ADC data: block:     4 ptr:    11 data: 0xC2E88140 module: 11 fail: 0 range: 0 id:   744 channel: 40 adc: 33088  ts: 0x00000000078EEA68
 ***            ADC data: block:     4 ptr:    13 data: 0xC279812E module:  9 fail: 0 range: 0 id:   633 channel: 57 adc: 33070  ts: 0x00000000078EECA2
 ***            ADC data: block:     4 ptr:    15 data: 0xC0438270 module:  1 fail: 0 range: 0 id:    67 channel:  3 adc: 33392  ts: 0x00000000078EED1A
 ***            ADC data: block:     4 ptr:    17 data: 0xC0D280BB module:  3 fail: 0 range: 0 id:   210 channel: 18 adc: 32955  ts: 0x00000000078EED38
 ***            ADC data: block:     4 ptr:    19 data: 0xC0F78176 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 33142  ts: 0x00000000078EED38
 ***            ADC data: block:     4 ptr:    21 data: 0xC1817BAB module:  6 fail: 0 range: 0 id:   385 channel:  1 adc: 31659  ts: 0x00000000078EF044
 ***            ADC data: block:     4 ptr:    23 data: 0xC2F481FD module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 33277  ts: 0x00000000078EF238
 ***            ADC data: block:     4 ptr:    25 data: 0xC27A7E9E module:  9 fail: 0 range: 0 id:   634 channel: 58 adc: 32414  ts: 0x00000000078EF472
 ***            ADC data: block:     4 ptr:    27 data: 0xC04E80E8 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 33000  ts: 0x00000000078EF4EA
 ***            ADC data: block:     4 ptr:    29 data: 0xC0D3809C module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 32924  ts: 0x00000000078EF508
 ***            ADC data: block:     4 ptr:    31 data: 0xC2C6810C module: 11 fail: 0 range: 0 id:   710 channel:  6 adc: 33036  ts: 0x00000000078EFA08
 ***            ADC data: block:     4 ptr:    33 data: 0xC257808A module:  9 fail: 0 range: 0 id:   599 channel: 23 adc: 32906  ts: 0x00000000078EFC42
 ***            ADC data: block:     4 ptr:    35 data: 0xC0107E1C module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 32284  ts: 0x00000000078F0322
 ***            ADC data: block:     4 ptr:    37 data: 0xC25A7FA5 module:  9 fail: 0 range: 0 id:   602 channel: 26 adc: 32677  ts: 0x00000000078F0412

[npg@aidas-gsi analyser]$ 

  179   Mon Mar 8 07:02:17 2021 CA, LSMonday 8th March 08:00 - 17:00
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude 1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

08:20 Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

      Writing to file R6_1148

09:00 Beam off for optimisation - writing to file R6_1212

      Expected back at 09:30

09:30 Beam will be off for a bit longer - an hour or two

09:34 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
      Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds 
to finish
      Something confusing, will look at data and report to VP
      Reverted to original merger and back... 45 MB/s
      R9

12.00 (LS)
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment7)
      FEE temps (attatchment8)
      Rate spectra (attachment9), beam still down
      Leakage Currents written to sheets (attachment10)
      Merger ~45M items/s
      Tapeserver ~43MB/s

14.00 Still no beam but is expected to return soon
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment11)
      FEE temps (attatchment12)
      Leakage Currents written to sheets (attachment13)
      Merger ~45M items/s
      Tapeserver ~44MB/s

14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)

15.00 Beam is back (attachment15)
      Still writing to R9 (roughly around R9_386)

15.28 No beam (attachment16)

15.41 Beam back (attachment17)
      Roughly R9 (roughly around R9_439)

16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
 
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment19)
      Spectra Rate (attachment20)
      FEE temps (attatchment21)
      Leakage Currents written to sheets (attachment22), 
      still rising but looking on Grafana 7 day period should start to lower around 17.00
      Merger ~45M items/s
      Tapeserver ~43MB/s

16.55 Beam down (attachment23), back a couple minutes later

15.51 beam down (attachment24)
      
Attachment 1: Screenshot_from_2021-03-08_07-04-21.png
Screenshot_from_2021-03-08_07-04-21.png
Attachment 2: Screenshot_from_2021-03-08_07-07-24.png
Screenshot_from_2021-03-08_07-07-24.png
Attachment 3: Screenshot_from_2021-03-08_07-07-49.png
Screenshot_from_2021-03-08_07-07-49.png
Attachment 4: Screenshot_from_2021-03-08_08-31-37.png
Screenshot_from_2021-03-08_08-31-37.png
Attachment 5: Screenshot_from_2021-03-08_08-32-58.png
Screenshot_from_2021-03-08_08-32-58.png
Attachment 6: Screenshot_from_2021-03-08_08-34-25.png
Screenshot_from_2021-03-08_08-34-25.png
Attachment 7: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 8: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 11: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 12: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 13: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 14: 1500_Grafana_LeakageCurrents.PNG
1500_Grafana_LeakageCurrents.PNG
Attachment 15: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 16: 1528_SpectraRate.PNG
1528_SpectraRate.PNG
Attachment 17: 1541_SpectraRate.PNG
1541_SpectraRate.PNG
Attachment 18: 1600_R9_469_hit_rates.PNG
1600_R9_469_hit_rates.PNG
Attachment 19: 1620_Statistics.PNG
1620_Statistics.PNG
Attachment 20: 1620_SpectraRate.PNG
1620_SpectraRate.PNG
Attachment 21: 1620_FEETemps.PNG
1620_FEETemps.PNG
Attachment 22: 1620_LeakageCurrents.PNG
1620_LeakageCurrents.PNG
Attachment 23: 1655_SpectraRate.PNG
1655_SpectraRate.PNG
Attachment 24: 1749_SpectraRate.PNG
1749_SpectraRate.PNG
  178   Sun Mar 7 23:13:50 2021 OHMonday 8th March
00:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


00:14 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4

02:15 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped

04:00 Noticed more bad merge events
      System wide checks has also noted an FPGA error.
      All other system wide checks same results as previous

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A3F29 BA0DF4D2 , WR/10=>   23DD31DC5CE3215, Readout Time =>   23DD31DC6F34000 
aida02 : White Rabbit=>  166A3F29 CC49ECD7 , WR/10=>   23DD31DC7A0FE15, Readout Time =>   23DD31DC8428000 
aida03 : White Rabbit=>  166A3F29 DF55B7E8 , WR/10=>   23DD31DC9889264, Readout Time =>   23DD31DCA85C000 
aida04 : White Rabbit=>  166A3F29 EECF82B7 , WR/10=>   23DD31DCB14C045, Readout Time =>   23DD31DCC2B8000 
aida05 : White Rabbit=>  166A3F29 FED2EFD2 , WR/10=>   23DD31DCCAEB195, Readout Time =>   23DD31DCDC14000 
aida06 : White Rabbit=>  166A3F2A 0EDC023D , WR/10=>   23DD31DCE49336C, Readout Time =>   23DD31DCF204000 
aida07 : White Rabbit=>  166A3F2A 1E8170D3 , WR/10=>   23DD31DCFD9BE7B, Readout Time =>   23DD31DD10A0000 
aida08 : White Rabbit=>  166A3F2A 30448F76 , WR/10=>   23DD31DD1A074BF, Readout Time =>   23DD31DD2A34000 
aida09 : White Rabbit=>  166A3F2A 3FE36E61 , WR/10=>   23DD31DD33057D6, Readout Time =>   23DD31DD3B94000 
aida10 : White Rabbit=>  166A3F2A 4E83A1F7 , WR/10=>   23DD31DD4A6C365, Readout Time =>   23DD31DD46A8000 
aida11 : White Rabbit=>  166A3F2A 5FD19086 , WR/10=>   23DD31DD661C1A7, Readout Time =>   23DD31DD7664000 
aida12 : White Rabbit=>  166A3F2A 6FE06F3D , WR/10=>   23DD31DD7FCD7EC, Readout Time =>   23DD31DD9684000 

04:07 Beam is back AIDA on file R6_815
      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10
      ASIC check ok

      Multiple merge data error events observed

06:04 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 11
      FEE Temperatures - attachment 12
      Bias and leakage currents ok - attachment 13

07:22 Beam noticed dropped but back quickly
Attachment 1: 210308_0014_Stats.png
210308_0014_Stats.png
Attachment 2: 2303108_0015_Temp.png
2303108_0015_Temp.png
Attachment 3: 210308_0016_Bias.png
210308_0016_Bias.png
Attachment 4: 210308_0120_System.png
210308_0120_System.png
Attachment 5: 210308_0213_Stats.png
210308_0213_Stats.png
Attachment 6: 210308_0213_Temp.png
210308_0213_Temp.png
Attachment 7: 210308_0215_Bias.png
210308_0215_Bias.png
Attachment 8: 210308_0403_Stats.png
210308_0403_Stats.png
Attachment 9: 210308_0404_Temp.png
210308_0404_Temp.png
Attachment 10: 210308_04011_Bias.png
210308_04011_Bias.png
Attachment 11: 210308_0602_Stats.png
210308_0602_Stats.png
Attachment 12: 210308_0603_Temp.png
210308_0603_Temp.png
Attachment 13: 210308_0604_Bias.png
210308_0604_Bias.png
  177   Sun Mar 7 07:23:26 2021 CA, LSMarch 7th
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:30 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

      DAQ continues to run - R1_1148

      Merger ok - 4.6M data items/s

      TapeServer ok - 43MB/s

08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots

09:19 zeroed all histograms

      writing to file R1_1204

      FRS on file 72

      DESPEC to add AIDA XY hit patterns to online analysis histograms

09:35 beam stop

09:56 beam back 

      writing to file R1_1249

      rate spectra - attachment 5

09:30 AIDA DAQ crashed - lost connection to aida05

      power cycle and reset

09:40 AIDA ok now, writing to file R2

09:50 LS takes over

11.10 (German time)
      System wide checks produces some fails: 
      FEE64 module aida06 global clocks failed, 6 (screenshot6)
      This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there

      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2 (screenshot7)
      Tried a recalibration for the two FEEs in FADC align & control tab but error still there

      Rest of the checks pass
      Statistics (screenshot8)
      FEE temps okay (screenshot9)
      Leakage currents written to sheets (screenshot10)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

13.00 System wide checks produces same fails as last time: 
      FEE64 module aida06 global clocks failed, 6
      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2
      Rest of the checks are okay

      Statistics (screenshot11)
      Spectra rate (screenshot12)
      FEE temps okay (screenshot13)
      Leakage currents written to sheets (screenshot14)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s
      There is 1.5TB left on current drive

14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold
      Slow comparators in FEE9-12 raised to 0xd
      

      

      
      
      



        
Attachment 1: Screenshot_from_2021-03-07_07-30-28.png
Screenshot_from_2021-03-07_07-30-28.png
Attachment 2: Screenshot_from_2021-03-07_07-31-27.png
Screenshot_from_2021-03-07_07-31-27.png
Attachment 3: Screenshot_from_2021-03-07_07-31-55.png
Screenshot_from_2021-03-07_07-31-55.png
Attachment 4: Screenshot_from_2021-03-07_07-53-05.png
Screenshot_from_2021-03-07_07-53-05.png
Attachment 5: Screenshot_from_2021-03-07_08-59-23.png
Screenshot_from_2021-03-07_08-59-23.png
Attachment 6: 1110_Systemchecks.PNG
1110_Systemchecks.PNG
Attachment 7: 1110_Systemchecks2.PNG
1110_Systemchecks2.PNG
Attachment 8: 1110_Statistics.PNG
1110_Statistics.PNG
Attachment 9: 1110_FEEtemps.PNG
1110_FEEtemps.PNG
Attachment 10: 1110_Leakagecurrents.PNG
1110_Leakagecurrents.PNG
Attachment 11: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 12: 1300_Spectrumrate.PNG
1300_Spectrumrate.PNG
Attachment 13: 1300_FEEtemps.PNG
1300_FEEtemps.PNG
Attachment 14: 1300_Leakagecurrents.PNG
1300_Leakagecurrents.PNG
  176   Sat Mar 6 23:37:22 2021 TDSunday 7 March 00.00-08:00
00.30 UNILAC problem - no beam

00:39 DAQ continues R1_518
      FRS 190W setting 

      ASIC settinfs 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

01:07 all system wide checks OK

      detector biases & leakage currents OK - see attachment 1
      FEE64 temperatures OK - see attachment 2
      good event statistics - OK - see attachment 3
      Merger OK - see attachment 4
      TapeServer OK - see attachment 5

01:20 zero'd all histograms
      ASIC check load

      rate spectra - see attachment 6

01:48 beam return

      run74 (R1_608)

03.05 all system wide checks OK

      detector biases & leakage currents OK - see attachment 7
      FEE64 temperatures OK - see attachment 8
      good event statistics - OK - see attachment 9
      Merger OK - see attachment 10
      TapeServer OK - see attachment 11

     rate spectra - see attachment 12

04.46 run74 ends (R1_841)

05.00 run75 (R1_859)

05:40 detector biases & leakage currents OK - see attachment 13
      FEE64 temperatures OK - see attachment 14
      good event statistics - OK - see attachment 15
      Merger OK - see attachment 16
      TapeServer OK - see attachment 17

     rate spectra - see attachment 18

07:29 run76 (R1_1055)
Attachment 1: 1.png
1.png
Attachment 2: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 3: Screenshot_2021-03-07_Statistics_aidas-gsi.png
Screenshot_2021-03-07_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png
Attachment 5: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: 2.png
2.png
Attachment 8: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 9: Screenshot_2021-03-07_Statistics_aidas-gsi(1).png
Screenshot_2021-03-07_Statistics_aidas-gsi(1).png
Attachment 10: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png
Attachment 11: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png
Attachment 12: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: 3.png
3.png
Attachment 14: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 15: Screenshot_2021-03-07_Statistics_aidas-gsi(2).png
Screenshot_2021-03-07_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png
Attachment 17: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png
Attachment 18: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png
  175   Sat Mar 6 06:52:06 2021 OH, LSSaturday 6th March
07:50 Pulser settings - attachment 1

07:52 System wide checks all ok
      Statistics ok - attachment 2
      Bias ok - attachment 3

09:30 The intensity of fragments is being increased
      They are trying to do a factor of 10 increase first which should take us to around 100Hz

10:00 System wide checks all ok
      Statistics - attachment 4
      Layout 1 showing implants - attachment 5
      Fee temperatures ok - attachment 6

12.00 System wide checks all okay, no failures
      Statistics (screenshot7)
      Spectrum rate (screenshot8)
      FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal 
      (screenshot10)
      Leakage currents okay, increase from previous values written to sheets (screenshot12)
      
      Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer 
than 
      usual, another reload shows all FEE temps as normal.
      
      Merger and tape server okay

14.00 System wide checks all okay, no failures
      Statistics (screenshot13)
      Spectrum rate (screenshot14)
      FEE temps normal, no issue with reload this time (screenshot15)
      Leakage currents okay still increasing, written to sheets (screenshot16)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)

16.00 System wide checks all okay, no failures
      Statistics (screenshot17)


      Spectrum rate (screenshot18)      
      FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
      Leakage currents added to sheet (screenshot21)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

17.17 no beam, preparing for the 190Ta setting

17:44 OH Takes over
      System wide checks all ok
      Statistics - attachment 22
      Temp ok - Attachment 23
      Bias and leakage currents ok - Attatchment 24

      While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
      The Directory for the experiment was then setup S452
      The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.

      ASIC Check all ok
      

18:10 Started writing data to file
      File directory S452
      Run number R1_0
      Seems to have skipped a few files First full file R1_10
      Currently writing to /media/SecondDrive/TapeData
      Current free space 4211084180 kB
      Current write speed 46163 kB/s
      Time remaining until full 91222 seconds
      25 hours remaining space

18:34 DESPEC starting new MBS file
      AIDA Currently on file R1_41

19:00 DESPEC Closing file
      AIDA Currently on R1_75

19:05 Comparing last years statistics to this years

FEE	Last year	This year    Factor increase
1	60960	        183111       3.00
2	183494	        201256       1.09
3	91506	        196647       2.15
4	73914	        208290       2.81
5	31979	        63571        1.99
6	54084	        144638       2.67
7	35621	        137605       3.86
8	38626	        95858        2.48
9	264436	        225790       0.85
10	209709	        198896       0.94
11	88443	        133899       1.51
12	183493	        204996       1.12

Sum     1316265	        1994557      1.51

19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
      AIDA on R1_104

      Looking int to high write  rate to see if we can do anything to reduce it
      It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24

19:46 Much of the rate is coming from a small number of channels - attachment 25

19:17 WR Timestamp error.
      Mentioned to Nic and he said that sometimes small glitches like this are observed
		 Base 		Current 	Difference
aida01 fault 	 0x9e8b : 	 0x9e8d : 	 2  
aida02 fault 	 0x9a8f : 	 0x9a91 : 	 2  
aida03 fault 	 0x29e9 : 	 0x29eb : 	 2  
aida04 fault 	 0x7f66 : 	 0x7f68 : 	 2  
aida05 fault 	 0x1f4f : 	 0x1f51 : 	 2  
aida06 fault 	 0x8faa : 	 0x8fac : 	 2  
aida07 fault 	 0x53f0 : 	 0x53f2 : 	 2  
aida08 fault 	 0x7066 : 	 0x7068 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     All other checks passed

20:52 DESPEC Stopping file
      AIDA on R1_221


20:53 DESPEC Stopped file because UCESB crashed
      AIDA on file R1_237

21:20 All system wide  checks ok
      Statistics DISC info- Attachment 27
      FEE Temp ok - Attachment 28
      Bias and leakage current ok - Attachment 29
      Statistics good events - Attachment 30

20:51 Started compressing data with command nice -n 10 gzip -v R1_*
      Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
      I am not sure if MBS can keep up with this rate
      We are still writing to file though.

23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us
Attachment 1: Pulser.png
Pulser.png
Attachment 2: 210306_0752_Stats.png
210306_0752_Stats.png
Attachment 3: 210306_0754_Bias.png
210306_0754_Bias.png
Attachment 4: 210306_0958_Stats.png
210306_0958_Stats.png
Attachment 5: 210306_0959_Layout1.png
210306_0959_Layout1.png
Attachment 6: 210306_1000_Temp.png
210306_1000_Temp.png
Attachment 7: 1102_Statistics.PNG
1102_Statistics.PNG
Attachment 8: 1102_SpectrumRate.PNG
1102_SpectrumRate.PNG
Attachment 9: 1102_FEETemps.PNG
1102_FEETemps.PNG
Attachment 10: 1102_FEETemps2.PNG
1102_FEETemps2.PNG
Attachment 11: 1102_FEETemps3.PNG
1102_FEETemps3.PNG
Attachment 12: 1102_LeakageCurrents.PNG
1102_LeakageCurrents.PNG
Attachment 13: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 14: 1300_SpectrumRate.PNG
1300_SpectrumRate.PNG
Attachment 15: 1300_FEETemps.PNG
1300_FEETemps.PNG
Attachment 16: 1300_LeakageCurrents.PNG
1300_LeakageCurrents.PNG
Attachment 17: 1500_Statistics.PNG
1500_Statistics.PNG
Attachment 18: 1500_SpectrumRate.PNG
1500_SpectrumRate.PNG
Attachment 19: 1500_FEETemps.PNG
1500_FEETemps.PNG
Attachment 20: 1500_FEETemps2.PNG
1500_FEETemps2.PNG
Attachment 21: 1500_LeakageCurrents.PNG
1500_LeakageCurrents.PNG
Attachment 22: 210306_1726_Stats.png
210306_1726_Stats.png
Attachment 23: 210306_1747_Temp.png
210306_1747_Temp.png
Attachment 24: 210327_1748_Bias.png
210327_1748_Bias.png
Attachment 25: 210326_1920_MergerStats.png
210326_1920_MergerStats.png
Attachment 26: 210306_1946_Layout1.png
210306_1946_Layout1.png
Attachment 27: 210306_2120_Stats.png
210306_2120_Stats.png
Attachment 28: 210306_2121_Temp.png
210306_2121_Temp.png
Attachment 29: 210306_2123_Bias.png
210306_2123_Bias.png
Attachment 30: 210326_1920_MergerStats.png
210326_1920_MergerStats.png
  174   Fri Mar 5 23:45:41 2021 TDS452 information
aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S452 shift checks - attachment 1

S452 shift plan - PDF attachment 2 - Google doc https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1745844766

Mattermost - https://mattermost.gsi.de/despec/channels/experiment-s452-nearline-sort

DESPEC Zoom meeting

Join Zoom Meeting
https://gsi-fair.zoom.us/j/98980822569

Meeting ID: 989 8082 2569
Passcode: S452

FRS Zoom meeting 

Meeting-ID: 973 4809 5114

password: FRS-BT2021

Kenncode: 456657
Attachment 1: S452__Shifts_checkpoints_v1_(1).pdf
S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf
Attachment 2: S452_Shift_Plan.pdf
S452_Shift_Plan.pdf
  173   Fri Mar 5 23:21:59 2021 CA, OHMarch 6th 2021 00:00 - 08:00 shift
00:23 206Hg setting - No beam in AIDA yet

      AIDA not writing to file

      Data being forwarded to MBS

      Fragments expected tonight (?)

00:45 GSI DAQ restart

      FEE64 Temps ok - attachment 1

      All system wide checks ok

01:00 Good event statistics - attachment 2

01:48 detector biases/ leakage currents - ok - attachment 3

02:10 beam on, plug removed

      beam delivered to AIDA

      rates spectra - attachment 4

      peaks at ~ 10 counts/channel in DSSD1 c.2000Hz

02:20 beam stop

02:26 beam restart - rate spectra - attachment 5

02:43 high energy channel spectra - attachment 6

      saved the high energy spectra to layout 2 in spectrum browser

03:00 beam stop

03:08 temperatures ok - attachment 7

      statistics ok - attachment 8

      bias/leakage currents ok - attachment 9

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

      all system wide checks ok

04:24 beam is back

      rate spectra - attachment 10

04:51 206Hg implanting in DSSD2

      Rate spectra - attachment 11

      HEC spectra - attachment 12

05:08 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias/leakage currents ok - attachment 15

      All system wide checks ok

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok 

05:45 beam stopped

05:50 beam back

06:45 beam running, but no implants in AIDA

      possible issues with beam delivery to S4

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

07:10 temperatures ok - attachment 16

      statistics ok - attachment 17

      detector bias/leakage currents ok - attachment 18

      all system wide checks OK














      

      

      
Attachment 1: Screenshot_from_2021-03-06_00-00-27.png
Screenshot_from_2021-03-06_00-00-27.png
Attachment 2: Screenshot_from_2021-03-06_00-05-18.png
Screenshot_from_2021-03-06_00-05-18.png
Attachment 3: Screenshot_from_2021-03-06_00-47-41.png
Screenshot_from_2021-03-06_00-47-41.png
Attachment 4: Screenshot_from_2021-03-06_01-17-12.png
Screenshot_from_2021-03-06_01-17-12.png
Attachment 5: Screenshot_from_2021-03-06_01-25-51.png
Screenshot_from_2021-03-06_01-25-51.png
Attachment 6: Screenshot_from_2021-03-06_01-42-58.png
Screenshot_from_2021-03-06_01-42-58.png
Attachment 7: Screenshot_from_2021-03-06_02-05-46.png
Screenshot_from_2021-03-06_02-05-46.png
Attachment 8: Screenshot_from_2021-03-06_02-06-32.png
Screenshot_from_2021-03-06_02-06-32.png
Attachment 9: Screenshot_from_2021-03-06_02-07-47.png
Screenshot_from_2021-03-06_02-07-47.png
Attachment 10: Screenshot_from_2021-03-06_03-24-28.png
Screenshot_from_2021-03-06_03-24-28.png
Attachment 11: Screenshot_from_2021-03-06_03-53-35.png
Screenshot_from_2021-03-06_03-53-35.png
Attachment 12: Screenshot_from_2021-03-06_03-54-17_-_1.png
Screenshot_from_2021-03-06_03-54-17_-_1.png
Attachment 13: Screenshot_from_2021-03-06_04-09-25.png
Screenshot_from_2021-03-06_04-09-25.png
Attachment 14: Screenshot_from_2021-03-06_04-10-07.png
Screenshot_from_2021-03-06_04-10-07.png
Attachment 15: Screenshot_from_2021-03-06_04-10-32.png
Screenshot_from_2021-03-06_04-10-32.png
Attachment 16: Screenshot_from_2021-03-06_06-07-19.png
Screenshot_from_2021-03-06_06-07-19.png
Attachment 17: Screenshot_from_2021-03-06_06-08-16.png
Screenshot_from_2021-03-06_06-08-16.png
Attachment 18: Screenshot_from_2021-03-06_06-08-44.png
Screenshot_from_2021-03-06_06-08-44.png
  172   Fri Mar 5 22:53:05 2021 OHThursday 5th of March
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.

The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.

Beam plug is in front of AIDA. This is causing a huge spike in the Ge.
  171   Thu Mar 4 11:16:13 2021 OHFurther cases of possible database issues
During testing for the S452 experiment yesterday we were running through the steps of a restart.

The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.

Checking the folder manually it could be seen that the Options file was there.

At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.

This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.

During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.

At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally.
Attachment 1: ErrorMessage.PNG
ErrorMessage.PNG
Attachment 2: errorMessage.txt
EXPERIMENTS/AIDA/Options/aida04 TS_SYNC_PHASE NOT available: error 0x10004; value TS_SYNC_PHASE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ExtClk NOT available: error 0x10004; value ExtClk does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.shift NOT available: error 0x10004; value Aida.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 MACB_TRIG_MODE NOT available: error 0x10004; value MACB_TRIG_MODE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.offset NOT available: error 0x10004; value Aida.offset does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 RunNumber NOT available: error 0x10004; value RunNumber does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_GroupBase NOT available: error 0x10004; value Aida_GroupBase does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Rate.channels NOT available: error 0x10004; value Rate.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.channels NOT available: error 0x10004; value Stat.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Vchannels NOT available: error 0x10004; value Aida.Vchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC.settings NOT available: error 0x10004; value ASIC.settings does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_D_Enable NOT available: error 0x10004; value Aida_Hist_D_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 WAVE_DMA_HWM NOT available: error 0x10004; value WAVE_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.shift NOT available: error 0x10004; value Stat.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Wchannels NOT available: error 0x10004; value Aida.Wchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_V_Enable NOT available: error 0x10004; value Aida_Hist_V_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Include.Aida NOT available: error 0x10004; value Include.Aida does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.channels NOT available: error 0x10004; value Aida.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataAcqPgm NOT available: error 0x10004; value DataAcqPgm does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_H_Enable NOT available: error 0x10004; value Aida_Hist_H_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataFormat NOT available: error 0x10004; value DataFormat does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC_DMA_HWM NOT available: error 0x10004; value ASIC_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_L_Enable NOT available: error 0x10004; value Aida_Hist_L_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
STATE for aida01 returned with an error
error: SOAP http transport timed out after 20000 ms
NONE
error: SOAP http transport timed out after 20000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
  170   Thu Mar 4 11:03:04 2021 NHS452 Operating Notes
Please note the following "quirks" currently present in S452 that are important

a) Strips/Thresholds
Thresholds are 0xA, 0xA, 0x10 [Maybe changed depending on rate, etc]
Some strips are very noisy - damage?

b) Powering FEE64s
Due to a misbehaving USB hub(?) or cable only 7 FEE TTYs are monitored
aida04 and aida05 can take minutes to start, unsure why.
aida05 is monitored on nnrpi1 and hence when aida05 is ready it is safe to assume all 12 are ready
Watch the logs as normal

c) Slow control & nnrpi2
For the Grafana script to read the DSSD HV, it must be ran in a screen session
If nnrpi2 is restarted please run the HV as a screen by Putty SSHing (localhost 22)
and running
screen -S caenhv /dev/ttyACM0
if screen was just closed you can do
screen -x caenhv
to reconnect

The pulser ideally should be started the same
screen -S pulser /dev/ttyUSB0
To allow remote changing by NH/GSI
The pulser is sent to all subsystems for correlation and should be kept on at 2 Hz for the experiment
IN the pulser putty you can check the settings by typing
display settings
And general commands:
help

The NIM crate can be monitored /dev/ttyUSB2 but you should be careful
It is very easy to turn off the NIM crate if you do
I run it in a tmux session with (
tmux attach -r
) to have it "readonly"

There is a small script on nnrpi2
./check_usbs.sh
to see which ttyUSB is which device
As they can change if the pi is rebooted

d) Scaler Readout
If powercycled FEEs, enable the scaler in Local Controls page by setting BuTiS interface control to 0x3

To enable scaler readout after GOing the run please open the Correlation Control page
(Act on all FEE64s) and set control to 0x1
This has to be done every time the FEE64s are set to GOing

e) New Tape Server
For waveform readout in future experiments, the tape server is in "Expert" mode which is harder to use
Please read the "How to use" instructions by clicking the button on the bottom right
Or in a terminal
cat /MIDAS/TclHttpd/Html/ETape/1.txt

f) GSI/MBS
If GSI team ask to restart AIDA MBS you may need to restart the MBS relay so it can reconnect

To start MBS in a terminal
ssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup
It should report it connected to AIDA-3 - if not check the MBS relay is running or restart the relay

Please check the MBS rate command
rate -nst -rst -nev -rev -ndata -rdata -st
That the "Streams" column does not frequently (ever) have 0 empty streams
If so AIDA data may be getting dropped and should be looked into
If another DESPEC DAQ has failed this will eventually happen and is not a problem
But if it occurs in normal operation it is concerning

Do NOT stop the AIDA run without talking to GSI team - if no AIDA data the whole MBS DAQ stops
and blocks all other subsystems. It is best to keep AIDA ticking over constantly and let
MBS start/stop files when relevant.
Of course if things have to stop it is OK but inform GSI shifters so they can take action
  169   Thu Mar 4 10:13:48 2021 OHAnyDesk connection instuction
Attachment 1: Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
  168   Wed Mar 3 15:02:19 2021 CAMBS command line startup
/MIDAS/Linux/startup/datarelaymbs 
  167   Sat Feb 27 12:47:57 2021 NHAIDA Rates/Situation
Updated figures as of 27 Feb 2021 - S4 is now closed for experiments but may open on Wednesday.
Snout has been grounded and all FEEs are seemingly grounded properly.

Figure  1: Good Events
Figure  2: Rate Histograms (Log-y)
Figure  3: Waveforms (Odd)
Figure  4: Waveforms (Even)
Figure  5: Waveforms @ 5MS/s (Odd)
Figure  6: Waveforms @ 5MS/s (Even)
Figure  7: Waveforms with Pulser (Odd)
Figure  8: Waveforms with Pulser (Even)*
Figure  9: Waveforms with Pulser @ 5MS/s (Odd)
Figure 10: Waveforms with Pulser @ 5MS/s (Even)*

* y-Scale enlarged from 0-10000 to 0-15000 to capture top of pulse
Attachment 1: firefox_2021-02-27_13-47-19.png
firefox_2021-02-27_13-47-19.png
Attachment 2: firefox_2021-02-27_13-46-59.png
firefox_2021-02-27_13-46-59.png
Attachment 3: firefox_2021-02-27_13-36-27.png
firefox_2021-02-27_13-36-27.png
Attachment 4: firefox_2021-02-27_13-36-52.png
firefox_2021-02-27_13-36-52.png
Attachment 5: firefox_2021-02-27_13-38-19.png
firefox_2021-02-27_13-38-19.png
Attachment 6: firefox_2021-02-27_13-37-53.png
firefox_2021-02-27_13-37-53.png
Attachment 7: firefox_2021-02-27_13-39-48.png
firefox_2021-02-27_13-39-48.png
Attachment 8: firefox_2021-02-27_13-42-55.png
firefox_2021-02-27_13-42-55.png
Attachment 9: firefox_2021-02-27_13-39-33.png
firefox_2021-02-27_13-39-33.png
Attachment 10: firefox_2021-02-27_13-44-10.png
firefox_2021-02-27_13-44-10.png
  166   Mon Feb 22 18:18:47 2021 NHAIDA Rates/Situation
The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.

(All data below taken with thresholds 0xA for all)

Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"

Leakage currents high but S4 very warm at the moment as roof is still closed.

Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.

Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done

Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)

aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3
Attachment 1: GoodRates.png
GoodRates.png
Attachment 2: ADCRates.png
ADCRates.png
Attachment 3: Rates.png
Rates.png
Attachment 4: Waves_0.png
Waves_0.png
Attachment 5: Waves_10.png
Waves_10.png
Attachment 6: EvenWaves_0.png
EvenWaves_0.png
Attachment 7: EvenWaves_10.png
EvenWaves_10.png
Attachment 8: Grafana.png
Grafana.png
  165   Fri Feb 12 17:31:04 2021 NHBackground run
A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21

Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64

S4 access will be limited. 

I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal.
  164   Sat Feb 6 10:45:55 2021 NHAIDA Status Feb 2021
> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
> 
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
> 
> The system has been powered up and biased successfully. To be monitored over the next few days.
> 
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
> 
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
> 
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
> 
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.

The hose has arrived and the water line will be prepared in my office mid next week.

All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)

A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.

The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.

There will be a DESPEC Dry Run next week Mon-Tue.
Attachment 1: pregrd050222_E.png
pregrd050222_E.png
Attachment 2: pregrd050221_O.png
pregrd050221_O.png
Attachment 3: postgrd050221_O.png
postgrd050221_O.png
Attachment 4: postgrd050221_E.png
postgrd050221_E.png
  163   Mon Feb 1 15:19:07 2021 NHAIDA Status Feb 2021
01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.

For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)

The system has been powered up and biased successfully. To be monitored over the next few days.

A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.

A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...

The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.

Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently
  162   Tue Nov 24 13:10:42 2020 NHWaveforms with frame grounded
AIDA Frame is now grounded to platform (will add photo later)

Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now)
Attachment 1: waves_odd.png
waves_odd.png
Attachment 2: waves_even.png
waves_even.png
Attachment 3: 20201124_164416.jpg
20201124_164416.jpg
Attachment 4: 20201124_164413.jpg
20201124_164413.jpg
  161   Tue Oct 6 15:48:06 2020 NHnnrpi kernel update
FYI:

The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were *huge* and filled with the same error many times a second

Oct  4 15:01:22 raspberrypi kernel: [1030048.603846]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753]                                                                                             

A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version.
now the message doesn't seem to appear so hopefully the pis won't break so rapidly.
ELOG V3.1.3-7933898