ELOG AIDA

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

AIDA

Fields marked with * are required

Entry time:

Thu Nov 3 01:25:30 2016

Author*:

Subject*:

10.20 DAQ found stalled

DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3
      AIDA SYNC stats - see attachment 4 - stats 10.1M/381/3600 => DAQ stalled at 23.05 + 7.4h => c.06.29
      System wide checks OK *except* SYNC pulses & SYNC errors - see attachments 5-8

Conclude DAQ stalling at c. 06.30/18.30 due to start of ORNL LN2 dewar refill cycle
      but unrelated to either the BRIKEN-supplied 50MHz TTL clock signal to external clock
      input of Master MACB or the ground connection provided by this coaxial cable.

Which implies the problem is either EMI/RFI or ac mains electrical ground.

AIDA powered by two single phase outlets from one of four sections of a single phase ac mains distribution
      box with power supplied by two heavy duty (3cm dia) cables from Denkenseiki Noisecut AVR type PMN (AVR4, B3F
      104V/51A). AIDA is connected to both NEMA 5-15R-style outlets of section A, LN2 refill to section D, BRIKEN
      to sections B, C and D.

Ambient temperature +24.8 deg C, d.p. +4.1 deg C, RH 26.3%

15.00 DSSSD cabling drain wire attachment to AIDA adaptor PCB (FEE64 # 5 & 6) not tightly secured.
      CG re-secured drain wire and this has significantly reduced 100kHz period transient noise
      as observed at preamplifier outputs.

14.48 Restart DAQ (R110)
      Master MACB mode 5, 50MHz TTL clock to external clock input

14.57 DAQ stalled

All system wide checks OK *except* SYNC errors - see attachment 9
      ASIC clock timestamp - nnaida19 fails 
      N.B. SYNC pulses check OK, counters incrementing between successive tests
      The latter is a problem we have seen previously but has not stopped DAQ running OK

aidas1> ./analyser /TapeData/Sept2016/R110_2
 *** TDR format 3.3.0 analyser - TD - October 2016
 *** ERROR: READ I/O error:       5002
                   blocks:      11861
          ADC data format:   82556471 (  877427.0 Hz)
        Other data format:   14217428 (  151105.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         27 (       0.3 Hz)
                               RESUME:         27 (       0.3 Hz)
                              SYNC100:     861083 (    9151.8 Hz)
                           FEE64 disc:   13353468 (  141923.4 Hz)
                             MBS info:       2823 (      30.0 Hz)
                           Other info:          0 (       0.0 Hz)

ADC data range bit set:       1976 (      21.0 Hz)

Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

*** Timestamp elapsed time:       94.089 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.084                0.000
  2                0.076                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.109                0.000
  6                0.012                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.003                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.032                0.000
 16                0.013                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.125                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.142                0.149
 24                0.076                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

*** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100       Disc        MBS      Other  
HEC Data
  0          0          0          0          0          0          0          0          0          0          0  
       0
  1     942953    5806860          0          0          3          3      35855    5770999          0          0  
      42
  2    4036003    4268394          0          0          2          2      35861    4232529          0          0  
       0
  3    3038422      39132          0          0          0          0      35892       3240          0          0  
       0
  4    1192440      42063          0          0          0          0      35889       6174          0          0  
      68
  5    5685966      39310          0          0          5          5      35849       3451          0          0  
       0
  6    3670394      46605          0          0          2          2      35885       7893       2823          0  
      70
  7    2661022      43508          0          0          0          0      35890       7618          0          0  
     231
  8    2165335      41382          0          0          0          0      35890       5492          0          0  
     359
  9    3037520     863964          0          0          1          1      35889     828073          0          0  
       0
 10    2616647     179334          0          0          0          0      35889     143445          0          0  
       0
 11    1083633      39064          0          0          0          0      35889       3175          0          0  
       0
 12    1759629      38368          0          0          0          0      35888       2480          0          0  
       0
 13    2419715     280498          0          0          0          0      35891     244607          0          0  
       0
 14    3189205    1325559          0          0          0          0      35887    1289672          0          0  
     747
 15    4886680      39482          0          0          2          2      35871       3607          0          0  
       7
 16    5087977      38869          0          0          1          1      35886       2981          0          0  
       0
 17    2905638     329411          0          0          0          0      35888     293523          0          0  
       0
 18    3327386      46016          0          0          0          0      35888      10128          0          0  
       0
 19    6770177      40711          0          0          4          4      35842       4861          0          0  
       7
 20    2225755      38524          0          0          0          0      35891       2633          0          0  
     344
 21    2061728      40299          0          0          0          0      35887       4412          0          0  
      90
 22    3052329     512109          0          0          0          0      35890     476219          0          0  
       7
 23    9325689      38673          0          0          3          3      35834       2833          0          0  
       3
 24    5414228      39293          0          0          4          4      35862       3423          0          0  
       1
 25          0          0          0          0          0          0          0          0          0          0  
       0
 26          0          0          0          0          0          0          0          0          0          0  
       0
 27          0          0          0          0          0          0          0          0          0          0  
       0
 28          0          0          0          0          0          0          0          0          0          0  
       0
 29          0          0          0          0          0          0          0          0          0          0  
       0
 30          0          0          0          0          0          0          0          0          0          0  
       0
 31          0          0          0          0          0          0          0          0          0          0  
       0
 32          0          0          0          0          0          0          0          0          0          0  
       0

*** Timewarps
 FEE       ADC      Pause     Resume    SYNC100       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0

*** Program elapsed time:    1.500s ( 7907.333 blocks/s, 494.208 Mb/s)

15.48 Restart aidas1
      Power cycle FEE64s

Restart AIDA DAQ

Same problem - DAQ stall (no output from merger) at, or shortly after, Start ASIC Readout with
      multiple FEE64s with zero AIDA SYNC rate
      *All* system wide checks OK

Cannot recover DAQ control of FEE64s with merger restart - power cycle FEE64s again

23.58 Attempt to change master MACB from MACB module #11 mode 5 to MACB module #13 mode 0

Successive fuse failures of USB ac mains relay

System wide ASIC clock timestamp check nnaida 5/6/7/8 13/14/15/16 21/22/23/24 i.e. first half of MACB tree

Re-seat HDMI cabling from  master MACB root module to first half of MACB tree - ReSYNC fails - see attachments
10 & 11

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1: