ELOG AIDA

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

AIDA

Fields marked with * are required

Entry time:

Sat Jun 15 16:53:22 2019

Author*:

Subject*:

00.30 AIDA file R2_26
      DTAS run 007
      BigRIPS run 2006

00:55
      Merger status - attachment 1
      TapeSever status - attachment 2

01:40
      Before zeroing stats - attachment 3
      Histos up to now saved in /homes/npg/spectra/specs-2019Jun16-01.46.50
      Before zeroing histos - XY attachment 4

02:22  Up to AIDA R2_41
      Stats - attachment 5
      XY implants - attachment 6

02:29 AIDA R2_42 
      DTAS run 011
      BigRIPS still 2006

04:08 Up to AIDA R2_55
      Temps - attachment 7
      Biases - attachment 8
      Stats - attachment 9
      XY implants - attachment 10

04:50 Did some major UI updates to merger /data02/2019data/dtas/scripts/parallelscript/runall_xe19_dev.sh
         If no parameters are passed, it asks to select each option one-by-one, showing available options
             For DTAS, if there is only one run for a given header, it automatically selects it
         After that, it checks that all the raw files actually exist
         Finally, it composes the command, asks the user if it looks OK, and then launches the script (or quits)
         (For experts, it can still take the original case where all 7 parameters are passed at the command line)
         There may still be some minor bugs, so for now we don't replace the original one until Alvaro can check

06:09 Stats - attachment 11
      XY implant profiles -- attachment 12

06:30 DTAS 018; AIDA R2_80; BigRIPS 2009

Things getting a bit noisier?
      Merger - attachment 13
      TapeServer - attachment 14
      Stats - attachment 15

The FEEs corresponding to DSSSD1 are all showing factors 2 to 10 higher than just 30 min. ago

F11 rates still near 180 cps, primary beam still around 130 pnA.

06:43 Before I can diagnose the high noise problem in DSSSD1, it goes back to normal
      Merger - attachment 16
      TapeServer - attachment 17
      Stats - attachment 18
   Maybe the issue was from looking at the details too often :)

07:46 Shift change.

DAQ continues running OK.
      Statistics show  noisy DSSD #1 (100-200kHz) - see attach 19
      Tapeserver also shows higher than normal write to disk rate - see attach 20
      After a few seconds write to disk rate goes back to normal - see attach 21
      Data items rate looks OK (500k). This was checked after the rates in DSSD#1 went back to normal - see attach 22
      FEE64 temperatures OK - see attach 23
      Implants look OK - see attach 24

System-wide checks OK except nnaida7 fails ADC calibration as usual.

Reset Statistics
      Stats now OK - see attach 25

Reset HEC implants

Timewarp events appear to be very frequent, consistently every 3-5 seconds.
      This seems to have been happening ever since beam was put on target.
      Analysed R2_94 using analyser in Desktop 3. No timewarps reported. See attach 26.
      
08:32 AIDA: R2_100,
      DTAS: 21  ,
      BigRips: 2010

08:41 Another noise spike from DSSD#1
      Observed up to 1,600k data items per second - see attach 27
      Write to disk ~10 MB/s as before.
      See LEC xy plot - attach 28
      HEC xy plot shows no hints of issues - see attach 29
      
      Rate seems manageable - see attach 30 for analysis of noisy run R2_103. 
      Note few seconds deadtime in some FEE64 and unequal number of Pause/Resume in nnaida 22

09:45 Rate in DSSD#1 back to normal. Data rate 500k, Write to disk 5 Mb/s.

10:00 Started compressing R2_* files to save disk space.
      Used nice gzip to compress to avoid conflicts with the DAQ.
      Status of compression can be checked in ELOG desktop in the terminal window running ls -alth

10:32 DTAS and BigRIPS stop runs
      AIDA on file R2_128

DTAS:24
      BigRIPS 2011

10:56 Lost beam. BigRIPS and DTAS DAQs continue running. Accelerator operators advise beam should be back in 10 minutes.

11:12 Beam is back. BigRIPS & DTAS continue running and did not stop. AIDA file R2_133.

11:25 System-wide checks OK except nnaida7 fails ADC calibration as usual.
      Stats OK, but nnaida19 is a bit more noisy than befor (80k) - see attach 31
      Temperatures OK - see attach 32
      TapeServer & Merger OK - see attach 33, 34 - rates have been low since 10 am
      HEC xy plot OK - see attach 35

Reset online spectra

Leakage current OK - attach 36

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1:

merger0055.png Original size: 1240x413

Attachment 2:

tapeserver0055.png Original size: 1054x535

Attachment 3:

stats0140.png Original size: 1278x509

Attachment 4:

xy0140.png Original size: 679x446

Attachment 5:

stats0222.png Original size: 1274x504

Attachment 6:

xy0222.png Original size: 670x437

Attachment 7:

temps0408.png Original size: 1280x691

Attachment 8:

bias0408.png Original size: 1012x346

Attachment 9:

stats0408.png Original size: 1276x528

Attachment 10:

xy0408.png Original size: 672x449

Attachment 11:

stats0609.png Original size: 1271x520

Attachment 12:

xy0222.png Original size: 670x437

Attachment 13:

merger0637.png Original size: 1239x412

Attachment 14:

tapeserver0637.png Original size: 1272x759

Attachment 15:

stats0637.png Original size: 1276x528

Attachment 16:

merger0643.png Original size: 1244x421

Attachment 17:

tapeserver0643.png Original size: 1085x533

Attachment 18:

stats0643.png Original size: 1276x516

Attachment 19:

190616_0752_stats.png Original size: 1284x1030

Attachment 20:

190616_0753_Tape.png Original size: 1284x1027

Attachment 21:

190616_0754_Tape2.png Original size: 1284x1027

Attachment 22:

190616_0755_Merger.png Original size: 1284x1027

Attachment 23:

190616_0756_temp.png Original size: 1284x1030

Attachment 24:

190616_0757_bias.png Original size: 999x341

Attachment 25:

190616_0758_implants.png Original size: 691x577

Attachment 26:

190616_0759_stats2.png Original size: 1284x1030

Attachment 27:

R2_94

 *** MIDAS TDR format 3.3.0 AIDA data analyser - TD - May 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251091553 (  603659.0 Hz)
        Other data format:   10028447 (   24109.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         30 (       0.1 Hz)
                               RESUME:         30 (       0.1 Hz)
                              SYNC100:    3807848 (    9154.6 Hz)
                           FEE64 disc:    6208062 (   14925.0 Hz)
                             MBS info:      12477 (      30.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     572653 (    1376.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      415.949 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.024                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.587                0.000
 20                0.070                0.066
 21                0.000                0.000
 22                0.036                0.000
 23                0.000                0.000
 24                0.034                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100       Disc        MBS      Other   HEC Data
  0          0          0          0          0          0          0          0          0          0          0          0
  1   12553209     188405          0          0          0          0     158672      29733          0          0        335
  2    9601433     182979          0          0          0          0     158672      24307          0          0        156
  3   11738121     179145          0          0          0          0     158672      20473          0          0         87
  4    1088901     180520          0          0          0          0     158672      21848          0          0        449
  5    6322990     179870          0          0          0          0     158672      21198          0          0        136
  6    9919114     188320          0          0          0          0     158672      17171      12477          0        277
  7    2576661     178589          0          0          0          0     158672      19917          0          0        325
  8    2002508     178918          0          0          0          0     158672      20246          0          0        102
  9    9936580     464160          0          0          0          0     158672     305488          0          0      51043
 10    8496941     190655          0          0          0          0     158672      31983          0          0        951
 11   11073766     188418          0          0          0          0     158672      29746          0          0        613
 12   13232572     457668          0          0          0          0     158672     298996          0          0      40479
 13   11036172     182908          0          0          0          0     158672      24236          0          0        707
 14    6677032    1807261          0          0          2          2     158663    1648594          0          0      28471
 15    7592310     488424          0          0          0          0     158672     329752          0          0      46474
 16    5104957     186452          0          0          0          0     158672      27780          0          0        643
 17   11612948     570491          0          0          0          0     158672     411819          0          0      59387
 18    7175349     516449          0          0          0          0     158672     357777          0          0      79769
 19   31417088     743216          0          0         19         19     158450     584728          0          0      37903
 20   18695084     570477          0          0          4          4     158646     411823          0          0      37531
 21    9345111     392641          0          0          0          0     158673     233968          0          0      52476
 22    9240185     418883          0          0          1          1     158659     260222          0          0      46533
 23   17169454     625851          0          0          0          0     158673     467178          0          0      43272
 24   17483067     767747          0          0          4          4     158660     609079          0          0      44534
 25          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0

 *** MBS Timewarps
 FEE 
  0          0
  1          0
  2          0
  3          0
  4          0
  5          0
  6          0
  7          0
  8          0
  9          0
 10          0
 11          0
 12          0
 13          0
 14          0
 15          0
 16          0
 17          0
 18          0
 19          0
 20          0
 21          0
 22          0
 23          0
 24          0
 25          0
 26          0
 27          0
 28          0
 29          0
 30          0
 31          0
 32          0

 *** Program elapsed time:   29.195s ( 1096.066 blocks/s,  68.504 Mb/s)

Attachment 28:

190616_0842_stats.png Original size: 1284x1027

Attachment 29:

190616_0845_LECxy.png Original size: 691x577

Attachment 30:

190616_0847_HEC.png Original size: 691x577

Attachment 31:

R2_103

 *** MIDAS TDR format 3.3.0 AIDA data analyser - TD - May 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257061452 ( 1275563.6 Hz)
        Other data format:    4058548 (   20138.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         61 (       0.3 Hz)
                               RESUME:         60 (       0.3 Hz)
                              SYNC100:    1843403 (    9147.1 Hz)
                           FEE64 disc:    2208976 (   10961.2 Hz)
                             MBS info:       6048 (      30.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     190202 (     943.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      201.528 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.151                0.000
 20                2.928                3.028
 21                0.000                0.000
 22                1.209                1.219
 23                0.000                0.000
 24                0.004                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100       Disc        MBS      Other   HEC Data
  0          0          0          0          0          0          0          0          0          0          0          0
  1    6127628      89654          0          0          0          0      76877      12777          0          0        149
  2    4778507      87656          0          0          0          0      76877      10779          0          0         44
  3    5808021      85924          0          0          0          0      76877       9047          0          0         24
  4     533673      86272          0          0          0          0      76877       9395          0          0        175
  5    3081169      86334          0          0          0          0      76877       9457          0          0         46
  6    4802389      90453          0          0          0          0      76877       7528       6048          0         92
  7    1137246      84834          0          0          0          0      76877       7957          0          0        133
  8     826476      85528          0          0          0          0      76877       8651          0          0         33
  9    4678288     181147          0          0          0          0      76877     104270          0          0      16983
 10    4292661      90787          0          0          0          0      76877      13910          0          0        370
 11    5316878      89748          0          0          0          0      76877      12871          0          0        224
 12    5886804     182653          0          0          0          0      76877     105776          0          0      13953
 13    5519728      87376          0          0          0          0      76877      10499          0          0        243
 14    2828881     614027          0          0          0          0      76877     537150          0          0       9820
 15    3123050     186238          0          0          0          0      76877     109361          0          0      15213
 16    2334635      88561          0          0          0          0      76877      11684          0          0        231
 17   59000131     254863          0          0          0          0      76877     177986          0          0      18924
 18    3043680     198550          0          0          0          0      76877     121673          0          0      26458
 19   14469583     277114          0          0          5          5      76819     200285          0          0      12988
 20   25281563     241249          0          0         35         35      75757     165422          0          0      12624
 21    4614607     158784          0          0          0          0      76877      81907          0          0      17650
 22   28908380     171588          0          0         20         19      76412      95137          0          0      15373
 23   53275873     262419          0          0          0          0      76877     185542          0          0      13906
 24    7391601     276789          0          0          1          1      76875     199912          0          0      14546
 25          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0

 *** MBS Timewarps
 FEE 
  0          0
  1          0
  2          0
  3          0
  4          0
  5          0
  6          0
  7          0
  8          0
  9          0
 10          0
 11          0
 12          0
 13          0
 14          0
 15          0
 16          0
 17          0
 18          0
 19          0
 20          0
 21          0
 22          0
 23          0
 24          0
 25          0
 26          0
 27          0
 28          0
 29          0
 30          0
 31          0
 32          0

 *** Program elapsed time:   44.744s (  715.177 blocks/s,  44.699 Mb/s)

Attachment 32:

190616_1127_stats.png Original size: 1284x1030