ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Wed Mar 10 07:16:02 2021

Author*:

Subject*:

08:16 System wide checks all ok
      Statistics - attachment 1
      Temperatures - attachment 2
      bias - attachment 3
      Merger running 4.5e6 events per second
      Tape data writing at 42MB per second

Currently no beam

09:45 While there is no beam we will perform a longer test of the new merger.
      Run R20 stopped and merger changed to neew version
      Started R21.

The now usual behaviour of additional files at start of merge observed.
      N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was 
set to going
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18

TapeData rate of 13280 kB/sec
     No errors observed in ucesb
     Correlations seen between time machine in AIDA and Ge in online
     Pulser rate in the online makes sense
     With the current data rate remaining HDD space on current drive will last around 89 hours.

09.20 analysis of file R21_27
      ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data

10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
    (procedure "XAIDAAccessClient__Get" line 1)
    invoked from within
"XAIDAAccessClient__Get $Addr"

aida02 restarted itself during the reset process
     Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
     System wide checks following the restart all ok.
     When restarting the MBS relay errors observed until a timestamp was observed:
     Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000

10:55 Statistics - attachment 5
      Temperature - attachment 6
      Bias - attachment 7

11:03 Time machine correlation spectra with new merger
      AIDA - FATIMA - Attachment 8
      AIDA - Ge - Attachment 9

11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the 
layout to determine which FEE/ASIC caused the events before checking ASIC control

12:12 System wide checks all ok *except adc calibration which is same as before
      Statistics - attachment 10
      Temperature - attachment 11
      Bias and leakage currents ok - attachment 12

A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.

After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
      They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon 
cable from the VME scalers.
      Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of 
DSSD3.
    
12:50 We have waveforms for some FEES
      Layout 7 attachment -14
      Layout 8 - attachment 15

14.00 (LS)
      Still no beam
      System wide checks okay except same as before:
      **FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module**
      
      Statistics (attachment16)
      Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
      aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
      aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
 
      FEE Temps (attachment17)
      Leakage currents written to sheets(attachment18)
      Merger ~ 45M items/s
      TapeServer~ 14MB/s
     
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file 
      R22_98, rate spectra attached (attachment19)

14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw 
      no timewarps so we are not losing anything.
      Restarted MBS relay, errors have not reappeared so far

16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE Temps (attachment22)
      Leakage currents written to sheets(attachment23)
      Merger ~ 4.3M items/s
      TapeServer~ 14MB/s  
      Current file R22_148

16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from 
S452f115
      Around R22_150

18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
      Rates of R22_122 (attachment25):
      First DSSSD (fee0-fee3) ~66/s
      Second DSSSD (fee4-fee7) ~49/s
      Third DSSSD (fee8-fee11) ~22/s

Rates of R22_178 (attachment26):
      First DSSSD (fee0-fee3) ~122/s
      Second DSSSD (fee4-fee7) ~94/s
      Third DSSSD (fee8-fee11) ~49/s

18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment27)
      Rate spectra (attachment28)
      FEE Temps (attachment29)
      Leakage currents written to sheets(attachment30), look to be on the way down
      Merger ~ 4.5M items/s
      TapeServer~ 15MB/s

18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier

19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.

20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre 
reboot)
      We are borrowing a HDD from the SHIP group

20:30 System wide checks all ok
      Statistics - attachment 31
      Temperature - attachment 32
      Bias - attachment 33

22:10 MBS DAQ has crashed
      Merger terminal is now showing a large number of bad merge events

It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.

22:42 System wide checks all ok
      Statistics - attachment 34
      Temperature - Attachment 35
      Bias and leakage currents ok - attachment 36

23:33 System wide checks all ok - attachment 37
      Temperatures - attachment 38
      Bias and leakage currents ok - attachment 39

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1:

210310_0813_Stats.png Original size: 1907x448

Attachment 2:

210310_0813_Temp.png Original size: 1913x484

Attachment 3:

210310_0714_Bias.png Original size: 502x330

Attachment 4:

R21_27

 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259319480 ( 1687258.2 Hz)
        Other data format:    2600520 (   16920.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1272 (       8.3 Hz)
                               RESUME:       1272 (       8.3 Hz)
                              SYNC100:        573 (       3.7 Hz)
                              WR48-63:        573 (       3.7 Hz)
                           FEE64 disc:     302889 (    1970.7 Hz)
                             MBS info:    2293941 (   14925.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.693 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000              150.055
  2                0.129              153.277
  3                1.695              152.203
  4                0.000               97.174
  5               15.630              152.471
  6                0.533              122.407
  7                0.048              144.955
  8               17.221              151.934
  9               26.571              143.076
 10                0.063              148.982
 11               11.592              141.197
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20482535       1736          0          0        381        381         52         52          0        870          0          0
  1   26566394       1030          0          0          0          0         56         56          0        918          0          0
  2   26980761    1455064          0          0          2          2         87         87     302889    1151997          0          0
  3   26884821    1140284          0          0          5          5         65         65          0    1140144          0          0
  4    7516935         23          0          0          0          0          7          7          0          9          0          0
  5   16098094       1219          0          0        577        577         31         31          0          3          0          0
  6    9441595         40          0          0          2          2         18         18          0          0          0          0
  7    8769482         38          0          0          2          2         17         17          0          0          0          0
  8   32618578        336          0          0         98         98         70         70          0          0          0          0
  9   32790396        426          0          0        136        136         77         77          0          0          0          0
 10   16087963         82          0          0          1          1         40         40          0          0          0          0
 11   35081926        242          0          0         68         68         53         53          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:37059.496s (    0.863 blocks/s,   0.054 Mb/s)

Attachment 5:

210310_1054_Stats.png Original size: 1903x448

Attachment 6:

210310_1055_Temp.png Original size: 1909x480

Attachment 7:

210310_1055_Bias.png Original size: 501x325

Attachment 8:

AIDA_Fatima_Correlation.PNG Original size: 1322x747

Attachment 9:

AIDA_Ge_Correlation.PNG Original size: 1322x740

Attachment 10:

210310_1210_Stats.png Original size: 1906x442

Attachment 11:

210310_1211_Temp.png Original size: 1914x473

Attachment 12:

210310_1212_Bias.png Original size: 506x341

Attachment 13:

AIDA_leakage.PNG Original size: 2475x747

Attachment 14:

210310_1348_Layout7.png Original size: 1913x856

Attachment 15:

210310_1252_Layout8.png Original size: 1890x847

Attachment 16:

1400_Statistics.PNG Original size: 1749x405

Attachment 17:

1400_FEETemps.PNG Original size: 1749x447

Attachment 18:

1400_LeakageCurrents.PNG Original size: 460x318

Attachment 19:

1400_SpectraRate.PNG Original size: 1744x817

Attachment 20:

1600_Statistics.PNG Original size: 1749x411

Attachment 21:

1600_SpectraRate.PNG Original size: 1740x811

Attachment 22:

1600_FEETemps.PNG Original size: 1750x443

Attachment 23:

1600_LeakageCurrents.PNG Original size: 463x316

Attachment 24:

1640_SpectraRate.PNG Original size: 1738x817

Attachment 25:

1700_R22_122_Rates.PNG Original size: 1002x810

Attachment 26:

1700_R22_178_Rates.PNG Original size: 1017x823

Attachment 27:

1800_Statistics.PNG Original size: 1747x408

Attachment 28:

1800_SpectraRate.PNG Original size: 1738x815

Attachment 29:

1800_FEETemps.PNG Original size: 1750x443

Attachment 30:

1800_LeakageCurrents.PNG Original size: 456x317

Attachment 31:

210310_2039_Stats.png Original size: 1912x439

Attachment 32:

210310_2040_Temp.png Original size: 1917x475

Attachment 33:

210310_2041_Bias.png Original size: 503x322

Attachment 34:

210310_2241_Stats.png Original size: 1909x407

Attachment 35:

210310_2241_Temp.png Original size: 1914x469

Attachment 36:

210310_2242_Bias.png Original size: 496x326

Attachment 37:

210310_2331_Stats.png Original size: 1914x475

Attachment 38:

210310_2332_Temp.png Original size: 1910x483

Attachment 39:

210310_2333_Bias.png Original size: 501x327

Attachment 40:

Drop attachments here...

ELOG V3.1.4-unknown