Friday 14th June 

Authors: TD, DK, CB

08:40

  elog appears to be down.  Relevant screenshots are in /tmp/june14/ so please upload when we can access

  Julabo chiller 75% M-H markers
  Room Temp +24.5 C deg, dew point +8.1 deg C, Room humidity 35.2%

  Pulser, ASIC settings as yesterday (copy/paste)

      ASIC settings 2019May19-17.54.03
      slow comparator 0xa

      BNC PB-4 Pulser 
      Amplitude 90,000
      x5 attenuator IN
      Frequency 25Hz range
      t_r 100ns, tau_d 50us
      - polarity
      Delay MIN, tail pulse

  DAQ continues OK
  Statistics as before.  nnaida19 counter appears to have overflowed and wrapped around
    See stats.png
  
  System-wide checks OK, except nnaida7 fails ADC calibration as before

  Biases and leak currents reasonable -- see biases.png

  FEE64 temperatures okay, see fee-temps.png

  Merger looks okay, ~600k items / sec; see merger.png

  TapeServer as before, around 4.5 Mbytes/sec; see tapeserver.png


13:52

   Pulser, ASIC settings same as this morning

   DAQ continues OK
   Statistics as before. See attach 190614_1354_stats. Counts RESET.

   System-wide checks OK, except nnaida7 fails ADC calibration as before

   Bias and leak currents OK. See attach 190614_1356_bias

   FEE64 temperatures OK, see attach 190614_1357_temps

   Merger keeps running at ~560k items /s; see 190614_1359_merger
   TapeServer as before. See 190614_tape

17:24 AIDA DAQ crashed by DTAS touching the clock function generator
      Function generator set at 50MHz square high +5V low 0V

18:38 problem loading ASIC settings - mismatch in slow comparator threshold and shaping time
      performed save/restore with 2019Jun13-08.56.35 - all looks ok now
      
      System wide checks ok except nnaida7 fails adc calibration

      FEE64 temperatures ok - attachment 12

      good event statistics ok - attachment 13

      bias/leakage currents ok - attachment 14

     
18:38 for reals

      DAQ Stopped
      R30_0 opened writing to disk
      
      Was to test DTAS sync whilst writing to file
18:43 DAQ stopped on file R30_0

18:45 DAQ restarted on file R31 with no writing to disk

21:50 DAQ found crashed (Valencia group believe it crashed around 20:00)

      Trying to reset FEEs resulted in an error message attachment 15 and 16

      Each FEE was rebooted using telnet to avoid a power cycle
      System reset after reboot

      System wide checks ok
      Check ASIC control 

      FEE temps ok

      DAQ Started again
      On startup FEEs 17, 20, 22 and 23 were all about 150k
      All relate to DSSD0
      Pressed on kapton and noise went down.

      Statistics now good

22.45 Review /var/log/messages and observe nnaida15 rebooted at 19.27 - reason unknown
      This probably explains why DAQ stopped.

Jun 14 19:27:32 aidas1 dhcpd: DHCPDISCOVER from 00:04:a3:2a:f3:ac via p2p2
Jun 14 19:27:32 aidas1 dhcpd: DHCPOFFER on 10.1.2.15 to 00:04:a3:2a:f3:ac via p2p2
Jun 14 19:27:32 aidas1 dhcpd: DHCPREQUEST for 10.1.2.15 (10.1.2.250) from 00:04:a3:2a:f3:ac via p2p2
Jun 14 19:27:32 aidas1 dhcpd: DHCPACK on 10.1.2.15 to 00:04:a3:2a:f3:ac via p2p2
Jun 14 19:27:32 aidas1 rpc.mountd[2868]: authenticated mount request from 10.1.2.15:784 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida15 (/data10/XilinxLinux/ppc_4xx/rfs)
Jun 14 19:27:59 aidas1 rpc.mountd[2868]: authenticated mount request from 10.1.2.15:864 for /MIDAS@aidas
(/MIDAS@aidas)


22.53 ADC data synchronised - attachment 17

23.05 zero DAQ statistics
      zero all histograms
      

23.20 Check ASIC Control x1 
      FEE64 Rate spectra - attachment 18

      1.8.L spectra - see attachments 19 & 20
      nnaida1 81 ch FWHM, nnaida3 77 ch FWHM

      p+n FEE64s 1.8.W spectra 20us FSR - attachment 21
      n+n FEE64s 1.8.W spectra 20us FSR - attachment 22

23.36 DAQ start file May19/R34_0

      DTAS+AIDA correlation test using gamma ray sources

      Correlation scaler reset issued during R34_0

      ASIC settings 2019Jun13-08.56.35

      BNC PB-4 Pulser 
      Amplitude 90,000
      x5 attenuator IN
      Frequency 25Hz range
      t_r 100ns, tau_d 50us
      - polarity
      Delay MIN, tail pulse

23.53 DAQ stop file May19/R34_2

      nnaida15 did not stop - state unknown - reboot nnaida15

      This did not work - unable to reset nnaida15

      Closed all FireFox broweser tabs, stopped all servers and restarted

00.14 DAQ start

      ASIC settings 2019Jun13-08.56.35

      BNC PB-4 Pulser 
      Amplitude 90,000
      x5 attenuator IN
      Frequency 25Hz range
      t_r 100ns, tau_d 50us
      - polarity
      Delay MIN, tail pulse

STATE for nnaida1 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida1:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida1:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida2 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida2:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida2:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida3 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida4 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida4:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida4:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida5 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida5:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida5:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida6 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida6:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida6:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida7 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida7:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida7:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
STATE for nnaida8 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida8:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida8:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout $timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControl