Sat. 12/12/15

Working with a 8-FEE setup (as in https://elog.ph.ed.ac.uk/AIDA/133). We could open a file with the 
TapeServer. Merger also seems to be working (cursor spinning, and merge statistics ~ok). However the 
TapeServer doesn’t write anything to disk (just empty an file).

A few observations; I only took ‘systematic’ notes after we had reduced the number of FEEs to 8, but it's 
likely we were also facing the same problems before with a full system:

+ When enabling data transfer from NNAIDA modules (in Run Control page), it switches to the right mode for 
all FEEs except for NNAIDA5. NNAIDA5 is the master FEE. The merger also doesn’t show any stats for NNAIDA5 
(but stats look ok for all other FEEs). The spectra for NNAIDA5 look ok, so the FEE is generating data.

+ A few times we observe the following behavior: DAQ running without trouble. We start Merger+TapeServer, 
and when attempting some actions in RunControl (start-stop, or reset), we loose the ability to communicate 
with a number of FEEs. These FEEs can not perform a ReSYNC, and the only way to get them back to life is 
going through a power cycle. It seems like the TaperServer or Merger is somehow upsetting the FEEs or the 
DAQ. We can still ‘ping’ all FEEs, even after the ‘connect failed’ message. 

Attached are a few screenshots, and the text in the terminals for Merger and TapeServer. The following is 
the error returned by one FEE when we can’t ReSync them:


STATE for nnaida3 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 
-query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type 
text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout 
$timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
GET OPTION for nnaida3 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 60000 
-query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type 
text/xml -timeout 60000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout 
$timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetOption NetVar=NetVar.DACQ.Chistenable"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetOption $args"
    (procedure "DataAcquisitionControlClient__GetOption" line 1)
    invoked from within
"DataAcquisitionControlClient__GetOption "$AcqGetOption""