Sat. 12/12/15

Working with a 8-FEE setup (as in https://elog.ph.ed.ac.uk/AIDA/133). We could open a file with the 
TapeServer. Merger also seems to be working (cursor spinning, and merge statistics ~ok). However the 
TapeServer doesn’t write anything to disk (just empty an file).

A few observations; I only took ‘systematic’ notes after we had reduced the number of FEEs to 8, but it's 
likely we were also facing the same problems before with a full system:

+ When enabling data transfer from NNAIDA modules (in Run Control page), it switches to the right mode for 
all FEEs except for NNAIDA5. NNAIDA5 is the master FEE. The merger also doesn’t show any stats for NNAIDA5 
(but stats look ok for all other FEEs). The spectra for NNAIDA5 look ok, so the FEE is generating data.

+ A few times we observe the following behavior: DAQ running without trouble. We start Merger+TapeServer, 
and when attempting some actions in RunControl (start-stop, or reset), we loose the ability to communicate 
with a number of FEEs. These FEEs can not perform a ReSYNC, and the only way to get them back to life is 
going through a power cycle. It seems like the TaperServer or Merger is somehow upsetting the FEEs or the 
DAQ. We can still ‘ping’ all FEEs, even after the ‘connect failed’ message. 

Attached are a few screenshots, and the text in the terminals for Merger and TapeServer. The following is 
the error returned by one FEE when we can’t ReSync them:


STATE for nnaida3 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 20000 
-query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type 
text/xml -timeout 20000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout 
$timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
GET OPTION for nnaida3 returned with an error
connect failed 
NONE
connect failed 
    while executing
"::http::geturl http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type text/xml -timeout 60000 
-query {<?xml version="1.0" encoding="UTF-8"..."
    ("eval" body line 1)
    invoked from within
"eval [list ::http::geturl $url] $args"
    (procedure "::http::geturl_followRedirects" line 4)
    invoked from within
"::http::geturl_followRedirects http://nnaida3:8015/DataAcquisitionControlServer -headers {} -type 
text/xml -timeout 60000 -query {<?xml version="1.0" ..."
    ("eval" body line 1)
    invoked from within
"eval ::http::geturl_followRedirects [list $url]  -headers [list $local_headers]  -type text/xml  -timeout 
$timeout  -query [list $request]  $local_pro..."
    (procedure "::SOAP::Transport::http::xfer" line 61)
    invoked from within
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetOption NetVar=NetVar.DACQ.Chistenable"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetOption $args"
    (procedure "DataAcquisitionControlClient__GetOption" line 1)
    invoked from within
"DataAcquisitionControlClient__GetOption "$AcqGetOption""

=> "Httpd for Merger/Tape Server" terminal:



System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-504.8.1.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS@aidas and MIDAS_LIBRARY = /MIDAS@aidas/TclHttpd/Linux64
PATH = /MIDAS@aidas/bin_Linux64:/MIDAS@aidas/TclHttpd/Linux64:/homes/npg/root/bin:/homes/npg/root/bin:/homes/npg/root/lib:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "aegc.ur9z+3w"
httpd started on port 8115

Custom startup from /MIDAS@aidas/config/TclHttpd/aidas1@8115/startup.tcl
/TapeServer
/DataBaseAccessServer
Loaded MemSasAccess
/SpectrumService
/NetVarService
/SigTaskService
loading tcl/MERGERunControl.tcl for namespace ::
/DataAcquisitionControlServer
DefineMessage unknown
Run Control Server Implementation for MERGE
RunControlServer loaded
loading Html/RunControl/implementation.tcl for namespace ::
RunControl loaded
loading Html/MERGE/RunControl/implementation.tcl for namespace ::
ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
created SigTask registers 0 0  0 0
Created UI registers
MERGE RunControl loaded
Completed custom startup from /MIDAS@aidas/TclHttpd/Html/MERGE/RunControl/stats.defn.tcl
Shared memory area located at 0x7f6b51f67000
Tape Server comms table located at 0x7f6b51f67000
setup Merge
DataAcqCommand: 1 Halt
Halt (1)
DataAcqCommand: 8 Unmapping spectra
Unmapping spectra (1)
DataAcqCommand: 3 Setup
Setup (1)
go Merge
DataAcqCommand: 3 connect xfer stream
connect xfer stream (1)
DataAcqCommand: 2 Go
Go (1)
DataAcqCommand: 16 MergeStart
MergeStart (1)












=> "Merger" terminal:
.
.
.
MERGE Data Link (25178): Started with args 27 11000 11028
MERGE Data Link (25170): MIDAS MERGE Data Link Build Aug 12 2014
MERGE Data Link (25178): Configuration: index = 27, SHM key=11000, TCP port = 11028
MERGE Data Link (25178): File mapped object /SHM_11000 accessed
MERGE Data Link (25178): Shared memory ID is 3
MERGE Data Link (25178): Memory mapping 402732 bytes
MERGE Data Link (25178): Shared memory segment located at address 0x7f9815f0a000.
MERGE Data Link (25178): File mapped object /SHM_11001 accessed
MERGE Data Link (25178): Shared memory ID is 3
MERGE Data Link (25178): Memory mapping 1073741824 bytes
MERGE Data Link (25178): Shared memory segment located at address 0x7f97d5f0a000.
MERGE Data Link (25172): Message Initialisation failed.
MERGE Data Link (25172): Message logger not contacted.
MERGE Data Link (25181): Message Initialisation failed.
MERGE Data Link (25181): Message logger not contacted.
MERGE Data Link (25170): Started with args 19 11000 11020
MERGE Data Link (25170): Configuration: index = 19, SHM key=11000, TCP port = 11020
MERGE Data Link (25170): File mapped object /SHM_11000 accessed
MERGE Data Link (25170): Shared memory ID is 3
MERGE Data Link (25170): Memory mapping 402732 bytes
MERGE Data Link (25170): Shared memory segment located at address 0x7f5e73f1b000.
MERGE Data Link (25170): File mapped object /SHM_11001 accessed
MERGE Data Link (25170): Shared memory ID is 3
MERGE Data Link (25170): Memory mapping 1073741824 bytes
MERGE Data Link (25170): Shared memory segment located at address 0x7f5e33f1b000.
MRGE Stats created and initialised
MERGE Data Link (25153): Starting the network interface
MERGE Data Link (25153): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25153): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25153): MERGE Data Link using TCP port 11003.
MERGE Data Link (25153): Entering server loop
MERGE Data Link (25153): listening on port 11003
MERGE Data Link (25180): Started with args 29 11000 11030
MERGE Data Link (25180): Configuration: index = 29, SHM key=11000, TCP port = 11030
MERGE Data Link (25180): File mapped object /SHM_11000 accessed
MERGE Data Link (25180): Shared memory ID is 3
MERGE Data Link (25180): Memory mapping 402732 bytes
MERGE Data Link (25180): Shared memory segment located at address 0x7fdf8abed000.
MERGE Data Link (25180): File mapped object /SHM_11001 accessed
MERGE Data Link (25180): Shared memory ID is 3
MERGE Data Link (25180): Memory mapping 1073741824 bytes
MERGE Data Link (25180): Shared memory segment located at address 0x7fdf4abed000.
MERGE Data Link (25172): MIDAS MERGE Data Link Build Aug 12 2014
MERGE Program (25183): Message Initialisation failed.
MERGE Program (25183): Message logger not contacted.
MERGE Program (25183): MIDAS MERGE Program (64-bit) Build Sep  3 2014
MERGE Program (25183): Started with args 11000
MERGE Program (25183): Configuration: SHM key=11000
MERGE Program (25183): File mapped object /SHM_11000 accessed
MERGE Program (25183): Shared memory ID is 3
MERGE Program (25183): Memory mapping 402732 bytes
MERGE Program (25183): Shared memory segment located at address 0x7f520d167000.
MERGE Program (25183): File mapped object /SHM_11001 accessed
MERGE Program (25183): Shared memory ID is 3
MERGE Program (25183): Memory mapping 1073741824 bytes
MERGE Program (25183): Shared memory segment located at address 0x7f51cd167000.
Creating NetVars #2
MERGE Data Link (25181): MIDAS MERGE Data Link Build Aug 12 2014
MERGE Data Link (25172): Started with args 21 11000 11022
MERGE Data Link (25172): Configuration: index = 21, SHM key=11000, TCP port = 11022
MERGE Data Link (25172): File mapped object /SHM_11000 accessed
MERGE Data Link (25172): Shared memory ID is 3
MERGE Data Link (25172): Memory mapping 402732 bytes
MERGE Data Link (25172): Shared memory segment located at address 0x7ff8a4728000.
MERGE Data Link (25172): File mapped object /SHM_11001 accessed
MERGE Data Link (25172): Shared memory ID is 3
MERGE Data Link (25172): Memory mapping 1073741824 bytes
MERGE Data Link (25172): Shared memory segment located at address 0x7ff864728000.
Output buffer length = 65504; format option = 4; transfer option = 3
NetVars created and initialised
MRGE Stats created and initialised
MERGE Data Link (25157): Starting the network interface
MERGE Data Link (25157): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25157): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25157): MERGE Data Link using TCP port 11007.
MERGE Data Link (25157): Entering server loop
MERGE Data Link (25157): listening on port 11007
MRGE Stats created and initialised
MERGE Data Link (25160): Starting the network interface
MERGE Data Link (25160): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25160): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25160): MERGE Data Link using TCP port 11010.
MERGE Data Link (25160): Entering server loop
MERGE Data Link (25160): listening on port 11010
MERGE Data Link (25181): Started with args 30 11000 11031
MERGE Data Link (25181): Configuration: index = 30, SHM key=11000, TCP port = 11031
MERGE Data Link (25182): Message Initialisation failed.
MERGE Data Link (25181): File mapped object /SHM_11000 accessed
MERGE Data Link (25182): Message logger not contacted.
MERGE Data Link (25181): Shared memory ID is 3
MERGE Data Link (25181): Memory mapping 402732 bytes
MERGE Data Link (25181): Shared memory segment located at address 0x7ffbe639a000.
MERGE Data Link (25181): File mapped object /SHM_11001 accessed
MERGE Data Link (25181): Shared memory ID is 3
MERGE Data Link (25181): Memory mapping 1073741824 bytes
MERGE Data Link (25181): Shared memory segment located at address 0x7ffba639a000.
MRGE Stats created and initialised
MERGE Data Link (25154): Starting the network interface
MERGE Data Link (25154): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25154): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25154): MERGE Data Link using TCP port 11004.
MERGE Data Link (25154): Entering server loop
MERGE Data Link (25154): listening on port 11004
MERGE Data Link (25182): MIDAS MERGE Data Link Build Aug 12 2014
MERGE Data Link (25182): Started with args 31 11000 11032
MERGE Data Link (25182): Configuration: index = 31, SHM key=11000, TCP port = 11032
MERGE Data Link (25182): File mapped object /SHM_11000 accessed
MERGE Data Link (25182): Shared memory ID is 3
MERGE Data Link (25182): Memory mapping 402732 bytes
MERGE Data Link (25182): Shared memory segment located at address 0x7f2495805000.
MERGE Data Link (25182): File mapped object /SHM_11001 accessed
MERGE Data Link (25182): Shared memory ID is 3
MERGE Data Link (25182): Memory mapping 1073741824 bytes
MERGE Data Link (25182): Shared memory segment located at address 0x7f2455805000.
MRGE Stats created and initialised
MERGE Data Link (25167): Starting the network interface
MERGE Data Link (25167): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25167): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25167): MERGE Data Link using TCP port 11017.
MERGE Data Link (25167): Entering server loop
MERGE Data Link (25167): listening on port 11017
MRGE Stats created and initialised
MERGE Data Link (25152): Starting the network interface
MERGE Data Link (25152): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25152): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25152): MERGE Data Link using TCP port 11002.
MERGE Data Link (25152): Entering server loop
MERGE Data Link (25152): listening on port 11002
MRGE Stats created and initialised
MERGE Data Link (25169): Starting the network interface
MERGE Data Link (25169): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25169): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25169): MERGE Data Link using TCP port 11019.
MERGE Data Link (25169): Entering server loop
MERGE Data Link (25169): listening on port 11019
MRGE Stats created and initialised
MERGE Data Link (25158): Starting the network interface
MERGE Data Link (25158): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25158): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25158): MERGE Data Link using TCP port 11008.
MERGE Data Link (25158): Entering server loop
MERGE Data Link (25158): listening on port 11008
MRGE Stats created and initialised
MERGE Data Link (25166): Starting the network interface
MERGE Data Link (25166): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25166): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25166): MERGE Data Link using TCP port 11016.
MERGE Data Link (25166): Entering server loop
MERGE Data Link (25166): listening on port 11016
MRGE Stats created and initialised
MERGE Data Link (25174): Starting the network interface
MERGE Data Link (25174): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25174): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25174): MERGE Data Link using TCP port 11024.
MERGE Data Link (25174): Entering server loop
MERGE Data Link (25174): listening on port 11024
MRGE Stats created and initialised
MERGE Data Link (25175): Starting the network interface
MERGE Data Link (25175): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25175): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25175): MERGE Data Link using TCP port 11025.
MERGE Data Link (25175): Entering server loop
MERGE Data Link (25175): listening on port 11025
MRGE Stats created and initialised
MERGE Data Link (25180): Starting the network interface
MERGE Data Link (25180): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25180): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25180): MERGE Data Link using TCP port 11030.
MERGE Data Link (25180): Entering server loop
MERGE Data Link (25180): listening on port 11030
MRGE Stats created and initialised
MERGE Data Link (25179): Starting the network interface
MERGE Data Link (25179): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25179): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25179): MERGE Data Link using TCP port 11029.
MERGE Data Link (25179): Entering server loop
MERGE Data Link (25179): listening on port 11029
MRGE Stats created and initialised
MERGE Data Link (25178): Starting the network interface
MERGE Data Link (25178): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25178): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25178): MERGE Data Link using TCP port 11028.
MERGE Data Link (25178): Entering server loop
MERGE Data Link (25178): listening on port 11028
MRGE Stats created and initialised
MERGE Data Link (25161): Starting the network interface
MERGE Data Link (25161): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25161): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25161): MERGE Data Link using TCP port 11011.
MERGE Data Link (25161): Entering server loop
MERGE Data Link (25161): listening on port 11011
MRGE Stats created and initialised
MERGE Data Link (25151): Starting the network interface
MERGE Data Link (25151): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25151): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25151): MERGE Data Link using TCP port 11001.
MERGE Data Link (25151): Entering server loop
MERGE Data Link (25151): listening on port 11001
MRGE Stats created and initialised
MERGE Data Link (25182): Starting the network interface
MERGE Data Link (25182): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25182): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25182): MERGE Data Link using TCP port 11032.
MERGE Data Link (25182): Entering server loop
MERGE Data Link (25182): listening on port 11032
MRGE Stats created and initialised
MERGE Data Link (25181): Starting the network interface
MERGE Data Link (25181): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25181): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25181): MERGE Data Link using TCP port 11031.
MERGE Data Link (25181): Entering server loop
MERGE Data Link (25181): listening on port 11031
MRGE Stats created and initialised
MERGE Data Link (25156): Starting the network interface
MERGE Data Link (25156): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25156): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25156): MERGE Data Link using TCP port 11006.
MERGE Data Link (25156): Entering server loop
MERGE Data Link (25156): listening on port 11006
MRGE Stats created and initialised
MERGE Data Link (25168): Starting the network interface
MERGE Data Link (25168): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25168): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25168): MERGE Data Link using TCP port 11018.
MERGE Data Link (25168): Entering server loop
MERGE Data Link (25168): listening on port 11018
MRGE Stats created and initialised
MERGE Data Link (25173): Starting the network interface
MERGE Data Link (25173): TCP socket receive buffer was 87380 - now 249856
MERGE Data Link (25173): TCP socket send buffer was 16384 - now 249856
MERGE Data Link (25173): MERGE Data Link using TCP port 11023.
MERGE Data Link (25173): Entering server loop