AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 23 of 46  ELOG logo
Entry  Mon Jul 18 12:00:16 2016, Patrick Coleman-Smith, Report of remote operation of Aida at RIKEN 11/7 to 14/7 

11th July ---------------------------------------------------------------------------------------------------------

09:32 UK time, logged in and power-up all FEEs OK. Setup fine, Temperature Max 12@64

System Checks all fine SYNC count 2243584.

Start Merger and Toggle Pause for no merging.

DAQ start with output enabled.

Toggle Pause in Merger -> “Want First SYNC” with no change.

Tried this several times with the same result.

Checked DAQ statistics => Buffers/sec = 50 to 65, SYNC/sec = 40 to 80  ( should be 384 )

Disabled all ASIC readout and disable all discriminators. DAQ still in GO state.

Fetched new version of ASIC4 with Enable/Disable ASIC readout commands in the “Expert” menu.

SYNC statics now all FEEs are 384/sec.

Merger … Toggle Pause and its fine. 334 items/sec.

DAQ : ASIC4 : Set all slow comparator Thresholds to 250 : Enable ASICs for readout.

Merger now 532811 items/sec.

DAQ SYNC statistics range 307 to 390/sec

LT 22:36 System Checks => Sync counts 6518784 all ok, Sync errors all passed.

LT 23:30 Merger 539826 items/sec. Temperatures => 12@68.12 ; SYNC statistics 361 to 385 ; SYNC counts 8099840 all ok ; Sync errors all ok ; ASIC clock timestamp all ok.

STOP then Power-down all fine.

13th July ----------------------------------------------------------------------------------------------------------------------

Log in and power-up fine.

Backup Merger file ( merge.AD )  and copy latest version from DL. Won’t copy as file already open.

So instead …. Disable ASIC readout, disable Discriminators and run the system.

When Merger is operating , Enable ASIC readout => Merger rate is 8,058,412 items/sec.

LT 19:44 => Temperatures max 12@68.81 , SYNC received 845824 all ok; Sync errors all ok.

LT 20:11 => Merge rate 7,991,198 items/sec, SYNC received 1551359 all ok; Sync errors all ok.

LT 21:07 => Temperatures Max 12@69.00 ; SYNC received 2742272 all.

LT 23:20 => Stopped the whole system.

Kill the Merge process and the Startup process. Copy new Merge.AD successful. Edit Merger startup script to include netint OutputAll 1.

Start up Merger using /MIDAS/Merger/AIDA/Startup in terminal connected to aidas1.

Merger : Setup and configure ; Go ; Toggle Pause All Ok.

Go DAQ   - disable all ASIC readout.

Merger : Toggle Pause ; Merging 8832 items/sec ( = 384 x 23 …. OK )

Connect to TapeServer setup for R3 into /TapeData/July2016 , Go Tape ok.

Merger : Toggle Transfer.

LT 00:08 => enable ASICS for readout. Run for a short while. Merge spinner hesitates for 2 seconds at a time. Merge rate 5,951,471 items/sec. Tape server rate 3,5148 Kbytes/sec.

Stop system, SYNC received 8085504 all ok ; sync error all ok. Power down. Sftp tape data files to nndhcp052 in Daresbury.

14th July -------------------------------------------------------------------------------------------------------------------------

Startup and run the system to tape again R4, ASICs with no discriminators. All fine.

LT 20:19 Enable Discriminators. Merger rate 3.9Mi/sec ( Mega items/second)

LT 20:57 Temperature Max 12@69 , Merge rate 4.2Mi/sec

UK 14:49 Network failure between Daresbury and Edinburgh. Lost contact repeatedly 4 times in a row. Contacted DL network support. No local networking faults.

LT 23:05 Re-connected to Aida in RIKEN. Merger 3.3Mi/sec.

Decided to stop due to concerns about flaky network and leaving the system powered up over the weekend.

SYNC received 7222272 all ok; sync error ok.

Power down. R4_244 is the last file. Written.

 

Entry  Wed May 29 06:53:56 2019, TD, Report - medium - unable to monitor FEE64 logs using pi_monitor 10.png11.png12.png
Unable to monitor FEE64 logs using pi_monitor using instructions https://elog.ph.ed.ac.uk/AIDA/559

At first load of URL http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml I obtain error messages - attachment 1

At reload of URL I obtain the expected page and 
Update USB Terminal List and Connect USB Terminals operate as expected - attachment  2

Selecting Parse USB Log Files for Details produces (after c. 60s) a connection reset error from the browser -
see attachment 3


pi@npg-monitor /tmp $ ls
log8015_19.05.28  log8015_counter  log8015_error  tcl2156  tclhttpd2156.default
pi@npg-monitor /tmp $ cat log8015_error
 [28/May/2019:09:18:01] sock8 error writing "sock8": broken pipe  /css/buttons.css
 [28/May/2019:09:18:11] sock8 DocSubstSystemFile {could not read "/MIDAS/TclHttpd/Html/error.html": no such file
or directory} /AIDA/Pi_Monitor/Pi_Monitor.tml
 [28/May/2019:09:18:11] sock8 Error 500 /AIDA/Pi_Monitor/Pi_Monitor.tml {can't read "JS": no such variable
    while executing
"if {$JS != ""} {
       append JS "\");"
       set html "<script>[set JS]</script>"
    }"
    (in namespace eval "::Pi_Monitor" script line 2)
    invoked from within
"namespace eval Pi_Monitor {
    if {$JS != ""} {
       append JS "\");"
       set html "<script>[set JS]</script>"
    }
}"
    invoked from within
"subst \n[Doc_Dynamic\]\n\n[global\ env\;\ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ Html\
settings.tml\]\]\n\n<!\n[global\ Httpd\;\ upvar\ ..."
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list subst $script]"
    (procedure "SubstFile" line 12)
    invoked from within
"SubstFile /MIDAS/TclHttpd/Html/AIDA/Pi_Monitor/Pi_Monitor.tml {}"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 [list SubstFile $path $interp]"
    (procedure "Subst_File" line 7)
    invoked from within
"Subst_File $template $interp"
    invoked from within
"TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)"
    (procedure "Doc_application/x-tcl-template" line 9)
    invoked from within
"$cmd $path $suffix $sock"
    (procedure "Doc_Handle" line 20)
    invoked from within
"Doc_Handle $prefix $path $suffix $sock"
    (procedure "DocDomain" line 44)
    invoked from within
"DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Pi_Monitor/Pi_Monitor.tml"
    ("eval" body line 1)
    invoked from within
"eval $Url(command,$prefix) [list $sock $suffix]"} /AIDA/Pi_Monitor/Pi_Monitor.tml
pi@npg-monitor /tmp $ cat log8015_19.05.28
10.1.1.250 - - [28/May/2019:09:18:01 UTC] "GET / HTTP/1.1" 200 - - "Mozilla/5.0 (X11; Linux x86_64; rv:60.0)
Gecko/20100101 Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:01 UTC] "GET /js/Common.js HTTP/1.1" 200 3828 "http://nnrpi2:8015/"
"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:11 UTC] "GET /AIDA/Pi_Monitor/Pi_Monitor.tml HTTP/1.1" 500 - - "Mozilla/5.0
(X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /AIDA/Pi_Monitor/Pi_Monitor.tml HTTP/1.1" 200 - - "Mozilla/5.0
(X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /AIDA/Pi_Monitor/Pi_Monitor.js HTTP/1.1" 200 542
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /js/Common.js HTTP/1.1" 304 3828
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /images/Space.gif HTTP/1.1" 200 -
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /js/FormChek.js HTTP/1.1" 304 60521
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /images/midas.gif HTTP/1.1" 200 2558
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /images/Space.gif HTTP/1.1" 200 -
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:30 UTC] "GET /images/tclp.gif HTTP/1.1" 200 1294
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:32 UTC] "POST /AIDA/Pi_Monitor/Pi_Monitor.tml HTTP/1.1" 200 -
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:32 UTC] "GET /images/Space.gif HTTP/1.1" 200 -
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:39 UTC] "POST /AIDA/Pi_Monitor/Pi_Monitor.tml HTTP/1.1" 200 -
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
10.1.1.250 - - [28/May/2019:09:18:39 UTC] "GET /images/Space.gif HTTP/1.1" 200 -
"http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Firefox/60.0"
pi@npg-monitor /tmp $ cat log8015_counter



set ::counter::H-urlhits(0) 0
set ::counter::H-urlhits(1) 0
set ::counter::H-urlhits(10) 0
set ::counter::H-urlhits(11) 0
set ::counter::H-urlhits(12) 0
set ::counter::H-urlhits(13) 0
set ::counter::H-urlhits(14) 0
set ::counter::H-urlhits(15) 0
set ::counter::H-urlhits(16) 0
set ::counter::H-urlhits(17) 0
set ::counter::H-urlhits(18) 0
set ::counter::H-urlhits(19) 0
set ::counter::H-urlhits(2) 0
set ::counter::H-urlhits(20) 0
set ::counter::H-urlhits(21) 0
set ::counter::H-urlhits(22) 0
set ::counter::H-urlhits(23) 0
set ::counter::H-urlhits(24) 0
set ::counter::H-urlhits(25) 0
set ::counter::H-urlhits(26) 0
set ::counter::H-urlhits(27) 0
set ::counter::H-urlhits(28) 0
set ::counter::H-urlhits(29) 0
set ::counter::H-urlhits(3) 0
set ::counter::H-urlhits(30) 0
set ::counter::H-urlhits(31) 0
set ::counter::H-urlhits(32) 0
set ::counter::H-urlhits(33) 0
set ::counter::H-urlhits(34) 0
set ::counter::H-urlhits(35) 0
set ::counter::H-urlhits(36) 0
set ::counter::H-urlhits(37) 0
set ::counter::H-urlhits(38) 0
set ::counter::H-urlhits(39) 0
set ::counter::H-urlhits(4) 0
set ::counter::H-urlhits(40) 0
set ::counter::H-urlhits(41) 0
set ::counter::H-urlhits(42) 0
set ::counter::H-urlhits(43) 0
set ::counter::H-urlhits(44) 0
set ::counter::H-urlhits(45) 0
set ::counter::H-urlhits(46) 0
set ::counter::H-urlhits(47) 0
set ::counter::H-urlhits(48) 0
set ::counter::H-urlhits(49) 0
set ::counter::H-urlhits(5) 0
set ::counter::H-urlhits(50) 0
set ::counter::H-urlhits(51) 0
set ::counter::H-urlhits(52) 0
set ::counter::H-urlhits(53) 0
set ::counter::H-urlhits(54) 0
set ::counter::H-urlhits(55) 0
set ::counter::H-urlhits(56) 0
set ::counter::H-urlhits(57) 0
set ::counter::H-urlhits(58) 0
set ::counter::H-urlhits(59) 0
set ::counter::H-urlhits(6) 0
set ::counter::H-urlhits(7) 0
set ::counter::H-urlhits(8) 0
set ::counter::H-urlhits(9) 0


set ::counter::Hour-urlhits(0) 0
set ::counter::Hour-urlhits(1) 0
set ::counter::Hour-urlhits(10) 0
set ::counter::Hour-urlhits(11) 0
set ::counter::Hour-urlhits(12) 0
set ::counter::Hour-urlhits(13) 0
set ::counter::Hour-urlhits(14) 0
set ::counter::Hour-urlhits(15) 0
set ::counter::Hour-urlhits(16) 0
set ::counter::Hour-urlhits(17) 0
set ::counter::Hour-urlhits(18) 0
set ::counter::Hour-urlhits(19) 0
set ::counter::Hour-urlhits(2) 0
set ::counter::Hour-urlhits(20) 0
set ::counter::Hour-urlhits(21) 0
set ::counter::Hour-urlhits(22) 0
set ::counter::Hour-urlhits(23) 0
set ::counter::Hour-urlhits(3) 0
set ::counter::Hour-urlhits(4) 0
set ::counter::Hour-urlhits(5) 0
set ::counter::Hour-urlhits(6) 0
set ::counter::Hour-urlhits(7) 0
set ::counter::Hour-urlhits(8) 0
set ::counter::Hour-urlhits(9) 0


set ::counter::Day-urlhits(0) 0

Before selecting Parse USB ...

pi@npg-monitor /tmp $ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 09:16 ?        00:00:01 init [2]  
root         2     0  0 09:16 ?        00:00:00 [kthreadd]
root         3     2  0 09:16 ?        00:00:00 [ksoftirqd/0]
root         5     2  0 09:16 ?        00:00:00 [kworker/0:0H]
root         6     2  0 09:16 ?        00:00:00 [kworker/u8:0]
root         7     2  0 09:16 ?        00:00:00 [rcu_preempt]
root         8     2  0 09:16 ?        00:00:00 [rcu_sched]
root         9     2  0 09:16 ?        00:00:00 [rcu_bh]
root        10     2  0 09:16 ?        00:00:00 [migration/0]
root        11     2  0 09:16 ?        00:00:00 [migration/1]
root        12     2  0 09:16 ?        00:00:00 [ksoftirqd/1]
root        14     2  0 09:16 ?        00:00:00 [kworker/1:0H]
root        15     2  0 09:16 ?        00:00:00 [migration/2]
root        16     2  0 09:16 ?        00:00:00 [ksoftirqd/2]
root        17     2  0 09:16 ?        00:00:00 [kworker/2:0]
root        18     2  0 09:16 ?        00:00:00 [kworker/2:0H]
root        19     2  0 09:16 ?        00:00:00 [migration/3]
root        20     2  0 09:16 ?        00:00:00 [ksoftirqd/3]
root        21     2  0 09:16 ?        00:00:00 [kworker/3:0]
root        22     2  0 09:16 ?        00:00:00 [kworker/3:0H]
root        23     2  0 09:16 ?        00:00:00 [khelper]
root        24     2  0 09:16 ?        00:00:00 [kdevtmpfs]
root        25     2  0 09:16 ?        00:00:00 [netns]
root        26     2  0 09:16 ?        00:00:00 [perf]
root        27     2  0 09:16 ?        00:00:00 [khungtaskd]
root        28     2  0 09:16 ?        00:00:00 [writeback]
root        29     2  0 09:16 ?        00:00:00 [crypto]
root        30     2  0 09:16 ?        00:00:00 [bioset]
root        31     2  0 09:16 ?        00:00:00 [kblockd]
root        32     2  0 09:16 ?        00:00:00 [kworker/1:1]
root        33     2  0 09:16 ?        00:00:00 [rpciod]
root        34     2  0 09:16 ?        00:00:00 [kswapd0]
root        35     2  0 09:16 ?        00:00:00 [fsnotify_mark]
root        36     2  0 09:16 ?        00:00:00 [nfsiod]
root        42     2  0 09:16 ?        00:00:00 [kthrotld]
root        43     2  0 09:16 ?        00:00:00 [kworker/0:1]
root        44     2  0 09:16 ?        00:00:00 [VCHIQ-0]
root        45     2  0 09:16 ?        00:00:00 [VCHIQr-0]
root        46     2  0 09:16 ?        00:00:00 [VCHIQs-0]
root        47     2  0 09:16 ?        00:00:00 [iscsi_eh]
root        48     2  0 09:16 ?        00:00:00 [dwc_otg]
root        49     2  0 09:16 ?        00:00:00 [DWC Notificatio]
root        51     2  0 09:16 ?        00:00:00 [VCHIQka-0]
root        52     2  0 09:16 ?        00:00:00 [SMIO]
root        53     2  0 09:16 ?        00:00:00 [deferwq]
root        54     2  0 09:16 ?        00:00:00 [kworker/u8:2]
root        55     2  0 09:16 ?        00:00:00 [kworker/2:1]
root        56     2  0 09:16 ?        00:00:00 [mmcqd/0]
root        58     2  0 09:16 ?        00:00:00 [jbd2/mmcblk0p2-]
root        59     2  0 09:16 ?        00:00:00 [ext4-rsv-conver]
root        74     2  0 09:16 ?        00:00:00 [kworker/3:1]
root       176     1  0 09:16 ?        00:00:00 udevd --daemon
root       200     2  0 09:16 ?        00:00:00 [kworker/1:4]
root       260     2  0 09:16 ?        00:00:00 [3f204000.spi]
root       299   176  0 09:16 ?        00:00:00 udevd --daemon
root       319   176  0 09:16 ?        00:00:00 udevd --daemon
root      1619     1  0 09:17 ?        00:00:00 /usr/sbin/ifplugd -i lo -q -f -u0 -d10 -w -I
root      1642     1  0 09:17 ?        00:00:00 /usr/sbin/ifplugd -i eth0 -q -f -u0 -d10 -w -I
root      1836     1  0 09:17 ?        00:00:00 /bin/sh /etc/init.d/rc 2
root      1845  1836  0 09:17 ?        00:00:00 startpar -p 4 -t 20 -T 3 -M start -P N -R 2
nobody    1903     1  0 09:17 ?        00:00:00 /usr/sbin/thd --daemon --triggers /etc/triggerhappy/triggers.d/
--socket /var/run/thd.socket --pidfile /var/run/thd.pid --user nobody /dev/input/event*
root      1909     2  0 09:17 ?        00:00:00 [kworker/0:2]
root      1912     1  0 09:17 ?        00:00:00 /usr/sbin/rsyslogd -c5
root      2037     1  0 09:17 ?        00:00:00 /usr/sbin/cron
ntp       2050     1  0 09:17 ?        00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
104       2082     1  0 09:17 ?        00:00:00 /usr/bin/dbus-daemon --system
root      2087  2050  0 09:17 ?        00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
root      2142  1845  0 09:17 ?        00:00:00 /bin/sh /etc/init.d/rc.local start
root      2148  2142  0 09:17 ?        00:00:00 /bin/sh -e /etc/rc.local
root      2151  2148  0 09:17 ?        00:00:00 /bin/csh -f /MIDAS/TclHttpd/linux-arm/TclHttpd-server
root      2156  2151  0 09:17 ?        00:00:03 /usr/bin/tclsh /MIDAS/TclHttpd/bin/httpd_Server.tcl -uid 0 -gid 0
root      2165     1  0 09:17 ?        00:00:00 dhclient -v -pf /run/dhclient.eth0.pid -lf
/var/lib/dhcp/dhclient.eth0.leases eth0
root      2210     1  0 09:17 ?        00:00:00 /usr/sbin/sshd
root      2269  2210  0 09:18 ?        00:00:00 sshd: pi [priv]  
pi        2273  2269  0 09:19 ?        00:00:00 sshd: pi@pts/0   
pi        2278  2273  0 09:19 pts/0    00:00:00 -bash
pi        2306  2278  0 09:24 pts/0    00:00:00 ps -ef

After selecting Parse USB ...

pi@npg-monitor /tmp $ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 09:16 ?        00:00:01 init [2]  
root         2     0  0 09:16 ?        00:00:00 [kthreadd]
root         3     2  0 09:16 ?        00:00:02 [ksoftirqd/0]
root         5     2  0 09:16 ?        00:00:00 [kworker/0:0H]
root         6     2  0 09:16 ?        00:00:00 [kworker/u8:0]
root         7     2  0 09:16 ?        00:00:00 [rcu_preempt]
root         8     2  0 09:16 ?        00:00:00 [rcu_sched]
root         9     2  0 09:16 ?        00:00:00 [rcu_bh]
root        10     2  0 09:16 ?        00:00:00 [migration/0]
root        11     2  0 09:16 ?        00:00:00 [migration/1]
root        12     2  0 09:16 ?        00:00:00 [ksoftirqd/1]
root        14     2  0 09:16 ?        00:00:00 [kworker/1:0H]
root        15     2  0 09:16 ?        00:00:00 [migration/2]
root        16     2  0 09:16 ?        00:00:00 [ksoftirqd/2]
root        17     2  0 09:16 ?        00:00:00 [kworker/2:0]
root        18     2  0 09:16 ?        00:00:00 [kworker/2:0H]
root        19     2  0 09:16 ?        00:00:00 [migration/3]
root        20     2  0 09:16 ?        00:00:00 [ksoftirqd/3]
root        21     2  0 09:16 ?        00:00:00 [kworker/3:0]
root        22     2  0 09:16 ?        00:00:00 [kworker/3:0H]
root        23     2  0 09:16 ?        00:00:00 [khelper]
root        24     2  0 09:16 ?        00:00:00 [kdevtmpfs]
root        25     2  0 09:16 ?        00:00:00 [netns]
root        26     2  0 09:16 ?        00:00:00 [perf]
root        27     2  0 09:16 ?        00:00:00 [khungtaskd]
root        28     2  0 09:16 ?        00:00:00 [writeback]
root        29     2  0 09:16 ?        00:00:00 [crypto]
root        30     2  0 09:16 ?        00:00:00 [bioset]
root        31     2  0 09:16 ?        00:00:00 [kblockd]
root        32     2  0 09:16 ?        00:00:00 [kworker/1:1]
root        33     2  0 09:16 ?        00:00:00 [rpciod]
root        34     2  0 09:16 ?        00:00:01 [kswapd0]
root        35     2  0 09:16 ?        00:00:00 [fsnotify_mark]
root        36     2  0 09:16 ?        00:00:00 [nfsiod]
root        42     2  0 09:16 ?        00:00:00 [kthrotld]
root        43     2  0 09:16 ?        00:00:01 [kworker/0:1]
root        44     2  0 09:16 ?        00:00:00 [VCHIQ-0]
root        45     2  0 09:16 ?        00:00:00 [VCHIQr-0]
root        46     2  0 09:16 ?        00:00:00 [VCHIQs-0]
root        47     2  0 09:16 ?        00:00:00 [iscsi_eh]
root        48     2  0 09:16 ?        00:00:00 [dwc_otg]
root        49     2  0 09:16 ?        00:00:00 [DWC Notificatio]
root        51     2  0 09:16 ?        00:00:00 [VCHIQka-0]
root        52     2  0 09:16 ?        00:00:00 [SMIO]
root        53     2  0 09:16 ?        00:00:00 [deferwq]
root        54     2  0 09:16 ?        00:00:00 [kworker/u8:2]
root        55     2  0 09:16 ?        00:00:00 [kworker/2:1]
root        56     2  0 09:16 ?        00:00:02 [mmcqd/0]
root        58     2  0 09:16 ?        00:00:00 [jbd2/mmcblk0p2-]
root        59     2  0 09:16 ?        00:00:00 [ext4-rsv-conver]
root        74     2  0 09:16 ?        00:00:00 [kworker/3:1]
root       176     1  0 09:16 ?        00:00:00 udevd --daemon
root       200     2  0 09:16 ?        00:00:00 [kworker/1:4]
root       260     2  0 09:16 ?        00:00:00 [3f204000.spi]
root       299   176  0 09:16 ?        00:00:00 udevd --daemon
root       319   176  0 09:16 ?        00:00:00 udevd --daemon
root      1619     1  0 09:17 ?        00:00:00 /usr/sbin/ifplugd -i lo -q -f -u0 -d10 -w -I
root      1642     1  0 09:17 ?        00:00:01 /usr/sbin/ifplugd -i eth0 -q -f -u0 -d10 -w -I
nobody    1903     1  0 09:17 ?        00:00:00 /usr/sbin/thd --daemon --triggers /etc/triggerhappy/triggers.d/
--socket /var/run/thd.socket --pidfile /var/run/thd.pid --user nobody /dev/input/event*
root      1909     2  0 09:17 ?        00:00:00 [kworker/0:2]
root      1912     1  0 09:17 ?        00:00:00 /usr/sbin/rsyslogd -c5
root      2037     1  0 09:17 ?        00:00:00 /usr/sbin/cron
ntp       2050     1  0 09:17 ?        00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
104       2082     1  0 09:17 ?        00:00:00 /usr/bin/dbus-daemon --system
root      2087  2050  0 09:17 ?        00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
root      2165     1  0 09:17 ?        00:00:00 dhclient -v -pf /run/dhclient.eth0.pid -lf
/var/lib/dhcp/dhclient.eth0.leases eth0
root      2210     1  0 09:17 ?        00:00:00 /usr/sbin/sshd
root      2269  2210  0 09:18 ?        00:00:00 sshd: pi [priv]  
pi        2273  2269  0 09:19 ?        00:00:00 sshd: pi@pts/0   
pi        2278  2273  0 09:19 pts/0    00:00:00 -bash
root      2309     2  0 09:28 ?        00:00:00 [kworker/1:1H]
root      2310     2  0 09:28 ?        00:00:00 [kworker/3:1H]
root      2311     2  0 09:29 ?        00:00:00 [kworker/0:0]
root      2312     1  0 09:29 tty1     00:00:00 /sbin/getty --noclear 38400 tty1
root      2313     1  0 09:29 tty2     00:00:00 /sbin/getty 38400 tty2
root      2314     1  0 09:29 tty3     00:00:00 /sbin/getty 38400 tty3
root      2315     1  0 09:29 tty4     00:00:00 /sbin/getty 38400 tty4
root      2316     1  0 09:29 tty5     00:00:00 /sbin/getty 38400 tty5
root      2317     1  0 09:29 tty6     00:00:00 /sbin/getty 38400 tty6
root      2318     1  0 09:29 ?        00:00:00 /sbin/getty -L ttyAMA0 115200 vt100
root      2319     2  0 09:29 ?        00:00:00 [kworker/u8:1]
pi        2332  2278  0 09:32 pts/0    00:00:00 ps -ef

The httpd server appears to have died
Entry  Wed Mar 29 05:08:04 2017, TD, Report - medium - delayed or missing authenticated mount requests following FEE64 power cycle messages.log
This is not, I think, a new observation.

Following FEE64 power cycles we sometimes observe delayed ( ~5-15 minutes ) authenticated mount requests
as the FEE64s mount their root filesystem from the AIDA server aidas1 - see attachment 1. 

There also appears to be an instance of a *missing* authenticated mount request - nnaida10 at 19.51

So this is *a* (but probably not *the*) cause of FEE64 timeouts observed following a FEE64 power cycle
and then waiting 10 minutes before attempting to start the MIDAS DAQ.


I have noticed that these issues appear to be significantly worse when we are using NTFS-mounted USB disks
to backup data from the AIDA server internal disks. Indeed, performance in general is noticeably slower when
using the NTFS-mounted USB disks.

Currently, we attempt to mitigate these issues by either throttling rsync or nice'ing cp. 
Entry  Sun Oct 30 03:26:55 2016, TD, Report - medium - cannot restore layout if currently displaying >16 galleries 
Cannot restore layout if currently displaying >16 galleries

Obtain the error message 'loading Spectrum.tcl > can't read "dbtagx(31,1)": no such element in array'
Entry  Tue Nov 8 00:12:56 2016, TD, Report - medium - TapeServer changes volume name following request for existing filename 9.png
When you request a filename that already exists and select OK to the prompt
suggesting a new filename, TapeServer adds an additional character to the 
volume name - see attachment 1
Entry  Tue Dec 6 02:30:35 2016, TD, Report - low - significant variation in FEE64 boot times 
Not something we were not aware of already but /var/log/messages illustrate the issue.

Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.8:713 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida8 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.6:819 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida6 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.4:964 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida4 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.18:698 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida18 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.7:668 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida7 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.5:855 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida5 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:02 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.17:726 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida17 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.10:676 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida10 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.16:881 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida16 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.12:813 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida12 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.11:855 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida11 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.14:761 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida14 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.9:814 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida9 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.13:803 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida13 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.15:803 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida15 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:06 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.19:821 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida19 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:07 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.22:747 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida22 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:07 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.1:920 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida1 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:07 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.23:810 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida23 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:07 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.20:780 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida20 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:07 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.24:805 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida24 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:13:07 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.21:694 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida21 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:14:57 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.14:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:57 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.10:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:57 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.8:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:57 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.4:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:57 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.7:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:58 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.13:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:58 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.12:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:58 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.16:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:58 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.15:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:14:59 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.9:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.1:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.20:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.23:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.6:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.18:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.24:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.22:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.11:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:00 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.19:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:01 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.5:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:01 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.21:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:03 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.17:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:15:58 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.2:1018 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida2 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:16:51 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.1.2:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:17:06 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.3:785 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida3 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:17:47 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.3:864 for /MIDAS@aidas
(/MIDAS@aidas)
Dec  6 11:18:55 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.15:893 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida15 (/data10/XilinxLinux/ppc_4xx/rfs)
Dec  6 11:19:35 aidas1 rpc.mountd[2926]: authenticated mount request from 10.1.2.15:864 for /MIDAS@aidas
(/MIDAS@aidas)
Entry  Tue Mar 8 15:32:53 2016, TD, Report - low - run control frame issue with multiple streams 7.png
The run control frame does not appear to update the data transfer 'X' tag 
for all FEE modules after selecting 'Enable #1 All'
Entry  Mon May 13 10:00:01 2019, TD, Report - low - SETUP/Perform Timestamp Resync produces error message 1.png
AIDA@RIKEN

Selecting SETUP (Run Control tab) or Perform Timestamp Resync (AIDA Control tab) produces error message - see
attachment 1

Selecting Perform Resync (AIDA Master Timestamp Control) works OK
Entry  Wed Feb 24 13:54:53 2016, TD, Report - low - FEE temperature plot displays incorrect dates 77.png
See Attachment 1

FEE temperature plot shows data for incorrect dates
Entry  Sun Dec 4 04:05:24 2016, TD, Report - low - DAQ SETUP sometimes produces Javascript error 10.png
DAQ SETUP sometimes produces Javascript error instead of a pop-up displaying
the status of the ReSync - see attachment 1

Re-run ReSync by selecting 'Perform ReSync' from 'Master Timestamp Control' 
Entry  Fri Sep 30 07:52:56 2016, TD, Report - low - AIDA correlation L/M/H statistics count 3x  
The AIDA correlation L, M & H statistics count 3x for each correlation trigger,
i.e. counts and rates displayed by statistics are 3x actual counts and rates 
Entry  Mon Apr 25 08:58:25 2016, TD, Report - high? - aidas1 - Binding NIS service FAILED  messages
During a reboot of aidas1 the message 

Binding NIS service FAILED

was observed during the services startup sequence. Is this an issue?

I can ping/ssh w29.ph.ed.ac.uk (Edinburgh NP group Sun server).

/var/log/messages appended
Entry  Tue Nov 14 17:08:49 2023, TD, Report - high - unable to change ASIC settings directory in Options  Screenshot_from_2023-11-14_18-07-27.pngScreenshot_from_2023-11-14_18-07-56.png
Attempting to change the text string specifying the ASIC settings directory ( 2023Nov14-15.25.24 ) in Options produces an error - see attachments 1 & 2

System aida-gsi, anydesk 246 067 683
Entry  Sun Oct 23 10:22:36 2016, TD, Report - high - merger SEGV 
Merger crashed during operation - see https://elog.ph.ed.ac.uk/AIDA/399
Entry  Wed Mar 29 13:02:17 2017, TD, Report - high - interaction between DHCP and FEE64 authenticated mount requests 
What has happened here?

Power cycle of FEE64s 
24x authenticated mount request c. 20.56

Followed by repeat for nnaida20 following DHCP activity c. 21.00

Mar 29 20:56:44 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.1:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:44 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.2:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:45 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.7:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:45 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.4:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:46 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.3:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:46 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.5:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:46 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.8:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.9:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.10:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.12:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.24:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.13:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.6:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.20:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.14:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.15:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.16:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.22:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.18:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.23:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.11:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:49 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.19:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:49 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.21:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:49 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.17:864 for /MIDAS@aidas
(/MIDAS@aidas)


Mar 29 21:00:43 aidas1 dhcpd: DHCPDISCOVER from 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 dhcpd: DHCPOFFER on 10.1.2.20 to 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 dhcpd: DHCPREQUEST for 10.1.2.20 (10.1.2.250) from 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 dhcpd: DHCPACK on 10.1.2.20 to 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.20:686 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida20 (/data10/XilinxLinux/ppc_4xx/rfs)
Mar 29 21:01:08 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.20:864 for /MIDAS@aidas
(/MIDAS@aidas)
Entry  Sat Jun 3 05:32:09 2017, TD, Report - high - MBS correlation scaler anomalies  testmbs
The analysis of files RIBF127R1/R1_1478 to R1_1486 is appended. This sequence of files should
encompass a correlation reset request (from BRIKEN to AIDA)

There appear to be a number of anomalies

1) MBS correlation scalers observed with one missing 16-bit data value (bits 16-31 or bits 0-15)
   across approximately five files (~20 minutes)

2) The first (presumably) MBS correlation scaler following reset corresponds to a time ~300s (25MHz clock)   
Entry  Tue Mar 8 15:09:49 2016, TD, Report - high - LEC/MEC fast comparator issues 6x
Fast comparator rate spectra and statistics do not appear to working as expected. See enclosed screenshots.
Have the hit/rate spectra and statistics been updated to reflect the firmware version 8 data format?
    Reply  Wed Mar 9 17:31:58 2016, TD, Report - high - LEC/MEC fast comparator issues Discriminator_Thresh_50
> 
> Fast comparator rate spectra and statistics do not appear to working as expected. See enclosed screenshots.
> Have the hit/rate spectra and statistics been updated to reflect the firmware version 8 data format?

I have done a simple test using the system in T9.
Detector on.
Acquisition only ... no Merge
Set pulser attenuation to 2X instead of 10X.

Set Fast Comparator threshold to 50 to be well out of the detector noise.

See attached rate spectrum which shows system operating as I think it should.
Entry  Tue Mar 8 16:29:07 2016, TD, Report - high - AIDA timestamps 8.png
Version 8 firmware

Analysing R37_* data files (see https://elog.ph.ed.ac.uk/AIDA/160)

Usinmg MIDASsort, I generate a spectrum showing the time difference (10ns clock cycles)
between sequential ADC data items. There is an offset of 1000.

I observe peaks at 1000, 1200, 1400 etc - as expected

I also observe peaks at +/-10 clock cycles relative to these main peaks - not expected

I also observe some discrete intermediate peaks - not expected 
    Reply  Thu Mar 10 16:35:39 2016, TD : PJCS, Report - high - AIDA timestamps Time_Difference_FEE13to14.png
> Version 8 firmware
> 
> Analysing R37_* data files (see https://elog.ph.ed.ac.uk/AIDA/160)
> 
> Usinmg MIDASsort, I generate a spectrum showing the time difference (10ns clock cycles)
> between sequential ADC data items. There is an offset of 1000.
> 
> I observe peaks at 1000, 1200, 1400 etc - as expected
> 
> I also observe peaks at +/-10 clock cycles relative to these main peaks - not expected
> 
> I also observe some discrete intermediate peaks - not expected 

I have investigated this timing using a sort program and R37_0.

If I only work with data from one module there is no problem. Peaks in the time difference spectrum are the
expected 200 apart.

If I then include a second module's ADC data the time difference spectrum starts to look as the one  Tom attached
but with a fixed 45 ticks offset from the '200' peaks.

I conclude the spectra appearance is due to the 500KHz ASIC readout clocks not being synchronized between the
different FEE64 modules.
ELOG V3.1.3-7933898