AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 12 of 46  ELOG logo
ID Date Authordown Subject
  255   Mon May 30 03:58:10 2016 TDpeakfind not working on aidas1
Peakfind appears not to be working on aidas1 - example spectrum attached


aidas1> peakfind -spectrum /MIDAS/SpecData/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf -limit 33000 64000 -fwhm 85
-accept 10
Peak finding for /MIDAS/SpecData/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf, Title 
No.     Channel            Area           Fwhm 
aidas1> ls -l /MIDAS/SpecData/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf
-rwxrwxrwx. 1 nfsnobody nfsnobody 262656 May 30 10:42 /MIDAS/SpecData/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf


Same spectrum analysed at Edinburgh (Sun/Solaris 10)

w29> peakfind -spectrum /home/td/riken/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf -limit 33000 64000 -fwhm 85
-accept 10
Peak finding for /home/td/riken/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf, Title 
No.     Channel            Area           Fwhm 
  1   34649.72(3.059)      123.5(  11.3)   65.98
  2   35542.99(0.855)     1756.0(  42.0)   79.13
  3   36363.14(0.854)     1820.0(  42.8)   79.66
  4   37203.16(0.776)     2139.9(  46.4)   80.00
  5   38087.83(0.852)     1863.0(  43.3)   79.62
  6   38984.27(0.842)     1853.9(  43.2)   79.27
  7   39889.04(0.832)     1755.9(  42.0)   78.59
  8   40790.80(0.855)     1930.7(  44.2)   79.45
w29> ls -l /home/td/riken/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf
-rwxr-xr-x   1 td       staff     262656 May 30 02:42 /home/td/riken/AIDA/nnaida4/2016May30-03.41.52/1.5.L.msf
Attachment 1: 1.5.L.msf
  264   Wed Jun 1 01:32:04 2016 TDCo/Bi source run (R19)
09.18 R19 (AIDA) stopped 
      AIDA SYNC lost for most (but not all) FEE64s - see attachments 1-2
      Statistics show similar numbers of SYNCs which may imply a recent event/failure
      
Attachment 1: 110.png
110.png
Attachment 2: 111.png
111.png
  265   Wed Jun 1 01:33:58 2016 TDWednesday 1 June 2016
09.30 Detector biases and leakage currents - ambient temp +24.9 deg C - attachments 1-2

10.08 Connection lost to nnaida27 and nnaida28 
      Most recent save of settings DB key 2016May31-23.32.46

12.00 Replaced nnaida17 & nnaida18 with FEE module previously used for nnaida31 & nnaida32

      Pulser walkthrough

      BNC PB-4 settings 

      90,000 - 10,000 @ 10,000 step
      x5 attenuator (was x10 previously)
      t_r = 50ns tau_d = 50us
      polarity -
      rate ~ 150Hz
      INT, delay MIN

      polarity + by EG&G Ortec 433A

      N.B. - polarity terminated at end of AIDA adaptor chain, + polarity not terminated

      Spectra saved 12.32 ( to directories tagged around June 1, 2016 05.29 )

      BNC PB4 pulser rate set to ~5Hz

      207Bi source at front of snout and 2x 60Co sources above DSSD3

14.00 Rebooted aidas1 to cleanup merge 
      
      Changed config to 24x FEE64s 

      Clock issues with nnaida23 - clock not locked status 0x6 - see attachments 3-9

18.00 Replace current nnaida19 (no histogramming, reSYNCs OK but timestamp not sync'd wrt timestamps of
      other FEE64s).
      Replace current nnaida23

      Following powerup obtained clock not locked - status 0x6 for nnaida5 (master)

      Replacement nnaida19 also does not histogram, reSYNCs OK but timestamp not sync'd wrt timestamps
      of other FEE64s)

      With help from PJCS investigate above issues and change timestamp master from nnaida5 to nnaida6
      ( /MIDAS/config/HttpdTcl/aidas1/startup.tcl and /MIDAS/config/HttpdTcl/nnaida5/startup.tcl ) to 
      change definition of timestamp master.

      Able to operate DAQ and obtain AIDA SYNCs and WAVE SYNCs with TS_SYNC 0x0 _only_ 
      PJCS will also investigate using system in T9 at STFC DL

      Restored settings 2016June1-17.27.?
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 200.png
200.png
Attachment 4: 201.png
201.png
Attachment 5: 202.png
202.png
Attachment 6: 204.png
204.png
Attachment 7: 205.png
205.png
Attachment 8: 206.png
206.png
Attachment 9: 207.png
207.png
  266   Wed Jun 1 07:49:45 2016 TDDetector noise - state of play
Screenshots of ASIC#1 waveforms for each side of each of the DSSSD #1 - #6 are attached.

Standard ASIC settings - tau = 8us

Typical threshold values are ~8500 (positive signals) and ~7800 (negative signals)

Large (off-scale) signals are the pulser, lower amplitude signals are betas

Currently, the noise looks OK.

Note that EURICA is currently open (in preparation for IMPACT run) and (just possibly) SAMURAI
dipole in adjacent experimental area  is now off (to be checked).

We (AIDA) have made no further changes since yesterday's addition of Field Plate grounding jumpers
to all four adaptor PCBs of DSSSD #1 and #6.
Attachment 1: 301.png
301.png
Attachment 2: 302.png
302.png
Attachment 3: 303.png
303.png
Attachment 4: 304.png
304.png
Attachment 5: 310.png
310.png
Attachment 6: 311.png
311.png
Attachment 7: 312.png
312.png
Attachment 8: 317.png
317.png
Attachment 9: 318.png
318.png
Attachment 10: 320.png
320.png
  271   Thu Jun 2 09:04:57 2016 TD[How to] Monitor AIDA
Every 4h

1) Record DSSSD #1 = #6 detector bias voltages and leakage currents in written logbook

   Desktop 6 - 2x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 3 = DSSSD # 4

   Addr 01

   Ch 0 = DSSSD # 5
   Ch 1 = DSSSD # 6

   Compare with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select Reload

   Check temperatures

       Virtex < 75 deg C
       PSU    < 45 deg C
       ASIC   < 65 deg C

   If not call for expert help. 

   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Statistics tab

   Check Reload

   Check 'Show ALL Data Acquisition Servers' 

   From 'Select counter to Monitor' select 'AIDA Discriminator'

   Select 'Update Once'

   Check *all* data acquisition servers have a non-zero rate

   N.B. If one, or more, data acquisition servers have zero rate call for expert help!

   From keyboard select Alt-Print Screen and save screenshot to /tmp directory as a .png file
   Upload the .png file to the EURICA 2016 Elog for the current run

   Compare rates with previous observations at same settings 

4) Desktop 1 - Firefox web browser - Statistics tab

   Check Reload

   Check 'Show ALL Data Acquisition Servers' 

   From 'Select counter to Monitor' select 'AIDA SYNC'

   Select 'Update Once'

   Check *all* data acquisition servers have a non-zero rate (typically ~300-500Hz)

   N.B. If one, or more, data acquisition servers have zero rate call for expert help!

5) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check all enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are *always* inactive call for expert help!

   Select Reload to observe merge rate (date items/s)

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

6) Desktop 2 - Firefox web browser - TapeServer tab

   Select Reload and observe number of blocks written to disk
  274   Sat Jun 4 08:46:20 2016 TDAIDA noise - state of play 2
Attachments  1-23 AIDA FEE64s nnaida1-nnaida23 ASIC #1 waveforms

Attachments 24-27 AIDA FEE64 nnaida24 ASIC #1 - #4 waveforms

Common y-scale 7000-9000, xscale ~1000 samples @ 20ns/sample

All *L and *W histograms saved to /MIDAS/SpecData at c. 17.00 Saturday 4 June 2016

Note

1) BNC PB-4 Pulser amplitude 90,000, x5 attenuator, frequency c. 5Hz + ~40kBq 60Co source
placed on AIDA snout

2) EURICA closed
Attachment 1: 101.png
101.png
Attachment 2: 102.png
102.png
Attachment 3: 103.png
103.png
Attachment 4: 104.png
104.png
Attachment 5: 105.png
105.png
Attachment 6: 106.png
106.png
Attachment 7: 107.png
107.png
Attachment 8: 108.png
108.png
Attachment 9: 109.png
109.png
Attachment 10: 110.png
110.png
Attachment 11: 111.png
111.png
Attachment 12: 112.png
112.png
Attachment 13: 113.png
113.png
Attachment 14: 114.png
114.png
Attachment 15: 115.png
115.png
Attachment 16: 116.png
116.png
Attachment 17: 117.png
117.png
Attachment 18: 118.png
118.png
Attachment 19: 119.png
119.png
Attachment 20: 120.png
120.png
Attachment 21: 121.png
121.png
Attachment 22: 122.png
122.png
Attachment 23: 123.png
123.png
Attachment 24: 124.png
124.png
Attachment 25: 125.png
125.png
Attachment 26: 126.png
126.png
Attachment 27: 127.png
127.png
  279   Mon Jun 6 08:51:50 2016 TDChange of set point for Julabo FL11006 recirculating chiller
The Julabo FL11006 recirculating chiller set point has been changed from +20.0 deg C to +15.0 deg C.

Ambient temperature in B2F/F11 +25.2 deg C, RH 39.8%, dew point +11.8 deg C

AIDA FEE temperatures decreasing slowly - see attachment 1

7.6.16 10.30 For comparison the latest AIDA FEE temperatures - see attachment 2
Attachment 1: 10.png
10.png
Attachment 2: 100.png
100.png
  289   Tue Jun 7 10:36:14 2016 TDnnaida19
aidas1> cat var/log/dmesg
Using Xilinx Virtex440 machine description
Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Top of RAM: 0x7000000, Total RAM: 0x7000000
Memory hole size: 0MB
Zone PFN ranges:
  DMA      0x00000000 -> 0x00007000
  Normal   0x00007000 -> 0x00007000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00007000
On node 0 totalpages: 28672
free_area_init_node: node 0, pgdat c038c384, node_mem_map c03b8000
  DMA zone: 224 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 28448 pages, LIFO batch:7
MMU: Allocated 1088 bytes of context maps for 255 contexts
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
PID hash table entries: 512 (order: 9, 2048 bytes)
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
Kernel virtual memory layout:
  * 0xffffe000..0xfffff000  : fixmap
  * 0xfde00000..0xfe000000  : consistent mem
  * 0xfde00000..0xfde00000  : early ioremap
  * 0xd1000000..0xfde00000  : vmalloc & ioremap
NR_IRQS:512
time_init: decrementer frequency = 400.000000 MHz
time_init: processor frequency   = 400.000000 MHz
clocksource: timebase mult[a00000] shift[22] registered
clockevent: decrementer mult[66666666] shift[32] cpu[0]
Console: colour dummy device 80x25
Mount-cache hash table entries: 512
NET: Registered protocol family 16
PCI: Probing PCI hardware
bio: create slab <bio-0> at 0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
NET: Registered protocol family 1
ROMFS MTD (C) 2007 Red Hat, Inc.
msgmni has been set to 214
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
irq: irq 2 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 16
83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
console [ttyS0] enabled
brd: module loaded
loop: module loaded
Device Tree Probing 'ethernet'
irq: irq 1 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 17
irq: irq 3 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 18
irq: irq 4 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 19
xilinx_lltemac 81c00000.ethernet: MAC address is now  0: 4:a3:2a:ed:9a
xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
XLlTemac: DCR address: 0x80
XLlTemac: buffer descriptor size: 32768 (0x8000)
XLlTemac: Allocating DMA descriptors with kmalloc
XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
XTemac: PHY detected at address 7.
xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
 Intel/Sharp Extended Query Table at 0x010A
 Intel/Sharp Extended Query Table at 0x010A
 Intel/Sharp Extended Query Table at 0x010A
 Intel/Sharp Extended Query Table at 0x010A
 Intel/Sharp Extended Query Table at 0x010A
 Intel/Sharp Extended Query Table at 0x010A
Using buffer write method
cfi_cmdset_0001: Erase suspend on write enabled
erase region 0: offset=0x0,size=0x20000,blocks=127
erase region 1: offset=0xfe0000,size=0x8000,blocks=4
fc000000.flash: 1 set(s) of 1 interleaved chips --> 16 partitions of 1024 KiB
cmdlinepart partition parsing not available
RedBoot partition parsing not available
Creating 5 MTD partitions on "fc000000.flash":
0x000000000000-0x000000500000 : "golden_firmware"
0x000000500000-0x000000800000 : "golden_kernel"
0x000000800000-0x000000d00000 : "user_firmware"
0x000000d00000-0x000000fe0000 : "user_kernel"
0x000000fe0000-0x000001000000 : "env_variables"
irq: irq 6 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 20
xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
SPI: XIlinx spi: bus number now 32766
irq: irq 5 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 21
xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
SPI: XIlinx spi: bus number now 32765
mice: PS/2 mouse device common for all mice
Device Tree Probing 'i2c'
irq: irq 7 on host /plb@0/interrupt-controller@81800000 mapped to virtual irq 22
 #0 at 0x81600000 mapped to 0xD1030000, irq=22
at24 0-0050: 1024 byte 24c08 EEPROM (writable)
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
eth0: XLlTemac: Options: 0x3fa
eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
eth0: XLlTemac: speed set to 1000Mb/s
eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
Sending DHCP requests ., OK
IP-Config: Got DHCP answer from 10.1.2.250, my address is 10.1.2.19
IP-Config: Complete:
     device=eth0, addr=10.1.2.19, mask=255.255.255.0, gw=10.1.2.250,
     host=nnaida19, domain=, nis-domain=(none),
     bootserver=10.1.2.250, rootserver=10.1.2.250, rootpath=/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida19,tcp
Looking up port of RPC 100003/2 on 10.1.2.250
Looking up port of RPC 100005/1 on 10.1.2.250
VFS: Mounted root (nfs filesystem) on device 0:12.
Freeing unused kernel memory: 168k init

cat var/log/messages
Jun  7 00:20:33 nnaida19 syslogd 1.4.2: restart.
Jun  7 00:20:38 nnaida19 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Jun  7 00:20:38 nnaida19 kernel: Using Xilinx Virtex440 machine description
Jun  7 00:20:39 nnaida19 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue
Nov 15 15:57:04 GMT 2011
Jun  7 00:20:39 nnaida19 kernel: Zone PFN ranges:
Jun  7 00:20:39 nnaida19 kernel:   DMA      0x00000000 -> 0x00007000
Jun  7 00:20:39 nnaida19 kernel:   Normal   0x00007000 -> 0x00007000
Jun  7 00:20:39 nnaida19 kernel: Movable zone start PFN for each node
Jun  7 00:20:40 nnaida19 kernel: early_node_map[1] active PFN ranges
Jun  7 00:20:40 nnaida19 kernel:     0: 0x00000000 -> 0x00007000
Jun  7 00:20:40 nnaida19 kernel: MMU: Allocated 1088 bytes of context maps for 255 contexts
Jun  7 00:20:40 nnaida19 kernel: Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
Jun  7 00:20:40 nnaida19 kernel: Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Jun  7 00:20:41 nnaida19 kernel: PID hash table entries: 512 (order: 9, 2048 bytes)
Jun  7 00:20:41 nnaida19 kernel: Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Jun  7 00:20:41 nnaida19 kernel: Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Jun  7 00:20:41 nnaida19 kernel: Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k
data, 130k bss, 168k init)
Jun  7 00:20:41 nnaida19 kernel: Kernel virtual memory layout:
Jun  7 00:20:42 nnaida19 kernel:   * 0xffffe000..0xfffff000  : fixmap
Jun  7 00:20:43 nnaida19 kernel:   * 0xfde00000..0xfe000000  : consistent mem
Jun  7 00:20:43 nnaida19 kernel:   * 0xfde00000..0xfde00000  : early ioremap
Jun  7 00:20:44 nnaida19 kernel:   * 0xd1000000..0xfde00000  : vmalloc & ioremap
Jun  7 00:20:44 nnaida19 kernel: NR_IRQS:512
Jun  7 00:20:44 nnaida19 kernel: clocksource: timebase mult[a00000] shift[22] registered
Jun  7 00:20:44 nnaida19 kernel: Console: colour dummy device 80x25
Jun  7 00:20:44 nnaida19 kernel: Mount-cache hash table entries: 512
Jun  7 00:20:45 nnaida19 kernel: NET: Registered protocol family 16
Jun  7 00:20:45 nnaida19 kernel: PCI: Probing PCI hardware
Jun  7 00:20:45 nnaida19 kernel: bio: create slab <bio-0> at 0
Jun  7 00:20:45 nnaida19 kernel: NET: Registered protocol family 2
Jun  7 00:20:45 nnaida19 kernel: IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
Jun  7 00:20:46 nnaida19 kernel: TCP established hash table entries: 4096 (order: 3, 32768 bytes)
Jun  7 00:20:46 nnaida19 kernel: TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
Jun  7 00:20:46 nnaida19 kernel: TCP: Hash tables configured (established 4096 bind 4096)
Jun  7 00:20:46 nnaida19 kernel: TCP reno registered
Jun  7 00:20:47 nnaida19 kernel: NET: Registered protocol family 1
Jun  7 00:20:47 nnaida19 kernel: ROMFS MTD (C) 2007 Red Hat, Inc.
Jun  7 00:20:47 nnaida19 kernel: msgmni has been set to 214
Jun  7 00:20:47 nnaida19 kernel: io scheduler noop registered
Jun  7 00:20:47 nnaida19 kernel: io scheduler anticipatory registered
Jun  7 00:20:48 nnaida19 kernel: io scheduler deadline registered
Jun  7 00:20:48 nnaida19 kernel: io scheduler cfq registered (default)
Jun  7 00:20:48 nnaida19 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Jun  7 00:20:48 nnaida19 kernel: 83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
Jun  7 00:20:48 nnaida19 kernel: console [ttyS0] enabled
Jun  7 00:20:49 nnaida19 kernel: brd: module loaded
Jun  7 00:20:49 nnaida19 kernel: loop: module loaded
Jun  7 00:20:49 nnaida19 kernel: Device Tree Probing 'ethernet'
Jun  7 00:20:49 nnaida19 kernel: xilinx_lltemac 81c00000.ethernet: MAC address is now  0: 4:a3:2b:27:54
Jun  7 00:20:49 nnaida19 kernel: xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
Jun  7 00:20:49 nnaida19 kernel: XLlTemac: DCR address: 0x80
Jun  7 00:20:50 nnaida19 kernel: XLlTemac: buffer descriptor size: 32768 (0x8000)
Jun  7 00:20:50 nnaida19 kernel: XLlTemac: Allocating DMA descriptors with kmalloc
Jun  7 00:20:50 nnaida19 kernel: XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
Jun  7 00:20:50 nnaida19 kernel: XTemac: PHY detected at address 7.
Jun  7 00:20:50 nnaida19 kernel: xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to
0xD1024000, irq=17
Jun  7 00:20:50 nnaida19 kernel: fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
Jun  7 00:20:50 nnaida19 kernel:  Intel/Sharp Extended Query Table at 0x010A
Jun  7 00:20:50 nnaida19 last message repeated 5 times
Jun  7 00:20:50 nnaida19 kernel: Using buffer write method
Jun  7 00:20:51 nnaida19 kernel: cfi_cmdset_0001: Erase suspend on write enabled
Jun  7 00:20:51 nnaida19 kernel: cmdlinepart partition parsing not available
Jun  7 00:20:51 nnaida19 kernel: RedBoot partition parsing not available
Jun  7 00:20:51 nnaida19 kernel: Creating 5 MTD partitions on "fc000000.flash":
Jun  7 00:20:51 nnaida19 kernel: 0x000000000000-0x000000500000 : "golden_firmware"
Jun  7 00:20:51 nnaida19 kernel: 0x000000500000-0x000000800000 : "golden_kernel"
Jun  7 00:20:51 nnaida19 kernel: 0x000000800000-0x000000d00000 : "user_firmware"
Jun  7 00:20:51 nnaida19 kernel: 0x000000d00000-0x000000fe0000 : "user_kernel"
Jun  7 00:20:51 nnaida19 kernel: 0x000000fe0000-0x000001000000 : "env_variables"
Jun  7 00:20:52 nnaida19 kernel: xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
Jun  7 00:20:52 nnaida19 kernel: SPI: XIlinx spi: bus number now 32766
Jun  7 00:20:52 nnaida19 kernel: xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
Jun  7 00:20:52 nnaida19 kernel: SPI: XIlinx spi: bus number now 32765
Jun  7 00:20:52 nnaida19 kernel: mice: PS/2 mouse device common for all mice
Jun  7 00:20:52 nnaida19 kernel: Device Tree Probing 'i2c'
Jun  7 00:20:52 nnaida19 kernel:  #0 at 0x81600000 mapped to 0xD1030000, irq=22
Jun  7 00:20:52 nnaida19 kernel: at24 0-0050: 1024 byte 24c08 EEPROM (writable)
Jun  7 00:20:53 nnaida19 kernel: TCP cubic registered
Jun  7 00:20:53 nnaida19 kernel: NET: Registered protocol family 17
Jun  7 00:20:53 nnaida19 kernel: RPC: Registered udp transport module.
Jun  7 00:20:53 nnaida19 kernel: RPC: Registered tcp transport module.
Jun  7 00:20:53 nnaida19 kernel: eth0: XLlTemac: Options: 0x3fa
Jun  7 00:20:53 nnaida19 kernel: eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
Jun  7 00:20:53 nnaida19 kernel: eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
Jun  7 00:20:53 nnaida19 kernel: eth0: XLlTemac: speed set to 1000Mb/s
Jun  7 00:20:53 nnaida19 kernel: eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
Jun  7 00:20:53 nnaida19 kernel: eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
Jun  7 00:20:54 nnaida19 kernel: Sending DHCP requests ., OK
Jun  7 00:20:54 nnaida19 kernel: IP-Config: Got DHCP answer from 10.1.2.250, my address is 10.1.2.19
Jun  7 00:20:54 nnaida19 kernel: IP-Config: Complete:
Jun  7 00:20:54 nnaida19 kernel:      device=eth0, addr=10.1.2.19, mask=255.255.255.0, gw=10.1.2.250,
Jun  7 00:20:54 nnaida19 kernel:      host=nnaida19, domain=, nis-domain=(none),
Jun  7 00:20:54 nnaida19 kernel:      bootserver=10.1.2.250, rootserver=10.1.2.250,
rootpath=/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida26,tcp
Jun  7 00:20:54 nnaida19 kernel: Looking up port of RPC 100003/2 on 10.1.2.250
Jun  7 00:20:54 nnaida19 kernel: Looking up port of RPC 100005/1 on 10.1.2.250
Jun  7 00:20:55 nnaida19 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Jun  7 00:20:55 nnaida19 kernel: Freeing unused kernel memory: 168k init
Jun  7 00:20:56 nnaida19 rpcbind: cannot create socket for udp6
Jun  7 00:20:56 nnaida19 rpcbind: cannot create socket for tcp6
Jun  7 00:21:06 nnaida19 xinetd[275]: IPv6 socket creation failed for service telnet, trying IPv4
Jun  7 00:21:06 nnaida19 xinetd[275]: IPv6 socket creation failed for service ftp, trying IPv4
Jun  7 00:21:06 nnaida19 xinetd[275]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Jun  7 00:21:06 nnaida19 xinetd[275]: Started working: 2 available services
Jun  7 00:21:15 nnaida19 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Jun  7 00:21:15 nnaida19 kernel: XAIDA: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Jun  7 00:21:15 nnaida19 kernel: XAIDA: xaida: driver assigned major number 254
Jun  7 00:21:22 nnaida19 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Jun  7 00:21:22 nnaida19 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Jun  7 00:21:22 nnaida19 kernel: AIDAMEM: aidamem: driver assigned major number 253
Jun  7 00:21:39 nnaida19 kernel: XAIDA: xaida_open:
Jun  7 00:21:39 nnaida19 kernel: AIDAMEM: aidamem_open:
Jun  7 00:36:12 nnaida19 syslogd 1.4.2: restart.
Jun  7 00:36:19 nnaida19 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Jun  7 00:36:20 nnaida19 kernel: Using Xilinx Virtex440 machine description
Jun  7 00:36:20 nnaida19 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue
Nov 15 15:57:04 GMT 2011
Jun  7 00:36:20 nnaida19 kernel: Zone PFN ranges:
Jun  7 00:36:20 nnaida19 kernel:   DMA      0x00000000 -> 0x00007000
Jun  7 00:36:20 nnaida19 kernel:   Normal   0x00007000 -> 0x00007000
Jun  7 00:36:21 nnaida19 kernel: Movable zone start PFN for each node
Jun  7 00:36:21 nnaida19 kernel: early_node_map[1] active PFN ranges
Jun  7 00:36:21 nnaida19 kernel:     0: 0x00000000 -> 0x00007000
Jun  7 00:36:21 nnaida19 kernel: MMU: Allocated 1088 bytes of context maps for 255 contexts
Jun  7 00:36:22 nnaida19 kernel: Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
Jun  7 00:36:22 nnaida19 kernel: Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Jun  7 00:36:22 nnaida19 kernel: PID hash table entries: 512 (order: 9, 2048 bytes)
Jun  7 00:36:22 nnaida19 kernel: Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Jun  7 00:36:23 nnaida19 kernel: Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Jun  7 00:36:23 nnaida19 kernel: Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k
data, 130k bss, 168k init)
Jun  7 00:36:23 nnaida19 kernel: Kernel virtual memory layout:
Jun  7 00:36:23 nnaida19 kernel:   * 0xffffe000..0xfffff000  : fixmap
Jun  7 00:36:24 nnaida19 kernel:   * 0xfde00000..0xfe000000  : consistent mem
Jun  7 00:36:24 nnaida19 kernel:   * 0xfde00000..0xfde00000  : early ioremap
Jun  7 00:36:24 nnaida19 kernel:   * 0xd1000000..0xfde00000  : vmalloc & ioremap
Jun  7 00:36:24 nnaida19 kernel: NR_IRQS:512
Jun  7 00:36:25 nnaida19 kernel: clocksource: timebase mult[a00000] shift[22] registered
Jun  7 00:36:25 nnaida19 kernel: Console: colour dummy device 80x25
Jun  7 00:36:25 nnaida19 kernel: Mount-cache hash table entries: 512
Jun  7 00:36:25 nnaida19 kernel: NET: Registered protocol family 16
Jun  7 00:36:25 nnaida19 kernel: PCI: Probing PCI hardware
Jun  7 00:36:25 nnaida19 kernel: bio: create slab <bio-0> at 0
Jun  7 00:36:25 nnaida19 kernel: NET: Registered protocol family 2
Jun  7 00:36:26 nnaida19 kernel: IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
Jun  7 00:36:26 nnaida19 kernel: TCP established hash table entries: 4096 (order: 3, 32768 bytes)
Jun  7 00:36:26 nnaida19 kernel: TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
Jun  7 00:36:26 nnaida19 kernel: TCP: Hash tables configured (established 4096 bind 4096)
Jun  7 00:36:26 nnaida19 kernel: TCP reno registered
Jun  7 00:36:26 nnaida19 kernel: NET: Registered protocol family 1
Jun  7 00:36:26 nnaida19 kernel: ROMFS MTD (C) 2007 Red Hat, Inc.
Jun  7 00:36:27 nnaida19 kernel: msgmni has been set to 214
Jun  7 00:36:27 nnaida19 kernel: io scheduler noop registered
Jun  7 00:36:27 nnaida19 kernel: io scheduler anticipatory registered
Jun  7 00:36:27 nnaida19 kernel: io scheduler deadline registered
Jun  7 00:36:27 nnaida19 kernel: io scheduler cfq registered (default)
Jun  7 00:36:27 nnaida19 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
Jun  7 00:36:27 nnaida19 kernel: 83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
Jun  7 00:36:28 nnaida19 kernel: console [ttyS0] enabled
Jun  7 00:36:28 nnaida19 kernel: brd: module loaded
Jun  7 00:36:28 nnaida19 kernel: loop: module loaded
Jun  7 00:36:28 nnaida19 kernel: Device Tree Probing 'ethernet'
Jun  7 00:36:28 nnaida19 kernel: xilinx_lltemac 81c00000.ethernet: MAC address is now  0: 4:a3:2b:27:54
Jun  7 00:36:28 nnaida19 kernel: xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
Jun  7 00:36:29 nnaida19 kernel: XLlTemac: DCR address: 0x80
Jun  7 00:36:29 nnaida19 kernel: XLlTemac: buffer descriptor size: 32768 (0x8000)
Jun  7 00:36:29 nnaida19 kernel: XLlTemac: Allocating DMA descriptors with kmalloc
Jun  7 00:36:29 nnaida19 kernel: XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
Jun  7 00:36:29 nnaida19 kernel: XTemac: PHY detected at address 7.
Jun  7 00:36:29 nnaida19 kernel: xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to
0xD1024000, irq=17
Jun  7 00:36:29 nnaida19 kernel: fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
Jun  7 00:36:29 nnaida19 kernel:  Intel/Sharp Extended Query Table at 0x010A
Jun  7 00:36:30 nnaida19 last message repeated 5 times
Jun  7 00:36:30 nnaida19 kernel: Using buffer write method
Jun  7 00:36:30 nnaida19 kernel: cfi_cmdset_0001: Erase suspend on write enabled
Jun  7 00:36:30 nnaida19 kernel: cmdlinepart partition parsing not available
Jun  7 00:36:30 nnaida19 kernel: RedBoot partition parsing not available
Jun  7 00:36:30 nnaida19 kernel: Creating 5 MTD partitions on "fc000000.flash":
Jun  7 00:36:30 nnaida19 kernel: 0x000000000000-0x000000500000 : "golden_firmware"
Jun  7 00:36:31 nnaida19 kernel: 0x000000500000-0x000000800000 : "golden_kernel"
Jun  7 00:36:31 nnaida19 kernel: 0x000000800000-0x000000d00000 : "user_firmware"
Jun  7 00:36:31 nnaida19 kernel: 0x000000d00000-0x000000fe0000 : "user_kernel"
Jun  7 00:36:31 nnaida19 kernel: 0x000000fe0000-0x000001000000 : "env_variables"
Jun  7 00:36:31 nnaida19 kernel: xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
Jun  7 00:36:31 nnaida19 kernel: SPI: XIlinx spi: bus number now 32766
Jun  7 00:36:31 nnaida19 kernel: xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
Jun  7 00:36:31 nnaida19 kernel: SPI: XIlinx spi: bus number now 32765
Jun  7 00:36:32 nnaida19 kernel: mice: PS/2 mouse device common for all mice
Jun  7 00:36:32 nnaida19 kernel: Device Tree Probing 'i2c'
Jun  7 00:36:32 nnaida19 kernel:  #0 at 0x81600000 mapped to 0xD1030000, irq=22
Jun  7 00:36:32 nnaida19 kernel: at24 0-0050: 1024 byte 24c08 EEPROM (writable)
Jun  7 00:36:32 nnaida19 kernel: TCP cubic registered
Jun  7 00:36:32 nnaida19 kernel: NET: Registered protocol family 17
Jun  7 00:36:32 nnaida19 kernel: RPC: Registered udp transport module.
Jun  7 00:36:32 nnaida19 kernel: RPC: Registered tcp transport module.
Jun  7 00:36:32 nnaida19 kernel: eth0: XLlTemac: Options: 0x3fa
Jun  7 00:36:32 nnaida19 kernel: eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
Jun  7 00:36:32 nnaida19 kernel: eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
Jun  7 00:36:33 nnaida19 kernel: eth0: XLlTemac: speed set to 1000Mb/s
Jun  7 00:36:33 nnaida19 kernel: eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
Jun  7 00:36:33 nnaida19 kernel: eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
Jun  7 00:36:33 nnaida19 kernel: Sending DHCP requests ., OK
Jun  7 00:36:33 nnaida19 kernel: IP-Config: Got DHCP answer from 10.1.2.250, my address is 10.1.2.19
Jun  7 00:36:33 nnaida19 kernel: IP-Config: Complete:
Jun  7 00:36:33 nnaida19 kernel:      device=eth0, addr=10.1.2.19, mask=255.255.255.0, gw=10.1.2.250,
Jun  7 00:36:33 nnaida19 kernel:      host=nnaida19, domain=, nis-domain=(none),
Jun  7 00:36:34 nnaida19 kernel:      bootserver=10.1.2.250, rootserver=10.1.2.250,
rootpath=/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida26,tcp
Jun  7 00:36:34 nnaida19 kernel: Looking up port of RPC 100003/2 on 10.1.2.250
Jun  7 00:36:34 nnaida19 kernel: Looking up port of RPC 100005/1 on 10.1.2.250
Jun  7 00:36:34 nnaida19 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Jun  7 00:36:34 nnaida19 kernel: Freeing unused kernel memory: 168k init
Jun  7 00:36:35 nnaida19 rpcbind: cannot create socket for udp6
Jun  7 00:36:35 nnaida19 rpcbind: cannot create socket for tcp6
Jun  7 00:36:40 nnaida19 xinetd[275]: IPv6 socket creation failed for service telnet, trying IPv4
Jun  7 00:36:40 nnaida19 xinetd[275]: IPv6 socket creation failed for service ftp, trying IPv4
Jun  7 00:36:40 nnaida19 xinetd[275]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Jun  7 00:36:40 nnaida19 xinetd[275]: Started working: 2 available services
Jun  7 00:36:47 nnaida19 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Jun  7 00:36:48 nnaida19 kernel: XAIDA: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Jun  7 00:36:48 nnaida19 kernel: XAIDA: xaida: driver assigned major number 254
Jun  7 00:36:57 nnaida19 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Jun  7 00:36:57 nnaida19 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Jun  7 00:36:57 nnaida19 kernel: AIDAMEM: aidamem: driver assigned major number 253
Jun  7 00:37:13 nnaida19 kernel: XAIDA: xaida_open:
Jun  7 00:37:13 nnaida19 kernel: AIDAMEM: aidamem_open:
Jun  7 00:47:40 nnaida19 kernel: XAIDA: xaida_open:
Jun  7 11:10:26 nnaida19 xinetd[275]: START: telnet pid=395 from=10.1.2.250
Jun  7 11:11:36 nnaida19 xinetd[275]: EXIT: telnet status=0 pid=395 duration=70(sec)
Jun  7 11:11:56 nnaida19 xinetd[275]: START: telnet pid=397 from=10.1.2.250
  290   Tue Jun 7 10:59:20 2016 TDmerge stats
Let me which merge stats for all data links would be most useful and I will upload screenshots. 
Attachment 1: 100.png
100.png
Attachment 2: 101.png
101.png
  292   Wed Jun 8 02:31:20 2016 TDaidas1 disk statistics
DAQ running to disk at relatively low rate (merger ~600k item/s, tapeserver ~4700kb/s)

Do these disk statistics look OK

aidas1> iostat -x
Linux 2.6.32-504.8.1.el6.x86_64 (aidas1) 	06/08/2016 	_x86_64_	(16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          19.56    3.00    1.15    0.74    0.00   75.55

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.04     0.00    0.01    0.00     0.11     0.00     8.53     0.00    1.37   0.96   0.00
sdc               0.01     0.00    0.01    1.58     0.05     0.00     0.03     0.00    0.38   0.31   0.05
sdg               0.02     0.00    0.18    1.58    42.62     0.01    24.21     0.00    0.59   0.41   0.07
sdf               0.05   507.94    2.16    7.81   516.00  4113.00   464.34     0.69   69.54   4.77   4.76
sdd               0.19     3.95    0.05    0.10     1.36    32.36   225.17     0.00    6.63   3.57   0.05
sdb               1.40    38.38    2.17    3.35   185.34   332.20    93.75     0.14   26.19   2.60   1.44
sde               0.01     4.25    3.54    4.94    64.06    60.96    14.74     0.30   35.48   3.90   3.30
dm-0              0.00     0.00    3.56   41.53   185.07   332.20    11.47    12.54  278.00   0.32   1.44
dm-1              0.00     0.00    0.01    0.00     0.05     0.00     8.00     0.00    3.24   1.03   0.00
dm-2              0.00     0.00    0.15    4.05     1.23    32.36     8.00     0.05   11.49   0.12   0.05
dm-3              0.00     0.00    0.00    0.00     0.02     0.00     8.00     0.00    1.05   1.05   0.00
dm-4              0.00     0.00    5.92  522.90   622.57  4174.35     9.07    59.55  112.60   0.15   7.94
dm-5              0.00     0.00    0.01    0.00     0.07     0.00     7.95     0.00    3.11   0.69   0.00

aidas1> iostat
Linux 2.6.32-504.8.1.el6.x86_64 (aidas1) 	06/08/2016 	_x86_64_	(16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          19.55    3.00    1.15    0.74    0.00   75.57

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.01         0.11         0.00       5682          0
sdc               1.58         0.05         0.00       2312        112
sdg               1.76        42.56         0.01    2170822        648
sdf               9.98       519.68      4114.55   26505990  209861264
sdd               0.15         1.38        32.32      70150    1648496
sdb               5.52       185.27       331.95    9449724   16931146
sde               8.47        63.97        60.90    3263022    3106264
dm-0             45.06       185.01       331.95    9436314   16931104
dm-1              0.01         0.05         0.00       2696          0
dm-2              4.20         1.24        32.32      63432    1648496
dm-3              0.00         0.02         0.00        864          0
dm-4            528.96       626.10      4175.46   31934274  212968288
dm-5              0.01         0.07         0.00       3778         24
  294   Wed Jun 8 03:57:59 2016 TDAnalysis of R1222
Timestamp data per FEE per 10s (i.e. 1 ch = 10s)

s3501-s3524 ADC data
s3601-s3624 PAUSE
s3701-s3724 RESUME
s3801-s3824 SYNC

Little evidence of data loss or high dead time
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
  295   Wed Jun 8 04:24:04 2016 TDaidas1 disk statistics (without data compression)
aidas1> iostat -xn
Linux 2.6.32-504.8.1.el6.x86_64 (aidas1) 	06/08/2016 	_x86_64_	(16 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.03     0.00    0.01    0.00     0.10     0.00     8.53     0.00    1.37   0.96   0.00
sdc               0.01     0.00    0.01    1.46     0.04     0.00     0.03     0.00    0.38   0.31   0.05
sdg               0.02     0.00    1.16    1.46   280.02     0.01   106.79     0.00    1.05   0.74   0.20
sdf               0.23   496.52    6.59    7.52  1630.98  4020.60   400.58     0.72   50.70   3.81   5.38
sdd               0.20     3.48    0.05    0.09     1.50    28.52   217.54     0.00    6.63   3.64   0.05
sdb               1.39    35.64    2.06    3.35   171.52   310.29    89.17     0.13   24.09   2.59   1.40
sde               0.01     3.89    3.13    4.56    56.50    56.00    14.63     0.27   35.22   3.91   3.01
dm-0              0.00     0.00    3.45   38.79   171.28   310.29    11.40    11.07  261.99   0.33   1.40
dm-1              0.00     0.00    0.01    0.00     0.05     0.00     8.00     0.00    3.24   1.03   0.00
dm-2              0.00     0.00    0.17    3.57     1.39    28.52     8.00     0.04   11.44   0.13   0.05
dm-3              0.00     0.00    0.00    0.00     0.01     0.00     8.00     0.00    1.05   1.05   0.00
dm-4              0.00     0.00   11.11  510.60  1967.40  4076.62    11.59    60.80  116.54   0.16   8.35
dm-5              0.00     0.00    0.01    0.00     0.07     0.00     7.96     0.00    3.64   0.68   0.00

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s     ops/s 
  rops/s    wops/s

aidas1> iostat -x
Linux 2.6.32-504.8.1.el6.x86_64 (aidas1) 	06/08/2016 	_x86_64_	(16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.42    3.35    1.10    0.72    0.00   77.41

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.03     0.00    0.01    0.00     0.10     0.00     8.53     0.00    1.37   0.96   0.00
sdc               0.01     0.00    0.01    1.46     0.04     0.00     0.03     0.00    0.38   0.31   0.05
sdg               0.02     0.00    1.16    1.46   279.98     0.01   106.78     0.00    1.05   0.74   0.20
sdf               0.23   496.63    6.62    7.52  1636.72  4021.54   400.39     0.72   50.64   3.81   5.39
sdd               0.20     3.48    0.05    0.09     1.50    28.52   217.54     0.00    6.63   3.64   0.05
sdb               1.39    35.64    2.06    3.35   171.49   310.28    89.16     0.13   24.09   2.59   1.40
sde               0.01     3.89    3.13    4.56    56.49    56.00    14.63     0.27   35.22   3.91   3.01
dm-0              0.00     0.00    3.45   38.78   171.26   310.28    11.40    11.06  261.97   0.33   1.40
dm-1              0.00     0.00    0.01    0.00     0.05     0.00     8.00     0.00    3.24   1.03   0.00
dm-2              0.00     0.00    0.17    3.56     1.39    28.52     8.00     0.04   11.44   0.13   0.05
dm-3              0.00     0.00    0.00    0.00     0.01     0.00     8.00     0.00    1.05   1.05   0.00
dm-4              0.00     0.00   11.13  510.72  1973.09  4077.56    11.59    60.83  116.57   0.16   8.36
dm-5              0.00     0.00    0.01    0.00     0.07     0.00     7.96     0.00    3.64   0.68   0.00
  296   Wed Jun 8 05:00:53 2016 TDAnalysis of R1229
Files R1229_0 ... R1229_3 (written by TapeServer without data compression)

Timestamp data per FEE per 10s (i.e. 1 ch = 10s)

s3501-s3524 ADC data
s3601-s3624 PAUSE
s3701-s3724 RESUME
s3801-s3824 SYNC
Attachment 1: 10.png
10.png
Attachment 2: 11.png
11.png
Attachment 3: 12.png
12.png
Attachment 4: 13.png
13.png
  297   Wed Jun 8 09:29:55 2016 TDDAQ state of play
17.25 State of play

      Running data to disk (uncompressed)

      link #18 (nnaida19 disabled)

      ~1.3m data items/s
      ~10.5Mb/s

      merge64.AD ~30% CPU
      driver ~4% CPU
      link64 <1% CPU

      nnaida19 ~20Hz PAUSE, increase LEC/MEC fast comparator from 0x5 to 0x20 -> PAUSE ~0Hz
Attachment 1: 20.png
20.png
Attachment 2: 21.png
21.png
Attachment 3: 22.png
22.png
Attachment 4: 23.png
23.png
Attachment 5: 24.png
24.png
Attachment 6: 30.png
30.png
Attachment 7: 31.png
31.png
Attachment 8: 32.png
32.png
Attachment 9: 34.png
34.png
  302   Fri Jun 10 12:51:19 2016 TDADC offsets - pass 1 -18.15 3 June 2016
Column order:

Channel = (fee# - 1)*64 + (asic# - 1)*16 + channel
FEE
ASIC
Channel
Offset (ADC channels)
Offset error (ADC channels)
INL (%)
# peaks
Spectrum name

ADC offset to be used as follows:

 adc(i) = INT( ABS( raw_adc(i) - 32768 ) - offset(i) + 0.5 )

This is a first pass analysis with ~90% of the ADC offsets and something approaching ~95% achievable
(64 channels of nnaida19 not operational).
 
Recommend ignoring those with INL > 1%, offsets > 500 channels
Attachment 1: offsets.dat
   0  1  1  0     221.27     0.49  -0.590   9 1.0.L                           
   1  1  1  1      96.21     0.96  -0.696   7 1.1.L                           
   2  1  1  2      73.14     0.48  -0.660   9 1.2.L                           
   3  1  1  3     254.76     0.48  -0.820   9 1.3.L                           
   4  1  1  4      64.25     0.49  -0.906   9 1.4.L                           
   5  1  1  5     316.39     0.48  -0.973   9 1.5.L                           
   6  1  1  6     156.65     0.48  -0.721   9 1.6.L                           
   7  1  1  7     150.55     0.48  -0.963   9 1.7.L                           
   8  1  1  8      59.64     0.47  -0.609   9 1.8.L                           
   9  1  1  9     237.20     0.48  -0.885   9 1.9.L                           
  10  1  1 10      32.75     0.47  -0.780   9 1.10.L                          
  11  1  1 11     115.78     0.47  -0.633   9 1.11.L                          
  12  1  1 12     207.48     0.63  -0.831   8 1.12.L                          
  13  1  1 13      96.96     0.46  -0.554   9 1.13.L                          
  14  1  1 14      60.74     0.47  -0.551   9 1.14.L                          
  15  1  1 15     932.86     3.62  -2.441   3 1.15.L INL > 1%                 
  16  1  2  0    -212.45     0.46  -0.802   9 2.0.L                           
  17  1  2  1    1295.62     0.50  -0.790   8 2.1.L                           
  18  1  2  2    -468.76     3.91  -3.122   3 2.2.L  INL > 1%                 
  19  1  2  3     140.15     0.45  -0.325   9 2.3.L                           
  20  1  2  4     -43.74     0.46  -0.602   9 2.4.L                           
  21  1  2  5    -859.66     3.94  -3.295   3 2.5.L  INL > 1%                 
  22  1  2  6      -3.00     0.47  -0.449   9 2.6.L                           
  23  1  2  7     -66.06     0.45  -0.844   9 2.7.L                           
  24  1  2  8    -420.04     3.64  -4.023   3 2.8.L  INL > 1%                 
  25  1  2  9    -801.25     3.86  -3.924   3 2.9.L  INL > 1%                 
  26  1  2 10     936.38     1.26 -14.980   3 2.10.L INL > 1%                 
  27  1  2 11    -581.39     3.66  -4.018   3 2.11.L INL > 1%                 
  28  1  2 12   -1109.67     3.84  -5.496   3 2.12.L INL > 1%                 
  29  1  2 13   -8728.93     4.31  -2.102   3 2.13.L INL > 1%                 
  30  1  2 14    5373.29     1.33 -14.201   3 2.14.L INL > 1%                 
  31  1  2 15   -3621.62     3.67  -4.831   3 2.15.L INL > 1%                 
  32  1  3  0    2825.79     3.75  -6.178   3 3.0.L  INL > 1%                 
  33  1  3  1    6377.89     1.50 -12.869   3 3.1.L  INL > 1%                 
  34  1  3  2   -7968.25     2.92  -0.503   4 3.2.L                           
  35  1  3  3    1059.21     3.81  -3.203   3 3.3.L  INL > 1%                 
  36  1  3  4    1188.25     3.67  -0.424   3 3.4.L                           
  37  1  3  5     812.02     3.76  -0.411   3 3.5.L                           
  38  1  3  6     402.38     2.35  -0.884   4 3.6.L                           
  39  1  3  7      31.54     0.91  -0.573   4 3.7.L                           
  40  1  3  8    1175.06     3.21  -0.495   3 3.8.L                           
  41  1  3  9     686.01     2.24  -0.827   4 3.9.L                           
  42  1  3 10     192.03     0.46  -0.766   9 3.10.L                          
  43  1  3 11     826.79     3.71  -0.598   3 3.11.L                          
  44  1  3 12     -15.18     0.47  -0.920   9 3.12.L                          
  45  1  3 13      -8.05     0.45  -0.897   9 3.13.L                          
  46  1  3 14   -1391.29     0.55  -0.538   9 3.14.L                          
  47  1  3 15  -44213.33     3.89 -27.410   3 3.15.L INL > 1%                 
  48  1  4  0   -1231.94     1.11  -0.559   6 4.0.L                           
  49  1  4  1   -1375.25     0.73  -0.903   8 4.1.L                           
  50  1  4  2   -1343.70     0.56  -0.600   9 4.2.L                           
  51  1  4  3   -1312.69     1.16  -0.251   6 4.3.L                           
  52  1  4  4  -48308.69     4.13 -27.547   3 4.4.L  INL > 1%                 
  53  1  4  5   -1140.83     0.74  -0.618   8 4.5.L                           
  54  1  4  6   -9999.99     0.00   0.000  37 4.6.L  > 10 peaks or no data    
  56  1  4  8     313.08     0.49  -0.721   9 4.8.L                           
  57  1  4  9   -9999.99     0.00   0.000  37 4.9.L  > 10 peaks or no data    
  59  1  4 11     418.03     0.49  -0.619   9 4.11.L                          
  60  1  4 12   -9999.99     0.00   0.000  37 4.12.L > 10 peaks or no data    
  61  1  4 13     257.28     0.80  -0.701   9 4.13.L                          
  62  1  4 14   -9999.99     0.00   0.000  37 4.14.L > 10 peaks or no data    
  63  1  4 15     262.00     0.52  -0.568   9 4.15.L                          
  64  2  1  0     227.18     0.47  -0.466   9 1.0.L                           
  65  2  1  1     278.51     0.58  -0.861   9 1.1.L                           
  66  2  1  2     -38.23     0.48  -0.466   9 1.2.L                           
  67  2  1  3      62.61     0.47  -0.495   9 1.3.L                           
  68  2  1  4     209.07     0.49  -0.608   9 1.4.L                           
  69  2  1  5     164.35     0.48  -0.534   9 1.5.L                           
  70  2  1  6      33.05     0.48  -0.531   9 1.6.L                           
  71  2  1  7     468.91     0.48  -0.559   9 1.7.L                           
  72  2  1  8     232.56     0.73  -0.823   9 1.8.L                           
  73  2  1  9   -9999.99     0.00   0.000  37 1.9.L  > 10 peaks or no data    
  74  2  1 10     297.71     1.27  -0.840   7 1.10.L                          
  75  2  1 11     194.00     0.48  -0.495   9 1.11.L                          
  76  2  1 12      52.37     0.48  -0.509   9 1.12.L                          
  77  2  1 13     224.36     0.46  -0.480   9 1.13.L                          
  78  2  1 14     -19.41     0.49  -0.443   9 1.14.L                          
  79  2  1 15     -92.64     0.45  -0.347   9 1.15.L                          
  80  2  2  0     -18.96     0.45  -0.343   9 2.0.L                           
  81  2  2  1     -11.28     0.46  -0.361   9 2.1.L                           
  82  2  2  2     -55.97     0.46  -0.377   9 2.2.L                           
  83  2  2  3      56.49     0.44  -0.351   9 2.3.L                           
  84  2  2  4      66.61     0.45  -0.352   9 2.4.L                           
  85  2  2  5    -254.65     0.44  -0.388   9 2.5.L                           
  86  2  2  6     -19.33     0.44  -0.401   9 2.6.L                           
  87  2  2  7     -47.39     0.46  -0.387   9 2.7.L                           
  88  2  2  8     -35.59     0.44  -0.341   9 2.8.L                           
  89  2  2  9    -171.18     0.45  -0.405   9 2.9.L                           
  90  2  2 10      12.24     0.44  -0.368   9 2.10.L                          
  91  2  2 11       7.99     0.43  -0.375   9 2.11.L                          
  92  2  2 12    -102.26     0.45  -0.339   9 2.12.L                          
  93  2  2 13     -58.76     0.45  -0.161   9 2.13.L                          
  94  2  2 14     -16.78     0.40  -0.209   9 2.14.L                          
  95  2  2 15      54.92     0.41  -0.312   9 2.15.L                          
  96  2  3  0     433.30     0.45  -0.332   9 3.0.L                           
  97  2  3  1     142.86     0.45  -0.372   9 3.1.L                           
  98  2  3  2     114.43     0.46  -0.199   9 3.2.L                           
  99  2  3  3     312.14     0.45  -0.266   9 3.3.L                           
 100  2  3  4     185.36     0.46  -0.364   9 3.4.L                           
 101  2  3  5     137.18     0.46  -0.328   9 3.5.L                           
 102  2  3  6     258.04     0.45  -0.270   9 3.6.L                           
 103  2  3  7     284.53     0.45  -0.260   9 3.7.L                           
 104  2  3  8     115.77     0.46  -0.448   9 3.8.L                           
 105  2  3  9     285.40     0.46  -0.355   9 3.9.L                           
 106  2  3 10       2.38     0.47  -0.331   9 3.10.L                          
 107  2  3 11      28.40     0.47  -0.501   9 3.11.L                          
 108  2  3 12     228.77     0.47  -0.343   9 3.12.L                          
 109  2  3 13     318.35     0.46  -0.379   9 3.13.L                          
 110  2  3 14      76.69     0.45  -0.258   9 3.14.L                          
 111  2  3 15     222.08     0.46  -0.391   9 3.15.L                          
 112  2  4  0     263.39     0.48  -0.267   9 4.0.L                           
 113  2  4  1    -261.43     0.48  -0.353   9 4.1.L                           
 114  2  4  2    -127.19     0.47  -0.301   9 4.2.L                           
 115  2  4  3      23.73     0.48  -0.356   9 4.3.L                           
 116  2  4  4     -51.36     0.48  -0.415   9 4.4.L                           
 117  2  4  5     121.99     0.50  -0.423   9 4.5.L                           
 118  2  4  6     134.29     0.53  -0.386   9 4.6.L                           
 119  2  4  7   -9999.99     0.00   0.000  37 4.7.L  > 10 peaks or no data    
 120  2  4  8     203.54     0.49  -0.387   9 4.8.L                           
 121  2  4  9     340.38     0.51  -0.434   9 4.9.L                           
 122  2  4 10   -9999.99     0.00   0.000  37 4.10.L > 10 peaks or no data    
 123  2  4 11     129.85     0.48  -0.366   9 4.11.L                          
 124  2  4 12     210.80     0.49  -0.352   9 4.12.L                          
 125  2  4 13      42.06     0.77  -0.318   9 4.13.L                          
 126  2  4 14     171.05     0.48  -0.312   9 4.14.L                          
 127  2  4 15     -32.40     0.47  -0.307   9 4.15.L                          
 128  3  1  0    -519.64     1.60   6.862   3 1.0.L  INL > 1%                 
 129  3  1  1    -217.21     1.59   0.926   5 1.1.L                           
 130  3  1  2    -165.60     0.55   0.532   9 1.2.L                           
 131  3  1  3      11.05     0.56   0.604   9 1.3.L                           
 132  3  1  4       5.91     2.23   0.146   3 1.4.L                           
 133  3  1  5     -39.75     0.53   0.630   9 1.5.L                           
 134  3  1  6    -172.29     0.53   0.670   9 1.6.L                           
 135  3  1  7    -325.49     0.54   0.663   9 1.7.L                           
 136  3  1  8     -83.54     0.51   0.694   9 1.8.L                           
 137  3  1  9    -217.62     0.51   0.644   9 1.9.L                           
 138  3  1 10    -183.08     0.51   0.671   9 1.10.L                          
 139  3  1 11     171.84     0.52   0.665   9 1.11.L                          
 140  3  1 12    -156.49     0.50   0.557   9 1.12.L                          
 141  3  1 13    -207.29     0.52   0.583   9 1.13.L                          
 142  3  1 14     -12.61     0.50   0.522   9 1.14.L                          
 143  3  1 15    -210.33     0.49   0.528   9 1.15.L                          
 144  3  2  0    -334.03     0.48   0.572   9 2.0.L                           
 145  3  2  1    -232.98     0.46   0.567   9 2.1.L                           
 146  3  2  2    -139.49     0.48   0.570   9 2.2.L                           
 147  3  2  3    -183.13     0.46   0.605   9 2.3.L                           
 148  3  2  4    -136.91     0.46   0.521   9 2.4.L                           
 149  3  2  5     -86.13     0.47   0.590   9 2.5.L                           
 150  3  2  6     -17.87     0.47   0.587   9 2.6.L                           
 151  3  2  7    -492.32     0.46   0.734   9 2.7.L                           
 152  3  2  8    -134.59     0.46   0.715   9 2.8.L                           
 153  3  2  9    -334.83     0.48   0.564   9 2.9.L                           
 154  3  2 10     -76.65     0.46   0.788   9 2.10.L                          
 155  3  2 11      -1.23     0.47   0.623   9 2.11.L                          
 156  3  2 12    -392.22     0.47   0.563   9 2.12.L                          
 157  3  2 13    -154.27     0.48   0.617   9 2.13.L                          
 158  3  2 14    -197.34     0.50   0.195   8 2.14.L                          
 159  3  2 15     -57.01     0.42   0.571   9 2.15.L                          
 160  3  3  0    -606.06     0.87  10.505   3 3.0.L  INL > 1%                 
 161  3  3  1   -3789.71     1.68   2.064   3 3.1.L  INL > 1%                 
 162  3  3  2     -96.46     0.50   0.949   9 3.2.L                           
 163  3  3  3     -89.01     0.50   0.720   9 3.3.L                           
 164  3  3  4    -210.86     0.49   0.674   9 3.4.L                           
 165  3  3  5      20.13     0.49   0.643   9 3.5.L                           
 166  3  3  6    -824.38     1.70  12.398   3 3.6.L  INL > 1%                 
 167  3  3  7    -569.86     1.48   5.507   3 3.7.L  INL > 1%                 
 168  3  3  8    -347.62     1.28  12.958   3 3.8.L  INL > 1%                 
 169  3  3  9     -18.29     0.49   0.682   9 3.9.L                           
 170  3  3 10      -6.87     0.49   0.716   9 3.10.L                          
 171  3  3 11    -211.50     1.07  10.153   3 3.11.L INL > 1%                 
 172  3  3 12    -300.73     0.49   0.610   9 3.12.L                          
 173  3  3 13    -278.88     0.48   0.640   9 3.13.L                          
 174  3  3 14     -66.74     0.51   0.501   9 3.14.L                          
 175  3  3 15      41.47     0.53   0.493   9 3.15.L                          
 176  3  4  0   -1198.10     2.48  21.463   3 4.0.L  INL > 1%                 
 177  3  4  1    -280.37     0.51   0.505   9 4.1.L                           
 178  3  4  2     -61.89     0.53   0.416   9 4.2.L                           
 179  3  4  3     -32.21     0.48   0.560   9 4.3.L                           
 180  3  4  4    -143.35     0.55   0.558   9 4.4.L                           
 181  3  4  5    -176.09     0.50   0.683   9 4.5.L                           
 182  3  4  6    -112.67     0.49   0.708   9 4.6.L                           
 183  3  4  7   -1929.12     2.29   8.062   3 4.7.L  INL > 1%                 
 184  3  4  8      85.28     0.52   0.519   9 4.8.L                           
 185  3  4  9    -177.96     0.50   0.699   9 4.9.L                           
 186  3  4 10    -172.42     0.53   0.503   9 4.10.L                          
 187  3  4 11      -5.35     0.50   0.562   9 4.11.L                          
 188  3  4 12    -123.40     0.50   0.595   9 4.12.L                          
 189  3  4 13     -13.68     0.49   0.443   9 4.13.L                          
 190  3  4 14    -311.86     0.51   0.519   9 4.14.L                          
 191  3  4 15     -20.82     0.54   0.397   9 4.15.L                          
 192  4  1  0    -175.71     0.63   0.632   9 1.0.L                           
 193  4  1  1    -169.28     0.88   0.711   9 1.1.L                           
 194  4  1  2      36.57     0.57   0.588   9 1.2.L                           
 195  4  1  3    -142.43     0.58   0.728   9 1.3.L                           
 196  4  1  4     -60.44     0.59   0.695   9 1.4.L                           
 197  4  1  5     -46.68     0.54   0.670   9 1.5.L                           
 198  4  1  6     -91.09     0.56   0.742   9 1.6.L                           
 199  4  1  7    -244.52     0.57   0.878   9 1.7.L                           
 200  4  1  8    -136.10     0.54   0.846   9 1.8.L                           
 201  4  1  9    -139.78     0.54   0.811   9 1.9.L                           
 202  4  1 10    -189.83     0.54   0.698   9 1.10.L                          
 203  4  1 11    -103.95     0.56   0.697   9 1.11.L                          
 204  4  1 12    -145.52     0.52   0.594   9 1.12.L                          
 205  4  1 13     -77.24     0.51   0.656   9 1.13.L                          
 206  4  1 14     115.73     0.54   0.500   9 1.14.L                          
 207  4  1 15      34.41     0.53   0.546   9 1.15.L                          
 208  4  2  0      12.08     0.51   0.747   9 2.0.L                           
 209  4  2  1     -87.60     0.53   0.548   9 2.1.L                           
 210  4  2  2     165.38     0.51   0.612   9 2.2.L                           
 211  4  2  3     -92.40     0.52   0.679   9 2.3.L                           
 212  4  2  4      38.84     0.51   0.722   9 2.4.L                           
 213  4  2  5     -84.11     0.52   0.836   9 2.5.L                           
 214  4  2  6      67.80     0.50   0.758   9 2.6.L                           
 215  4  2  7    -142.00     0.56   0.634   9 2.7.L                           
 216  4  2  8     -36.78     0.53   0.647   9 2.8.L                           
 217  4  2  9     -14.18     0.50   0.709   9 2.9.L                           
 218  4  2 10    -209.22     0.51   0.787   9 2.10.L                          
 219  4  2 11     -34.54     0.48   0.758   9 2.11.L                          
 220  4  2 12      89.64     0.50   0.640   9 2.12.L                          
 221  4  2 13      86.28     0.53   0.738   9 2.13.L                          
 222  4  2 14    -248.70     0.49   0.580   9 2.14.L                          
 223  4  2 15     -89.65     0.47   0.627   9 2.15.L                          
 224  4  3  0    -541.15     1.02   7.322   3 3.0.L  INL > 1%                 
 225  4  3  1     -57.44     0.52   0.690   9 3.1.L                           
 226  4  3  2     -61.68     0.53   0.537   9 3.2.L                           
 227  4  3  3      41.12     0.53   0.504   9 3.3.L                           
 228  4  3  4      72.75     0.50   0.495   9 3.4.L                           
 229  4  3  5     134.73     0.52   0.613   9 3.5.L                           
 230  4  3  6     150.77     0.53   0.597   9 3.6.L                           
 231  4  3  7     -36.75     0.52   0.675   9 3.7.L                           
 232  4  3  8    -628.38     0.80   4.788   3 3.8.L  INL > 1%                 
 233  4  3  9    -238.74     0.52   0.659   9 3.9.L                           
 234  4  3 10       1.92     0.50   0.575   9 3.10.L                          
 235  4  3 11    -252.34     0.52   0.561   9 3.11.L                          
 236  4  3 12      52.68     0.51   0.605   9 3.12.L                          
 237  4  3 13      80.91     0.51   0.547   9 3.13.L                          
 238  4  3 14     157.18     0.51   0.553   9 3.14.L                          
 239  4  3 15    -140.31     0.51   0.441   9 3.15.L                          
 240  4  4  0    -137.72     0.52   0.634   9 4.0.L                           
 241  4  4  1     -86.83     0.51   0.641   9 4.1.L                           
 242  4  4  2      70.27     0.50   0.668   9 4.2.L                           
 243  4  4  3     -19.07     0.51   0.573   9 4.3.L                           
 244  4  4  4    -220.03     0.52   0.656   9 4.4.L                           
 245  4  4  5    -197.80     0.52   0.561   9 4.5.L                           
 246  4  4  6    -190.22     0.52   0.709   9 4.6.L                           
 247  4  4  7    -277.14     0.51   0.442   9 4.7.L                           
 248  4  4  8    -195.68     0.51   0.702   9 4.8.L                           
 249  4  4  9    -239.19     0.52   0.731   9 4.9.L                           
 250  4  4 10     -76.81     0.50   0.459   9 4.10.L                          
 251  4  4 11     -25.19     0.52   0.681   9 4.11.L                          
 252  4  4 12    -219.24     0.53   0.587   9 4.12.L                          
 253  4  4 13     -50.33     0.51   0.465   9 4.13.L                          
 254  4  4 14       1.96     0.54   0.538   9 4.14.L                          
 255  4  4 15    -331.97     0.52   0.457   9 4.15.L                          
 256  5  1  0     258.71     0.48  -0.433   9 1.0.L                           
 257  5  1  1      44.40     0.59  -0.711   9 1.1.L                           
 258  5  1  2     145.79     0.47  -0.498   9 1.2.L                           
 259  5  1  3      -2.75     0.48  -0.355   9 1.3.L                           
 260  5  1  4     342.67     0.48  -0.511   9 1.4.L                           
 261  5  1  5     278.31     0.49  -0.489   9 1.5.L                           
 262  5  1  6     117.25     0.47  -0.523   9 1.6.L                           
 263  5  1  7     253.76     0.49  -0.400   9 1.7.L                           
 264  5  1  8     275.29     0.48  -0.305   9 1.8.L                           
 265  5  1  9      87.88     0.47  -0.406   9 1.9.L                           
 266  5  1 10     129.59     0.48  -0.388   9 1.10.L                          
 267  5  1 11     120.29     0.48  -0.426   9 1.11.L                          
 268  5  1 12     180.13     0.47  -0.420   9 1.12.L                          
 269  5  1 13     148.06     0.46  -0.299   9 1.13.L                          
 270  5  1 14      74.49     0.48  -0.360   9 1.14.L                          
 271  5  1 15      21.34     0.46  -0.330   9 1.15.L                          
 273  5  2  1      78.70     0.46  -0.309   9 2.1.L                           
 274  5  2  2     349.48     0.46  -0.289   9 2.2.L                           
 275  5  2  3     -32.10     0.46  -0.364   9 2.3.L                           
 276  5  2  4     -13.87     0.45  -0.232   9 2.4.L                           
 277  5  2  5     154.01     0.47  -0.310   9 2.5.L                           
 278  5  2  6     105.69     0.45  -0.281   9 2.6.L                           
 279  5  2  7     277.13     0.45  -0.246   9 2.7.L                           
 280  5  2  8    -204.23     0.45  -0.327   9 2.8.L                           
 281  5  2  9     -42.14     0.46  -0.276   9 2.9.L                           
 282  5  2 10      44.38     0.45  -0.525   9 2.10.L                          
 284  5  2 12     -54.70     0.44  -0.248   9 2.12.L                          
 285  5  2 13     297.19     0.45  -0.552   9 2.13.L                          
 286  5  2 14     -83.88     0.43  -0.408   9 2.14.L                          
 287  5  2 15     119.39     0.39  -0.541   9 2.15.L                          
 288  5  3  0    -236.27     0.46  -0.869   9 3.0.L                           
 289  5  3  1    -329.49     0.47  -0.283   9 3.1.L                           
 290  5  3  2    -248.64     0.47  -0.348   9 3.2.L                           
 291  5  3  3    -269.61     0.49  -0.859   9 3.3.L                           
 292  5  3  4    9447.04     5.83 -33.252   3 3.4.L  INL > 1%                 
 293  5  3  5     -79.92     0.48  -0.288   9 3.5.L                           
 294  5  3  6    -143.18     0.48  -0.384   9 3.6.L                           
 295  5  3  7      53.67     0.47  -0.310   9 3.7.L                           
 296  5  3  8    -291.96     0.46  -0.384   9 3.8.L                           
 297  5  3  9    -303.34     0.48  -0.232   9 3.9.L                           
 298  5  3 10    -176.21     0.49  -0.242   9 3.10.L                          
 299  5  3 11    -132.74     0.93  -0.396   6 3.11.L                          
 300  5  3 12     -80.88     0.47  -0.155   9 3.12.L                          
 301  5  3 13    -136.35     0.46  -0.173   9 3.13.L                          
 302  5  3 14     -29.55     0.46  -0.229   9 3.14.L                          
 303  5  3 15    -653.05     0.95  -0.302   6 3.15.L                          
... 1217 more lines ...
  304   Fri Jun 10 14:14:56 2016 TDnnaida19
See attachments 1 and 2 and spot the difference ...

Yes - nnaida19 histogramming appears to have started working as mysteriously as it stopped.

However, although nnaida19 ReSYNCs the timestamp appears to wander (see attachment 3) and
nnaida19 still fails the ASIC timestamp and SYNC errors not counting system-wide tests.
Ideas anyone?
Attachment 1: 4.png
4.png
Attachment 2: 5.png
5.png
Attachment 3: 6.png
6.png
  308   Sat Jun 11 15:48:49 2016 TD[How To] Monitor Pause-Resume Times
The basic GREAT data format analyser will report per FEE cumulative
pause-resume times which should be << timestamp elapsed time (which
is time taken to write the data file analysed).

aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA40_52
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  147698907 (  924090.7 Hz)
        Other data format:  113421093 (  709628.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       6592 (      41.2 Hz)
                               RESUME:       6592 (      41.2 Hz)
                              SYNC100:      60971 (     381.5 Hz)
                           FEE64 disc:  112649552 (  704801.4 Hz)
                             MBS info:     697386 (    4363.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     312853 (    1957.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      159.832 s
 FEE module #:   1 elapsed dead time       0.030 s
 FEE module #:   2 elapsed dead time       0.291 s
 FEE module #:   3 elapsed dead time       0.029 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.132 s
 FEE module #:   6 elapsed dead time       0.078 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.041 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.099 s
 FEE module #:  12 elapsed dead time       0.038 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.008 s
 FEE module #:  15 elapsed dead time       0.074 s
 FEE module #:  16 elapsed dead time       1.036 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.026 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.247 s
 FEE module #:  21 elapsed dead time       0.421 s
 FEE module #:  22 elapsed dead time       0.024 s
 FEE module #:  23 elapsed dead time       1.628 s
 FEE module #:  24 elapsed dead time       0.493 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   26.523s ( 1206.480 blocks/s,  75.405 Mb/s)
  309   Sat Jun 11 16:06:16 2016 TD[How To] Monitor per FEE64 Activity
Launch terminal and enter commands

setenv MIDASBASE /MIDAS@aidas.240215
setenv PATH $MIDASBASE/bin_Linux64:$PATH
MIDAS64-session

From the pop-up select 'MIDAS Data Analysis'

From 'MIDAS Base Frame' select 'Sort Tool' 

From 'MIDAS Sort (64 bit) Control Frame' select 'Browse' and select 'sort2b.f' from the pop-up

From 'MIDAS Sort (64 bit) Control Frame' select 'Compile Sortfile' 

From 'MIDAS Sort (64 bit) Control Frame' select 'Disk Info'

From 'MIDASsort Disk Info' select 'Browse' and select the latest complete file from the current 
run from the data dietcory (currently /TapeData/NP1512)

From 'MIDASsort Disk Info' select 'Add' 

From 'MIDASsort Disk Info' select 'Quit' 

From 'MIDAS Sort (64 bit) Control Frame' select 'Start'
  The terminal from which MIDAS was launched will report messages from MIDASsort
  and end of sort

From 'MIDAS Sort (64 bit) Control Frame' select 'View Spectra'

From 'Sort Spectrum Directory Browser' select spectra s3501-s3524 which display
spectra showing (ADC) timestamps per FEE (s3501 = nnaida1, s3502 = nnaida2 etc)

The 65k spectra are rolling and the scale is 1ch = 0.1s.

Examine the activity of each FEE for extended (>1s) periods of inactivity.  
  
When the sort is complete from 'MIDAS Sort (64 bit) Control Frame' select 'Pause' then 'Kill'



  
  313   Mon Jun 13 03:45:15 2016 TDMonday 13 June 2016
10.20 AIDA62 found stalled - no SYNCs, no data transfer
      All FEE64s pass all system-wide tests *except* nnaida3 & nnaida8 which fail ADC calibration
      Data files indicate DAQ stopped c. 06.56
      DAQ stop and file closed

10.30 Detector bias & leakage currents, trigger rates and FEE64 temperatures - see attcahments 1-4
      Julabo FL11006 set point +17 deg C (B2F/F11 ambient temperature +25.3 deg C, d.p. +11.6 deg C, RH 42.7%)

10.48 AIDA63
      Pulser walkthrough

      BNC PB4 settings

      Amplitudes 10,000-90,000 @ 10,000 step
      x5 attenuator IN
      Frequency 250Hz range 
      t_r 50ns tau_d 50us
      - polarity (+ polarity via EG&G Ortec 433A Sum & Invert Amp)
      INT ref, tail pulse, MIN delay

      ASIC settings 2016Jun12-09.09.37

      See attachment 5 showing 9x pulser peaks, noise/beta/background and low amplitude pulser-related peak

      Spectra saved to /MIDAS/SpecData c. 11.00 13 June 2016
      Note directory names are derived from internal FEE64 date/time settings which be wrong or unset

      All spectra cleared

11.06 AIDA64
     
      Per AIDA63 *except*
  
      BNC PB-4 amplitudes

      10,000-2,000 @ 2,000 step

      Spectra saved to /MIDAS/SpecData c. 11.13 13 June 2016

      Note that not all pulse peak amplitudes will be above the slow comparator threshold 

      Attachments 6-10 hit/stat spectra and example energy spectra

EURICA standalone gamma energy and efficiency measurements using 152Eu and 133Ba.
The gamma-ray sources were placed immediately upstream of DSSSD#1 inside the AIDA 
snout.

16.00 Obtained 'unable to connect' to nnaida1 through multiple power cycles/reboots and restarts
      of MERGE unable to clear problem. Finally reosrted to a reboot of aidas1 which did 'fix' the
      problem.

      AIDA65
      Brief test run  to check DAQ/merger/TapeServer - OK

16.06 AIDA66 (AIDA) 1 (EURICA)

      ~11kBq (current activity) 60Co source immediately upstream of DSSSD#1 inside the AIDA snout.

      ASIC settings 2016Jun12-09.09.37

      FADC disabled

      BNC PB-4 pulser
      Amplitude 90,000
      Frequency 25Hz range (probably ~1-few Hz)

      Attachments 11-12 Detector biases & leakage currents -OK

16.37 Attachment 13 - SyncCheck plots

16.50 FR adds ~40kBq 60Co source on top of the AIDA snout immediately above the DSSSDs
      DAQ continues

aidas1> ~/td/GREAT/analyser2 AIDA66_17
 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***   SYNC100: block:     1 ptr:  5567 data: 0x82400595 module:  2 information type:  4 information field:
0x00000595 ts: 0x000000595DD400A0 OLD: ts: 0x07158F300DD4009D tag: 3 type: 0
 *** FEE64 discriminator: block: 15627 ptr: 14341 data: 0x97620008 module: 23 information type:  6 information
field: 0x00020008 ts: 0x005B300000A0 OLD: ts: 0x005B40000050 tag: 2 type: 6
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  161285942 (  973887.8 Hz)
        Other data format:   99834058 (  602824.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        118 (       0.7 Hz)
                               RESUME:        118 (       0.7 Hz)
                              SYNC100:      63176 (     381.5 Hz)
                           FEE64 disc:   98983953 (  597691.6 Hz)
                             MBS info:     786693 (    4750.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      26988 (     163.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      165.610 s
 FEE module #:   1 elapsed dead time       0.378 s
 FEE module #:   2 elapsed dead time       0.464 s
 FEE module #:   3 elapsed dead time       0.002 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       4.701 s
 FEE module #:   6 elapsed dead time       0.228 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.304 s
 FEE module #:  10 elapsed dead time       0.154 s
 FEE module #:  11 elapsed dead time       0.901 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.203 s
 FEE module #:  14 elapsed dead time       0.164 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.346 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:  123.023s (  260.113 blocks/s,  16.257 Mb/s)

23.00 Found DAQ stalled - zero AIDA SYNCs from all FEE64s, and data trsabfer halted
      Disk files indicate stall at c. 18.58 (AIDA66_58)

      See attachments 14-19

      All FEE64s pass all system-wide tests

      Installed new versions of check.tcl/check.tml - see attachment 20
      Collect All SYNC Error Counters for Baseline selected

23.15 Standard DAQ stop/go sequence restarts DAQ

      AIDA67
   
      No changes from AIDA66

[td@localhost 1E47-5D81]$ ~/GREAT/analyser2 AIDA67_7
 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***   SYNC100 timestamp: block:     1 ptr:  6123 data: 0x8B4028BF module: 11 information type:  4 information 
field: 0x000028BF ts: 0x0000028BF38C00A0 OLD: ts: 0x0DAF8F30038C0093 tag: 2 type: 6
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  162789406 (  931310.8 Hz)
        Other data format:   98330594 (  562544.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         81 (       0.5 Hz)
                               RESUME:         81 (       0.5 Hz)
                              SYNC100:      66680 (     381.5 Hz)
                           FEE64 disc:   97373010 (  557066.5 Hz)
                             MBS info:     890742 (    5095.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      40512 (     231.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      174.796 s
 FEE module #:   1 elapsed dead time       0.162 s
 FEE module #:   2 elapsed dead time       0.232 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.213 s
 FEE module #:   6 elapsed dead time       0.273 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.433 s
 FEE module #:  10 elapsed dead time       0.123 s
 FEE module #:  11 elapsed dead time       0.417 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.127 s
 FEE module #:  14 elapsed dead time       0.000 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.095 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.030 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:  338.012s (   94.671 blocks/s,   5.917 Mb/s)

***            ADC data: block:     1 ptr:  6003 data: 0xC63781A0 module: 24 fail: 0 range: 0 id:  1591 
channel: 55 adc: 33184  ts: 0x************
 ***            ADC data: block:     1 ptr:  6005 data: 0xC2837F23 module: 10 fail: 0 range: 0 id:   643 
channel:  3 adc: 32547  ts: 0x************
 ***            ADC data: block:     1 ptr:  6007 data: 0xC27E7DF9 module:  9 fail: 0 range: 0 id:   638 
channel: 62 adc: 32249  ts: 0x************
 ***            ADC data: block:     1 ptr:  6009 data: 0xC1BA7D10 module:  6 fail: 0 range: 0 id:   442 
channel: 58 adc: 32016  ts: 0x************
 ***            ADC data: block:     1 ptr:  6011 data: 0xC37A7DE7 module: 13 fail: 0 range: 0 id:   890 
channel: 58 adc: 32231  ts: 0x************
 ***            ADC data: block:     1 ptr:  6013 data: 0xC62C8141 module: 24 fail: 0 range: 0 id:  1580 
channel: 44 adc: 33089  ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6015 data: 0x81641120 module:  1 information type:  6 information 
field: 0x00041120 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6017 data: 0x81640806 module:  1 information type:  6 information 
field: 0x00040806 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6019 data: 0x8B622000 module: 11 information type:  6 information 
field: 0x00022000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6021 data: 0x81621000 module:  1 information type:  6 information 
field: 0x00021000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6023 data: 0x86640912 module:  6 information type:  6 information 
field: 0x00040912 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6025 data: 0x81640008 module:  1 information type:  6 information 
field: 0x00040008 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6027 data: 0x86644009 module:  6 information type:  6 information 
field: 0x00044009 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6029 data: 0x81622880 module:  1 information type:  6 information 
field: 0x00022880 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6031 data: 0x8B622000 module: 11 information type:  6 information 
field: 0x00022000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6033 data: 0x81620510 module:  1 information type:  6 information 
field: 0x00020510 ts: 0x************
 ***            ADC data: block:     1 ptr:  6035 data: 0xC17774D5 module:  5 fail: 0 range: 0 id:   375 
channel: 55 adc: 29909  ts: 0x************
 ***            ADC data: block:     1 ptr:  6037 data: 0xC31480E9 module: 12 fail: 0 range: 0 id:   788 
channel: 20 adc: 33001  ts: 0x************
 ***            ADC data: block:     1 ptr:  6039 data: 0xC1BD7ED2 module:  6 fail: 0 range: 0 id:   445 
channel: 61 adc: 32466  ts: 0x************
 ***            ADC data: block:     1 ptr:  6041 data: 0xC3957E68 module: 14 fail: 0 range: 0 id:   917 
channel: 21 adc: 32360  ts: 0x************
 ***            ADC data: block:     1 ptr:  6043 data: 0xC25E7E3C module:  9 fail: 0 range: 0 id:   606 
channel: 30 adc: 32316  ts: 0x************
 ***            ADC data: block:     1 ptr:  6045 data: 0xC2777F9B module:  9 fail: 0 range: 0 id:   631 
channel: 55 adc: 32667  ts: 0x************
 ***            ADC data: block:     1 ptr:  6047 data: 0xC3627F99 module: 13 fail: 0 range: 0 id:   866 
channel: 34 adc: 32665  ts: 0x************
 ***            ADC data: block:     1 ptr:  6049 data: 0xC0C08485 module:  3 fail: 0 range: 0 id:   192 
channel:  0 adc: 33925  ts: 0x************
 ***            ADC data: block:     1 ptr:  6051 data: 0xC3B77E63 module: 14 fail: 0 range: 0 id:   951 
channel: 55 adc: 32355  ts: 0x************
 ***            ADC data: block:     1 ptr:  6053 data: 0xC28A7F9E module: 10 fail: 0 range: 0 id:   650 
channel: 10 adc: 32670  ts: 0x************
 ***            ADC data: block:     1 ptr:  6055 data: 0xC2B77ED9 module: 10 fail: 0 range: 0 id:   695 
channel: 55 adc: 32473  ts: 0x************
 ***            ADC data: block:     1 ptr:  6057 data: 0xC2657E47 module:  9 fail: 0 range: 0 id:   613 
channel: 37 adc: 32327  ts: 0x************
 ***            ADC data: block:     1 ptr:  6059 data: 0xC5417EB7 module: 21 fail: 0 range: 0 id:  1345 
channel:  1 adc: 32439  ts: 0x************
 ***            ADC data: block:     1 ptr:  6061 data: 0xC5777ECD module: 21 fail: 0 range: 0 id:  1399 
channel: 55 adc: 32461  ts: 0x************
 ***            ADC data: block:     1 ptr:  6063 data: 0xC3777D25 module: 13 fail: 0 range: 0 id:   887 
channel: 55 adc: 32037  ts: 0x************
 ***            ADC data: block:     1 ptr:  6065 data: 0xC2417E9B module:  9 fail: 0 range: 0 id:   577 
channel:  1 adc: 32411  ts: 0x************
 ***            ADC data: block:     1 ptr:  6067 data: 0xC63782CB module: 24 fail: 0 range: 0 id:  1591 
channel: 55 adc: 33483  ts: 0x************
 ***            ADC data: block:     1 ptr:  6069 data: 0xC24A7F3D module:  9 fail: 0 range: 0 id:   586 
channel: 10 adc: 32573  ts: 0x************
 ***            ADC data: block:     1 ptr:  6071 data: 0xC2DC8028 module: 11 fail: 0 range: 0 id:   732 
channel: 28 adc: 32808  ts: 0x************
 ***            ADC data: block:     1 ptr:  6073 data: 0xC281801A module: 10 fail: 0 range: 0 id:   641 
channel:  1 adc: 32794  ts: 0x************
 ***            ADC data: block:     1 ptr:  6075 data: 0xC1BA7E55 module:  6 fail: 0 range: 0 id:   442 
channel: 58 adc: 32341  ts: 0x************
 ***            ADC data: block:     1 ptr:  6077 data: 0xC1817EBA module:  6 fail: 0 range: 0 id:   385 
channel:  1 adc: 32442  ts: 0x************
 ***            ADC data: block:     1 ptr:  6079 data: 0xC4B67EFE module: 18 fail: 0 range: 0 id:  1206 
channel: 54 adc: 32510  ts: 0x************
 ***            ADC data: block:     1 ptr:  6081 data: 0xC3BA7DF5 module: 14 fail: 0 range: 0 id:   954 
channel: 58 adc: 32245  ts: 0x************
 ***            ADC data: block:     1 ptr:  6083 data: 0xC0BD7E2A module:  2 fail: 0 range: 0 id:   189 
channel: 61 adc: 32298  ts: 0x************
 ***            ADC data: block:     1 ptr:  6085 data: 0xC37A7D4F module: 13 fail: 0 range: 0 id:   890 
channel: 58 adc: 32079  ts: 0x************
 ***            ADC data: block:     1 ptr:  6087 data: 0xC2837F32 module: 10 fail: 0 range: 0 id:   643 
channel:  3 adc: 32562  ts: 0x************
 ***            ADC data: block:     1 ptr:  6089 data: 0xC17779E8 module:  5 fail: 0 range: 0 id:   375 
channel: 55 adc: 31208  ts: 0x************
 ***            ADC data: block:     1 ptr:  6091 data: 0xC0407D64 module:  1 fail: 0 range: 0 id:    64 
channel:  0 adc: 32100  ts: 0x************
 ***            ADC data: block:     1 ptr:  6093 data: 0xC27E7EBB module:  9 fail: 0 range: 0 id:   638 
channel: 62 adc: 32443  ts: 0x************
 ***            ADC data: block:     1 ptr:  6095 data: 0xC1BD7EA1 module:  6 fail: 0 range: 0 id:   445 
channel: 61 adc: 32417  ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6097 data: 0x8B648040 module: 11 information type:  6 information 
field: 0x00048040 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6099 data: 0x8B632000 module: 11 information type:  6 information 
field: 0x00032000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6101 data: 0x8B640014 module: 11 information type:  6 information 
field: 0x00040014 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6103 data: 0x8B622000 module: 11 information type:  6 information 
field: 0x00022000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6105 data: 0x85640100 module:  5 information type:  6 information 
field: 0x00040100 ts: 0x************
 ***            ADC data: block:     1 ptr:  6107 data: 0xC2048185 module:  8 fail: 0 range: 0 id:   516 
channel:  4 adc: 33157  ts: 0x************
 ***            ADC data: block:     1 ptr:  6109 data: 0xC3817CB5 module: 14 fail: 0 range: 0 id:   897 
channel:  1 adc: 31925  ts: 0x************
 ***            ADC data: block:     1 ptr:  6111 data: 0xC0B76DE1 module:  2 fail: 0 range: 0 id:   183 
channel: 55 adc: 28129  ts: 0x************
 ***            ADC data: block:     1 ptr:  6113 data: 0xC5347FB5 module: 20 fail: 0 range: 0 id:  1332 
channel: 52 adc: 32693  ts: 0x************
 ***            ADC data: block:     1 ptr:  6115 data: 0xC2B87EFA module: 10 fail: 0 range: 0 id:   696 
channel: 56 adc: 32506  ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6117 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6119 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6121 data: 0x82640003 module:  2 information type:  6 information 
field: 0x00040003 ts: 0x************
 ***   SYNC100 timestamp: block:     1 ptr:  6123 data: 0x8B4028BF module: 11 information type:  4 information 
field: 0x000028BF ts: 0x0000028BF38C00A0 OLD: 
ts: 0x0DAF8F30038C0093 tag: 2 type: 6
 ***   SYNC100 timestamp: block:     1 ptr:  6123 data: 0x8B4028BF module: 11 information type:  4 information 
field: 0x000028BF ts: 0x028BF38C00A0
 ***            ADC data: block:     1 ptr:  6125 data: 0xC30B80FA module: 12 fail: 0 range: 0 id:   779 
channel: 11 adc: 33018  ts: 0x028BF38C00BD
 ***            ADC data: block:     1 ptr:  6127 data: 0xC0817E77 module:  2 fail: 0 range: 0 id:   129 
channel:  1 adc: 32375  ts: 0x028BF38C00C2
 ***            ADC data: block:     1 ptr:  6129 data: 0xC277758A module:  9 fail: 0 range: 0 id:   631 
channel: 55 adc: 30090  ts: 0x028BF38C00CA
 *** FEE64 discriminator: block:     1 ptr:  6131 data: 0x8B648000 module: 11 information type:  6 information 
field: 0x00048000 ts: 0x028BF38C01D1
 *** FEE64 discriminator: block:     1 ptr:  6133 data: 0x8B640040 module: 11 information type:  6 information 
field: 0x00040040 ts: 0x028BF38C01D2
 *** FEE64 discriminator: block:     1 ptr:  6135 data: 0x8B640004 module: 11 information type:  6 information 
field: 0x00040004 ts: 0x028BF38C01D5
 *** FEE64 discriminator: block:     1 ptr:  6137 data: 0x8B640010 module: 11 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C01D9
 ***            ADC data: block:     1 ptr:  6139 data: 0xC3B77874 module: 14 fail: 0 range: 0 id:   951 
channel: 55 adc: 30836  ts: 0x028BF38C020A
 ***            ADC data: block:     1 ptr:  6141 data: 0xC0C0848E module:  3 fail: 0 range: 0 id:   192 
channel:  0 adc: 33934  ts: 0x028BF38C0281
 ***            ADC data: block:     1 ptr:  6143 data: 0xC63C8026 module: 24 fail: 0 range: 0 id:  1596 
channel: 60 adc: 32806  ts: 0x028BF38C02DC
 *** FEE64 discriminator: block:     1 ptr:  6145 data: 0x81641102 module:  1 information type:  6 information 
field: 0x00041102 ts: 0x028BF38C0395
 *** FEE64 discriminator: block:     1 ptr:  6147 data: 0x81640020 module:  1 information type:  6 information 
field: 0x00040020 ts: 0x028BF38C0396
 *** FEE64 discriminator: block:     1 ptr:  6149 data: 0x8B620001 module: 11 information type:  6 information 
field: 0x00020001 ts: 0x028BF38C0397
 ***            ADC data: block:     1 ptr:  6151 data: 0xC2B77CE0 module: 10 fail: 0 range: 0 id:   695 
channel: 55 adc: 31968  ts: 0x028BF38C03E1
 *** FEE64 discriminator: block:     1 ptr:  6153 data: 0x8E640200 module: 14 information type:  6 information 
field: 0x00040200 ts: 0x028BF38C03E5
 *** FEE64 discriminator: block:     1 ptr:  6155 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C046F
 *** FEE64 discriminator: block:     1 ptr:  6157 data: 0x82640009 module:  2 information type:  6 information 
field: 0x00040009 ts: 0x028BF38C0470
 *** FEE64 discriminator: block:     1 ptr:  6159 data: 0x82640002 module:  2 information type:  6 information 
field: 0x00040002 ts: 0x028BF38C0473
 ***            ADC data: block:     1 ptr:  6161 data: 0xC3777F1E module: 13 fail: 0 range: 0 id:   887 
channel: 55 adc: 32542  ts: 0x028BF38C05F3
 ***            ADC data: block:     1 ptr:  6163 data: 0xC0BA7F03 module:  2 fail: 0 range: 0 id:   186 
channel: 58 adc: 32515  ts: 0x028BF38C063A
 ***            ADC data: block:     1 ptr:  6165 data: 0xC63781EA module: 24 fail: 0 range: 0 id:  1591 
channel: 55 adc: 33258  ts: 0x028BF38C06C4
 ***            ADC data: block:     1 ptr:  6167 data: 0xC25E7EC0 module:  9 fail: 0 range: 0 id:   606 
channel: 30 adc: 32448  ts: 0x028BF38C070A
 ***            ADC data: block:     1 ptr:  6169 data: 0xC310814C module: 12 fail: 0 range: 0 id:   784 
channel: 16 adc: 33100  ts: 0x028BF38C07C5
 ***            ADC data: block:     1 ptr:  6171 data: 0xC2817FBE module: 10 fail: 0 range: 0 id:   641 
channel:  1 adc: 32702  ts: 0x028BF38C07C9
 *** FEE64 discriminator: block:     1 ptr:  6173 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C0851
 *** FEE64 discriminator: block:     1 ptr:  6175 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x028BF38C0854
 ***            ADC data: block:     1 ptr:  6177 data: 0xC1817E3E module:  6 fail: 0 range: 0 id:   385 
channel:  1 adc: 32318  ts: 0x028BF38C095D
 *** FEE64 discriminator: block:     1 ptr:  6179 data: 0x86631000 module:  6 information type:  6 information 
field: 0x00031000 ts: 0x028BF38C0998
 ***            ADC data: block:     1 ptr:  6181 data: 0xC5807DFE module: 22 fail: 0 range: 0 id:  1408 
channel:  0 adc: 32254  ts: 0x028BF38C09BC
 ***     MBS information: block:     1 ptr:  6183 data: 0x8680FCC8 module:  6 information type:  8 information 
index:  0 MBS data:FCC8 ts: 0x028BF38C09F7
 ***     MBS information: block:     1 ptr:  6185 data: 0x8681680D module:  6 information type:  8 information 
index:  1 MBS data:680D ts: 0x028BF38C09F7
 ***     MBS information: block:     1 ptr:  6187 data: 0x86820007 module:  6 information type:  8 information 
index:  2 MBS data:0007 ts: 0x028BF38C09F7
 ***            ADC data: block:     1 ptr:  6189 data: 0xC0BD7F10 module:  2 fail: 0 range: 0 id:   189 
channel: 61 adc: 32528  ts: 0x028BF38C0A22
 ***            ADC data: block:     1 ptr:  6191 data: 0xC1777A9E module:  5 fail: 0 range: 0 id:   375 
channel: 55 adc: 31390  ts: 0x028BF38C0B47
 ***            ADC data: block:     1 ptr:  6193 data: 0xC3928023 module: 14 fail: 0 range: 0 id:   914 
channel: 18 adc: 32803  ts: 0x028BF38C0CFA
 ***            ADC data: block:     1 ptr:  6195 data: 0xC3427F14 module: 13 fail: 0 range: 0 id:   834 
channel:  2 adc: 32532  ts: 0x028BF38C0CFB
 *** FEE64 discriminator: block:     1 ptr:  6197 data: 0x85640100 module:  5 information type:  6 information 
field: 0x00040100 ts: 0x028BF38C0D7B
 *** FEE64 discriminator: block:     1 ptr:  6199 data: 0x85640028 module:  5 information type:  6 information 
field: 0x00040028 ts: 0x028BF38C0D7C
 *** FEE64 discriminator: block:     1 ptr:  6201 data: 0x85640020 module:  5 information type:  6 information 
field: 0x00040020 ts: 0x028BF38C0D7F
 ***            ADC data: block:     1 ptr:  6203 data: 0xC5817E5C module: 22 fail: 0 range: 0 id:  1409 
channel:  1 adc: 32348  ts: 0x028BF38C0DA4
 ***            ADC data: block:     1 ptr:  6205 data: 0xC3817F2D module: 14 fail: 0 range: 0 id:   897 
channel:  1 adc: 32557  ts: 0x028BF38C0DC2
 ***            ADC data: block:     1 ptr:  6207 data: 0xC0B76C9E module:  2 fail: 0 range: 0 id:   183 
channel: 55 adc: 27806  ts: 0x028BF38C0E0A
 *** FEE64 discriminator: block:     1 ptr:  6209 data: 0x82648000 module:  2 information type:  6 information 
field: 0x00048000 ts: 0x028BF38C0F44
 *** FEE64 discriminator: block:     1 ptr:  6211 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C0F45
 *** FEE64 discriminator: block:     1 ptr:  6213 data: 0x82630008 module:  2 information type:  6 information 
field: 0x00030008 ts: 0x028BF38C0F46
 *** FEE64 discriminator: block:     1 ptr:  6215 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x028BF38C0F46
 ***            ADC data: block:     1 ptr:  6217 data: 0xC0817E94 module:  2 fail: 0 range: 0 id:   129 
channel:  1 adc: 32404  ts: 0x028BF38C0F9A
 *** FEE64 discriminator: block:     1 ptr:  6219 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C1018
 ***            ADC data: block:     1 ptr:  6221 data: 0xC61980E8 module: 24 fail: 0 range: 0 id:  1561 
channel: 25 adc: 33000  ts: 0x028BF38C1024
 ***            ADC data: block:     1 ptr:  6223 data: 0xC6248081 module: 24 fail: 0 range: 0 id:  1572 
channel: 36 adc: 32897  ts: 0x028BF38C1024
 ***            ADC data: block:     1 ptr:  6225 data: 0xC5398017 module: 20 fail: 0 range: 0 id:  1337 
channel: 57 adc: 32791  ts: 0x028BF38C103B
 ***            ADC data: block:     1 ptr:  6227 data: 0xC4727E47 module: 17 fail: 0 range: 0 id:  1138 
channel: 50 adc: 32327  ts: 0x028BF38C1089
 ***            ADC data: block:     1 ptr:  6229 data: 0xC20080B8 module:  8 fail: 0 range: 0 id:   512 
channel:  0 adc: 32952  ts: 0x028BF38C10C1
 ***            ADC data: block:     1 ptr:  6231 data: 0xC0C08465 module:  3 fail: 0 range: 0 id:   192 
channel:  0 adc: 33893  ts: 0x028BF38C1159
 ***            ADC data: block:     1 ptr:  6233 data: 0xC2657E31 module:  9 fail: 0 range: 0 id:   613 
channel: 37 adc: 32305  ts: 0x028BF38C12C2
 *** FEE64 discriminator: block:     1 ptr:  6235 data: 0x82648010 module:  2 information type:  6 information 
field: 0x00048010 ts: 0x028BF38C12DF
 *** FEE64 discriminator: block:     1 ptr:  6237 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x028BF38C12E3
 ***            ADC data: block:     1 ptr:  6239 data: 0xC36C801D module: 13 fail: 0 range: 0 id:   876 
channel: 44 adc: 32797  ts: 0x028BF38C14CB
 *** FEE64 discriminator: block:     1 ptr:  6241 data: 0x81641000 module:  1 information type:  6 information 
field: 0x00041000 ts: 0x028BF38C153A
 *** FEE64 discriminator: block:     1 ptr:  6243 data: 0x8B632000 module: 11 information type:  6 information 
field: 0x00032000 ts: 0x028BF38C153A
 *** FEE64 discriminator: block:     1 ptr:  6245 data: 0x8B640040 module: 11 information type:  6 information 
field: 0x00040040 ts: 0x028BF38C153A
 ***            ADC data: block:     1 ptr:  6247 data: 0xC5F7816D module: 23 fail: 0 range: 0 id:  1527 
channel: 55 adc: 33133  ts: 0x028BF38C154C
 ***            ADC data: block:     1 ptr:  6249 data: 0xC3BB7E35 module: 14 fail: 0 range: 0 id:   955 
channel: 59 adc: 32309  ts: 0x028BF38C1592
 *** FEE64 discriminator: block:     1 ptr:  6251 data: 0x97630080 module: 23 information type:  6 information 
field: 0x00030080 ts: 0x028BF38C164D
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 9.png
9.png
Attachment 10: 10.png
10.png
Attachment 11: 20.png
20.png
Attachment 12: 21.png
21.png
Attachment 13: 3.png
3.png
Attachment 14: 31.png
31.png
Attachment 15: 32.png
32.png
Attachment 16: 33.png
33.png
Attachment 17: 35.png
35.png
Attachment 18: 36.png
36.png
Attachment 19: 37.png
37.png
Attachment 20: 40.png
40.png
  314   Tue Jun 14 01:42:56 2016 TDTuesday 14 June 2016
09.35 AIDA67 stalled AIDA67_43 (c. 01.27)
      DAQ stopped and file closed 09.32

      SYNC error counters - see attachments 1-2

      All FEE64s pass all tests *except* SYNC counter error

09.37 AIDA68

      ASICs checked

09.42 Detector bias & leakage currents, trigger rates - see attachments 3-5 

10.03 AIDA68 stopped (loss of FEE64 synchronisation)

      Power-cycle & reboot FEE64s

10.22 AIDA69 starts

      All FEE64s pass all system-wide tests *except* nnaida1 fails clock status (0x6)

10.35 SyncCheck looks OK

10.42 AIDA69_4 FEE64 synchronisation OK


13.26 AIDA69 stalled AIDA69_63(c. 13.26)

15.22 AIDA70 

      ASICs checked

15.31 AIDA70 stopped (loss of FEE64 synchronisation)

      See attachment 6

      Power-cycle & reboot FEE64s

      System-wide checks OK *except* FADC calibration nnaida6, 20, 22 & 23
      nnaida1 clock check OK
      SYNC errors baseline set

15.54 AIDA71 starts

16.04 AIDA71_1 FEE64 synchronisation OK

17.20 AIDA71_28 FEE64 synchronisation OK
      SyncCheck OK

19.55 AIDA71 DAQ stalled AIDA71_64 c. 19.02

      See attachment 7

      Power-cycle & reboot FEE64s

20.14 AIDA72 starts

      ASICs checked
      SyncCheck OK

22.13 DAQ running AIDA72_33
Attachment 1: 50.png
50.png
Attachment 2: 51.png
51.png
Attachment 3: 52.png
52.png
Attachment 4: 52.png
52.png
Attachment 5: 54.png
54.png
Attachment 6: 55.png
55.png
Attachment 7: 56.png
56.png
ELOG V3.1.3-7933898