AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 18 of 46  ELOG logo
ID Date Authordown Subject
  541   Wed Mar 29 05:08:04 2017 TDReport - medium - delayed or missing authenticated mount requests following FEE64 power cycle
This is not, I think, a new observation.

Following FEE64 power cycles we sometimes observe delayed ( ~5-15 minutes ) authenticated mount requests
as the FEE64s mount their root filesystem from the AIDA server aidas1 - see attachment 1. 

There also appears to be an instance of a *missing* authenticated mount request - nnaida10 at 19.51

So this is *a* (but probably not *the*) cause of FEE64 timeouts observed following a FEE64 power cycle
and then waiting 10 minutes before attempting to start the MIDAS DAQ.


I have noticed that these issues appear to be significantly worse when we are using NTFS-mounted USB disks
to backup data from the AIDA server internal disks. Indeed, performance in general is noticeably slower when
using the NTFS-mounted USB disks.

Currently, we attempt to mitigate these issues by either throttling rsync or nice'ing cp. 
  543   Wed Mar 29 13:02:17 2017 TDReport - high - interaction between DHCP and FEE64 authenticated mount requests
What has happened here?

Power cycle of FEE64s 
24x authenticated mount request c. 20.56

Followed by repeat for nnaida20 following DHCP activity c. 21.00

Mar 29 20:56:44 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.1:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:44 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.2:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:45 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.7:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:45 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.4:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:46 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.3:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:46 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.5:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:46 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.8:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.9:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.10:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.12:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.24:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.13:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.6:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.20:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:47 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.14:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.15:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.16:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.22:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.18:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.23:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:48 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.11:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:49 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.19:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:49 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.21:864 for /MIDAS@aidas
(/MIDAS@aidas)
Mar 29 20:56:49 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.1.17:864 for /MIDAS@aidas
(/MIDAS@aidas)


Mar 29 21:00:43 aidas1 dhcpd: DHCPDISCOVER from 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 dhcpd: DHCPOFFER on 10.1.2.20 to 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 dhcpd: DHCPREQUEST for 10.1.2.20 (10.1.2.250) from 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 dhcpd: DHCPACK on 10.1.2.20 to 00:04:a3:2a:d0:11 via p2p2
Mar 29 21:00:43 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.20:686 for
/data10/XilinxLinux/ppc_4xx/rfs/nnaida20 (/data10/XilinxLinux/ppc_4xx/rfs)
Mar 29 21:01:08 aidas1 rpc.mountd[2812]: authenticated mount request from 10.1.2.20:864 for /MIDAS@aidas
(/MIDAS@aidas)
  565   Sat May 13 03:16:16 2017 TDAnalysis of May2017/R1_0 to May2017/ R1_79
See https://elog.ph.ed.ac.uk/AIDA/555

Date files May2017/R1_0 to May2017/R_79 are an extended (c. 3.8 day) background run with
BNC PB-4 Pulser OFF and and a relatively high slow comparator threshold 0x64 to reduce
data volume.

Attachment 1 - DSSSD#1-6 m_p versus m_n 

Attachment 2 - DSSSD#1-6 x versus y hit patterns

Attachment 3 - DSSSD#1-6 E_p versus E_n (dispersion ~20-23keV/channel)

Attachment 4 - DSSSD#1-6 E_p versus E_n (with tags)

Attachment 5 - 2D contour levels & colours for all 2D spectra above

Attachment 6 - MIDASsort subroutine

Attachment 7 - MIDASsort namelist I/O variables

Attachments 8-10 - ADC offset calculation logs etc

Attachment 11 - spectrum titles

MIDASsort calculates channel and ADC data as follows

       ch = channel + ( module - 1 ) * 64 + ( range * 2048 )
       adc_data = INT( RSHIFT( ABS( adc_data - 32768 ), 3 ) - offset(ch) + 0.5 )

       where

       channel = FEE channel # (0-63)
       module = FEE module # (1-24)
       range = LEC or HEC (LEC = 0, HEC = 1)

ADC offsets were calculated using the pulser walkthrough data of files May2017/R4
and command

cd ~/td/calibration
offsets /homes/npg/Spectra/R4 100 1400 15 10

where the peakfind peak search parameters are limits 100-1400 channels, fwhm=15 channels and acceptance=10%

Probably ~10% of ADC offsets need manual checking/adjustment

The 2D spectra (attachments 1-4) were produced for events with

multiplicities 0 < m_p < 8 *and* 0 < m_n < 8
all x-y combinations incremented

no clustering, gain matching or software thresholds

All DSSSDs show Uranium 4n+2 decay sequence background spectra from the intrinsic activity of the DSSSDs
and airborne decay products (e.g. 222Rn). Not clear why statistics are higher for DSSSD#1 cf. DSSSD#2-6.
One possibility is the activity of the aluminized Mylar window just upstream of DSSSD#1? Surrounding air
and Al snout would presumably affect DSSSD#1 and #6 about the same.
  581   Sun May 21 10:58:01 2017 TDJulabo FL11006 recirculating chiller refilled
18.55 Julabo FL11006 recirculating chiller refilled to "H" marker of the filling level indicator.

      See http://www2.ph.ed.ac.uk/~td/AIDA/Information/Julabo%20FL11006.pdf
      
      To refill from 60% L-H to 100% L-H required c. 4 litres of water.
      Following refill there is a trickle of water from the overflow connector which is being 
      collected by hose and bucket - this flow should cease when the fluid level falls below
      the overflow level.
      
19.20 FEE64 temperatures OK - see attachment 1
  583   Mon May 22 08:29:26 2017 TDOffline analysis R25_12 to R25_22
Offline analysis of AIDA data files R25_12 to R25_22 using MIDASsort

Note

- no calibration (raw ADC data only)
- no thresholds 
- no clustering
- events delimited by > 2us time difference between successive ADC data words 
- HEC = high energy channel = 20GeV FSR range for high energy implants
- LEC = low energy channel = 20MeV FSR range for decays (and pulser)

Attachment 1-4
--------------
s2000-s2005 per DSSSD LEC m_lec_p versus m_lec_n

            following spectra require 0 < m_lec_p < 8 and 0 < m_lec_n < 8
            all combinations of x & y plotted

s2010-s2015 per DSSSD LEC x versus y
s2020-s2025 per DSSSD LEC E_p versus E_n        (20keV/ch nominal)
s2030-s2035 per DSSSD LEC-LEC time              (2.56us/channel)

Attachment 5-8
---------------
s2000-s2005 per DSSSD LEC m_hec_p versus m_hec_n

            following spectra require 0 < m_hec_p < 8 and 0 < m_hec_n < 8
            all combinations of x & y plotted

s2010-s2015 per DSSSD HEC x versus y           
s2020-s2025 per DSSSD HEC E_p versus E_n       (20MeV/channel nominal)
s2030-s2035 per DSSSD HEC-HEC time             (2.56us/channel)

Attachment 9-11
---------------
s2200-s2205 per pixel, per DSSSD HEC-LEC time  (2.56us/channel)

Attachment 12-14
----------------
s2210-s2215 per pixel, per DSSSD HEC-LEC time  (163.840us/channel)

Attachments 15-17
-----------------
s2300-s2305 per DSSSD HEC Energy versus HEC-LEC time (20MeV/channel, 163.840us/channel)
s2400-s2405 per DSSSD LEC Energy versus HEC-LEC time (20keV/channel, 163.840us/channel)
s2500-s2505 per DSSSD x & y strips versus HEC-LEC time (163.840us/channel)

Attachment 18
-------------
Projection s2503 y-strips 0-60 -> HEC-LEC time 163.804us/channel
  589   Mon May 22 10:29:11 2017 TDFEE64 system console to Raspberry pi /dev/ttyUSB mapping
ttyUSB0 =>  last host=nnaida24, => last addr=10.1.2.24, ; #panics => 0 ; Not Finished 
ttyUSB1 =>  last host=nnaida23, => last addr=10.1.2.23, ; #panics => 0 ; Not Finished 
ttyUSB10 =>  last host=nnaida18, => last addr=10.1.1.18, ; #panics => 0 ; Not Finished 
ttyUSB11 =>  last host=nnaida17, => last addr=10.1.1.17, ; #panics => 0 ; Not Finished 
ttyUSB12 =>  last host=nnaida4, => last addr=10.1.2.4, ; #panics => 0 ; Not Finished 
ttyUSB13 =>  last host=nnaida3, => last addr=10.1.2.3, ; #panics => 0 ; Not Finished 
ttyUSB14 =>  last host=nnaida12, => last addr=10.1.2.12, ; #panics => 0 ; Not Finished 
ttyUSB15 =>  last host=nnaida11, => last addr=10.1.2.11, ; #panics => 0 ; Not Finished 
ttyUSB16 =>  last host=nnaida20, => last addr=10.1.2.20, ; #panics => 0 ; Not Finished 
ttyUSB17 =>  last host=nnaida19, => last addr=10.1.2.19, ; #panics => 0 ; Not Finished 
ttyUSB18 =>  last host=nnaida6, => last addr=10.1.1.6, ; #panics => 0 ; Not Finished 
ttyUSB19 =>  last host=nnaida5, => last addr=10.1.1.5, ; #panics => 0 ; Not Finished 
ttyUSB2 =>  last host=nnaida16, => last addr=10.1.2.16, ; #panics => 0 ; Not Finished 
ttyUSB20 =>  last host=nnaida14, => last addr=10.1.1.14, ; #panics => 0 ; Not Finished 
ttyUSB21 =>  last host=nnaida13, => last addr=10.1.1.13, ; #panics => 0 ; Not Finished 
ttyUSB22 =>  last host=nnaida22, => last addr=10.1.1.22, ; #panics => 0 ; Not Finished 
ttyUSB23 =>  last host=nnaida21, => last addr=10.1.1.21, ; #panics => 0 ; Not Finished 
ttyUSB3 =>  last host=nnaida15, => last addr=10.1.2.15, ; #panics => 0 ; Not Finished 
ttyUSB4 =>  last host=nnaida8, => last addr=10.1.2.8, ; #panics => 0 ; Not Finished 
ttyUSB5 =>  last host=nnaida7, => last addr=10.1.2.7, ; #panics => 0 ; Not Finished 
ttyUSB6 =>  last host=nnaida2, => last addr=10.1.1.2, ; #panics => 0 ; Not Finished 
ttyUSB7 =>  last host=nnaida1, => last addr=10.1.1.1, ; #panics => 0 ; Not Finished 
ttyUSB8 =>  last host=nnaida10, => last addr=10.1.1.10, ; #panics => 0 ; Not Finished 
ttyUSB9 =>  last host=nnaida9, => last addr=10.1.1.9, ; #panics => 0 ; Not Finished 
  593   Wed May 24 06:12:48 2017 TDLead bricks removed from BRIKEN aperture for source tests - AIDA NOT SHIELDED FROM BEAM
To install the 252Cf fission source to test the 3He neutron counters, Pb bricks which
had blocked the BRIKEN aperture have now been removed.
  594   Wed May 24 06:33:19 2017 TDAnalysis of RIBF123R1/R23_44, R23_91 & R25_142
Álvaro Tolosa Delgado reports observation of c. 1s 'oscillations' in implant-decay time spectra - see below

Hello

The worst cases (oscillations in decay-curve) are:

-->R23_91_115

--> R25_142_149

Other bad cases:

--> R23_44_70

Cheers!

Check sample of data files - see attachments 1-3 - no obvious data structure issues but higher deadtime for
nnaida17 for R23_* files

Control check using file R25_12 - see attachment 4 
  600   Thu May 25 03:03:38 2017 TDOffline analysis of R23_91 to R23_115
Offline analysis of RIBF123R1_91 to R23_115

See https://elog.ph.ed.ac.uk/AIDA/583 & https://elog.ph.ed.ac.uk/AIDA/594

No evidence of ~1s oscillations but we do observe ~360us  oscillations in HEC-HEC, LEC-LEC and HEC-LEC times

Attachment 1 - per DSSSD HEC-HEC time 2.56us/channel - 387-246=141=361us

Attachments 2-4 - per DSSSD LEC-LEC time 2.56us/channel - 858-721=138=353us

Attachments 5-7 - per DSSSD HEC-LEC time 2.56us/channel - 699-567=132=338us

Note also 'random' spikes in s2200 DSSSD#1 HEC-LEC time. This may suggest an increase
in noise or fluctuations in noise level. 

Attachment 8 - per DSSSD m_p versus m_n - note additional feature in DSSSD#1 & #2 

Attachments 9-10 - per DSSSD HEC-LEC time 163.84us/channel - no ~1s oscillations
  602   Fri May 26 05:19:40 2017 TDOffline analysis RIBF123R1/R23_44 to R23_70
Observe same noise structures to https://elog.ph.ed.ac.uk/AIDA/600

Extended noisy periods ~few Hz compose of fine structure with period ~350us
  605   Sun May 28 09:20:29 2017 TDAnalysis of RIBF123R1/R21
  607   Sun May 28 10:25:11 2017 TDAnalysis of RIBF123R1/R23
  609   Mon May 29 05:26:39 2017 TDSeagate Backup Plus Hub 8Tb USB disk #1
aidas1> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidas1-lv_root
                       50G   42G  4.7G  91% /
tmpfs                 3.9G  1.1G  2.9G  27% /dev/shm
/dev/sda1             477M   77M  376M  17% /boot
/dev/mapper/vg_aidas1-lv_home
                      860G  580G  236G  72% /home
/dev/mapper/Data1-data10
                      7.7T  6.6T  763G  90% /data10
/dev/sdg2             7.3T  7.1T  190G  98% /ntfs
aidas1> ls /ntfs
Autorun.inf     Mar2017  NP1306    Seagate             Warranty.pdf
BackupPlus.ico  May2015  NP1512    Sept2016
Dec2016         May2016  PE16      Start_Here_Mac.app
Feb2015         Nov2016  RIBF03R1  Start_Here_Win.exe
aidas1> cd /ntfs
/ntfs
aidas1> du -h
22G	./Dec2016
41G	./Feb2015
14G	./Mar2017
123G	./May2015
7.9G	./May2016
97G	./Nov2016
1.1T	./NP1306
1.6T	./NP1512
1.7T	./PE16
101G	./RIBF03R1/BigRIPS
2.2M	./RIBF03R1/BRIKEN/1622FallonPreparation/CheckCal_161122_2208_003
25K	./RIBF03R1/BRIKEN/1622FallonPreparation/CheckCal_161124_1215
8.2G	./RIBF03R1/BRIKEN/1622FallonPreparation
500K	./RIBF03R1/BRIKEN/1626FallonParasitic/Conf
120K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id204_id184/ELOG_RIBF123_id204_id184_files
164K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id204_id184
788K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id224_id205/ELOG_RIBF123_id224_id204_files
844K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id224_id205
19M	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123
313M	./RIBF03R1/BRIKEN/1626FallonParasitic/FallonOnlines
187G	./RIBF03R1/BRIKEN/1626FallonParasitic
28K	./RIBF03R1/BRIKEN/BRIKENrootTools
195G	./RIBF03R1/BRIKEN
71M	./RIBF03R1/RIBF03R1_logs
2.5T	./RIBF03R1
512	./Seagate/Registration
512	./Seagate
115G	./Sept2016
0	./Start_Here_Mac.app/Contents/Frameworks
26M	./Start_Here_Mac.app/Contents/MacOS
304K	./Start_Here_Mac.app/Contents/Resources
4.0K	./Start_Here_Mac.app/Contents/_CodeSignature
26M	./Start_Here_Mac.app/Contents
26M	./Start_Here_Mac.app
7.1T	.
aidas1> 
  610   Mon May 29 07:05:06 2017 TDMerger error messages and SEGV
Overnight and earlier today Obtained the following Merger error messages

Merge: Attempting resynchronisation to timestamp 26c3b4700056 or greater
Merge: Last sync pulse time: 26c3b46c00a0
Merge: Last timestamp seen: 26c3b4700055
Merge: Restarting at timestamp 26c3b47000a0
Merge: Timestamp (26c8062c00a0)[Q 5, Stream 5] is older than previous (26c8162bf81d) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26c8162bf81e or greater
Merge: Last sync pulse time: 26c8162800a0
Merge: Last timestamp seen: 26c8162bf81d
Merge: Restarting at timestamp 26c8162c00a0
Merge: Timestamp (26caa2109e35)[Q 5, Stream 5] is older than previous (26cab11e8415) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cab11e8416 or greater
Merge: Last sync pulse time: 26cab11c00a0
Merge: Last timestamp seen: 26cab11e8415
Merge: Restarting at timestamp 26cab12000a0
Merge: Timestamp (26cbfe8400a0)[Q 5, Stream 5] is older than previous (26cc0e83f4e5) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cc0e83f4e6 or greater
Merge: Last sync pulse time: 26cc0e8000a0
Merge: Last timestamp seen: 26cc0e83f4e5
Merge: Restarting at timestamp 26cc0e8400a0
Merge: Timestamp (26cf059000a0)[Q 5, Stream 5] is older than previous (26cf158ffb05) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cf158ffb06 or greater
Merge: Last sync pulse time: 26cf158c00a0
Merge: Last timestamp seen: 26cf158ffb05
Merge: Restarting at timestamp 26cf159000a0
Merge: Timestamp (26cf899c00a0)[Q 5, Stream 5] is older than previous (26cf999bf9ad) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cf999bf9ae or greater
Merge: Last sync pulse time: 26cf999800a0
Merge: Last timestamp seen: 26cf999bf9ad
Merge: Restarting at timestamp 26cf999c00a0
Merge: Timestamp (26dc07cac965)[Q 5, Stream 5] is older than previous (26dc16e6255d) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26dc16e6255e or greater
Merge: Last sync pulse time: 26dc16e400a0
Merge: Last timestamp seen: 26dc16e6255d
Merge: Restarting at timestamp 26dc16e800a0
Merge: Timestamp (26e035dac9cd)[Q 5, Stream 5] is older than previous (26e043cef5cd) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26e043cef5ce or greater
Merge: Last sync pulse time: 26e043cc00a0
Merge: Last timestamp seen: 26e043cef5cd
Merge: Restarting at timestamp 26e043d000a0
Merge: Timestamp (26e20f9000a0)[Q 5, Stream 5] is older than previous (26e21f8ffbdd) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26e21f8ffbde or greater
Merge: Last sync pulse time: 26e21f8c00a0
Merge: Last timestamp seen: 26e21f8ffbdd
Merge: Restarting at timestamp 26e21f9000a0
Merge: Timestamp (26e2817000a0)[Q 5, Stream 5] is older than previous (26e2916ffd25) [Q 5, Stream 5]

 :
 :
 :

This afternoon obtained the following error messages

MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Data Link (23172): Received SEGV signal; exiting.
MERGE Server: Received signal 17.

Analysis of final two files of R33 - see attachments 1 & 2
Note idle time >> dead time (defined by PAUSE/RESUME) for nnaida5

Offline analysis R33_52 confirms ADC & disc data are synchronised - see attachments 3 & 4
  618   Wed May 31 16:43:52 2017 TDMerger & TapeServer server startup scripts modified
aidas1> cd /MIDAS/Merger/AIDA
aidas1> ls
GS_configuration   GS_configuration.BAK-030516  Startup~
GS_configuration~  Startup                      Startup.BAK-010617
aidas1> diff Startup Startup.BAK-010617
63,64c63
< # 1 June 2017 - increase sleep time from 1000000 (11.6 days) to 150000000 (4.75 years)
< sleep 150000000
---
> sleep 1000000
aidas1> cd /MIDAS/TapeServer/startup
aidas1> ls
AidaTapeServer  AidaTapeServer~  AidaTapeServer.BAK-010617
aidas1> diff AidaTapeServer AidaTapeServer.BAK-010617
24,26c24
< # 1 June 2017 - increase sleep time from 1000000 (11.6 days) to 150000000 (4.75 years)
< sleep 150000000
< 
---
> sleep 1000000
  619   Wed May 31 16:48:54 2017 TDThursday 1 June 00.00-08.00
00.48 BigRIPS DAQ now running

      BRIKEN start run 48
      BigRIPS start run 1059
      AIDA file RIBF127R1/R1_664

      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39

      84Zn BigRIPS setting, 78Ni degrader setting - 5mm (in F7) + 9mm (in F11) + 1mm (in F11) = 16 mm Al

00.59 *all* online and SortSas histograms zero'd 

01.06 Detector biases & leakage currents OK - see attachments 1 & 2
      FEE64 Temperatures OK - see attachment 3
      good events statistics OK - see attachment 4
      No warnings/errors reported by merger since DAQ GO - see attachment 5

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

01.21 analysis of file RIBF127R1/R1_672 - dead time < 0.5% - see attachment 6

01.25 online analysis - ADC data synchronised - see attachment 7

01.27 per DSSSD HEC x versus y spectra - see attachment 8

01.30 per DSSSD rates file RIBF127R1/R1_674

 *** DSSSD # 1 count:    262381 old count:    211655 dt:      1.01 s  LEC rate:  50193.72 Hz
 *** DSSSD # 2 count:    349331 old count:    280852 dt:      1.01 s  LEC rate:  67760.43 Hz
 *** DSSSD # 3 count:    122673 old count:     97882 dt:      1.01 s  LEC rate:  24530.86 Hz
 *** DSSSD # 4 count:    105174 old count:     84961 dt:      1.01 s  LEC rate:  20000.90 Hz
 *** DSSSD # 5 count:     40546 old count:     32480 dt:      1.01 s  LEC rate:   7981.36 Hz
 *** DSSSD # 6 count:     79704 old count:     63990 dt:      1.01 s  LEC rate:  15549.11 Hz
 *** DSSSD # 1 count:       106 old count:        86 dt:      1.01 s  HEC rate:     19.79 Hz
 *** DSSSD # 2 count:       231 old count:       182 dt:      1.01 s  HEC rate:     48.49 Hz
 *** DSSSD # 3 count:       198 old count:       152 dt:      1.01 s  HEC rate:     45.52 Hz
 *** DSSSD # 4 count:       204 old count:       157 dt:      1.01 s  HEC rate:     46.51 Hz
 *** DSSSD # 5 count:        72 old count:        61 dt:      1.01 s  HEC rate:     10.88 Hz
 *** DSSSD # 6 count:         7 old count:         6 dt:      1.01 s  HEC rate:      0.99 Hz

01.54 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_679

01.58 BRIKEN start run 49
      BigRIPS start run 1060
      AIDA file RIBF127R1/R1_680

02.25 AIDA-BRIKEN-BigRIPS DAQ sync OK

02.58 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_694

03.00 BRIKEN start run 50
      BigRIPS start run 1061
      AIDA file RIBF127R1/R1_694

03.06 Detector biases & leakage currents OK - see attachments 9 & 10
      FEE64 Temperatures OK - see attachment 11
      good events statistics OK - see attachment 12
      1 error message reported by merger since DAQ GO - see attachment 13

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

03.14 analysis of file RIBF127R1/R1_697 - dead time < 0.5% - see attachment 14

03.17 online analysis - ADC & disc data synchronised - see attachment 15
      per DSSSD HEC x versus y spectra - see attachment 16

03.19 per DSSSD rates file RIBF127R1/R1_699

 *** DSSSD # 1 count:    181020 old count:    126616 dt:      1.03 s  LEC rate:  52680.27 Hz
 *** DSSSD # 2 count:    242594 old count:    169469 dt:      1.03 s  LEC rate:  70808.12 Hz
 *** DSSSD # 3 count:     87966 old count:     61977 dt:      1.03 s  LEC rate:  25165.57 Hz
 *** DSSSD # 4 count:     77118 old count:     55136 dt:      1.03 s  LEC rate:  21285.53 Hz
 *** DSSSD # 5 count:     31436 old count:     22904 dt:      1.03 s  LEC rate:   8261.67 Hz
 *** DSSSD # 6 count:     61421 old count:     44839 dt:      1.03 s  LEC rate:  16056.62 Hz
 *** DSSSD # 1 count:       102 old count:        76 dt:      1.03 s  HEC rate:     25.18 Hz
 *** DSSSD # 2 count:       179 old count:       134 dt:      1.03 s  HEC rate:     43.57 Hz
 *** DSSSD # 3 count:       136 old count:        92 dt:      1.03 s  HEC rate:     42.61 Hz
 *** DSSSD # 4 count:       136 old count:        93 dt:      1.03 s  HEC rate:     41.64 Hz
 *** DSSSD # 5 count:        62 old count:        56 dt:      1.03 s  HEC rate:      5.81 Hz
 *** DSSSD # 6 count:         8 old count:         4 dt:      1.03 s  HEC rate:      3.87 Hz

03.20 all SortSas spectra zero'd

03.33 histogram layouts 1-8 - see attachments 17-26
      all histograms zero'd

03.59 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_709

04.03 BRIKEN start run 51
      BigRIPS start run 1062
      AIDA file RIBF127R1/R1_710

05.01 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_722

05.03 BRIKEN start run 52
      BigRIPS start run 1063
      AIDA file RIBF127R1/R1_723

05.06 Detector biases & leakage currents OK - see attachments 27 & 28
      FEE64 Temperatures OK - see attachment 29
      good events statistics OK - see attachment 30
      1 error message reported by merger since DAQ GO - see attachment 31

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

05.12 analysis of file RIBF127R1/R1_725 - dead time < 0.5% - see attachment 32

03.14 online analysis - ADC & disc data synchronised - see attachment 33
      per DSSSD HEC x versus y spectra - see attachment 34

05.16 per DSSSD rates file RIBF127R1/R1_727

 *** DSSSD # 1 count:    207037 old count:    154849 dt:      1.04 s  LEC rate:  50002.29 Hz
 *** DSSSD # 2 count:    284982 old count:    210778 dt:      1.04 s  LEC rate:  71096.22 Hz
 *** DSSSD # 3 count:     97704 old count:     72256 dt:      1.04 s  LEC rate:  24382.20 Hz
 *** DSSSD # 4 count:     86608 old count:     65020 dt:      1.04 s  LEC rate:  20683.86 Hz
 *** DSSSD # 5 count:     31556 old count:     23485 dt:      1.04 s  LEC rate:   7732.97 Hz
 *** DSSSD # 6 count:     63379 old count:     47102 dt:      1.04 s  LEC rate:  15595.29 Hz
 *** DSSSD # 1 count:        72 old count:        51 dt:      1.04 s  HEC rate:     20.12 Hz
 *** DSSSD # 2 count:       176 old count:       124 dt:      1.04 s  HEC rate:     49.82 Hz
 *** DSSSD # 3 count:       163 old count:       116 dt:      1.04 s  HEC rate:     45.03 Hz
 *** DSSSD # 4 count:       153 old count:       126 dt:      1.04 s  HEC rate:     25.87 Hz
 *** DSSSD # 5 count:        53 old count:        37 dt:      1.04 s  HEC rate:     15.33 Hz
 *** DSSSD # 6 count:         8 old count:         3 dt:      1.04 s  HEC rate:      4.79 Hz

05.51 all SortSas spectra zero'd

05.54 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_735

07.01 BRIKEN start run 53
      BigRIPS start run 1064
      AIDA file RIBF127R1/R1_738

06.38 beam lost - ion source problem - AIDA file RIBF127R1/R1_745 - continue as background run

07.11 Detector biases & leakage currents OK - see attachments 35 & 36
      FEE64 Temperatures OK - see attachment 37
      good events statistics OK - see attachment 38
      1 error message reported by merger since DAQ GO - see attachment 39

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

07.17 analysis of file RIBF127R1/R1_754 - dead time < 0.5% - see attachment 40

07.19 online analysis - ADC & disc data synchronised - see attachment 41
      per DSSSD HEC x versus y spectra - see attachment 42

07.19 beam back

07.20 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_755

07.22 BRIKEN start run 54
      BigRIPS start run 1065
      AIDA file RIBF127R1/R1_756

07.24 per DSSSD rates file RIBF127R1/R1_756

 *** DSSSD # 1 count:    142610 old count:     89049 dt:      1.04 s  LEC rate:  51612.27 Hz
 *** DSSSD # 2 count:    191205 old count:    119983 dt:      1.04 s  LEC rate:  68630.70 Hz
 *** DSSSD # 3 count:     66544 old count:     40853 dt:      1.04 s  LEC rate:  24756.27 Hz
 *** DSSSD # 4 count:     60318 old count:     38762 dt:      1.04 s  LEC rate:  20771.72 Hz
 *** DSSSD # 5 count:     23029 old count:     14967 dt:      1.04 s  LEC rate:   7768.68 Hz
 *** DSSSD # 6 count:     45712 old count:     29870 dt:      1.04 s  LEC rate:  15265.61 Hz
 *** DSSSD # 1 count:        66 old count:        46 dt:      1.04 s  HEC rate:     19.27 Hz
 *** DSSSD # 2 count:       122 old count:        78 dt:      1.04 s  HEC rate:     42.40 Hz
 *** DSSSD # 3 count:       116 old count:        70 dt:      1.04 s  HEC rate:     44.33 Hz
 *** DSSSD # 4 count:        98 old count:        67 dt:      1.04 s  HEC rate:     29.87 Hz
 *** DSSSD # 5 count:        41 old count:        29 dt:      1.04 s  HEC rate:     11.56 Hz
 *** DSSSD # 6 count:         8 old count:         6 dt:      1.04 s  HEC rate:      1.93 Hz

07.25 all SortSas spectra zero'd

07.53 beam lost - RF problem -  AIDA file RIBF127R1/R1_764

07.55 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_763

07.57 beam back

07.57 BRIKEN start run 55
      BigRIPS start run 1066
      AIDA file RIBF127R1/R1_763
  627   Sat Jun 3 05:32:09 2017 TDReport - high - MBS correlation scaler anomalies
The analysis of files RIBF127R1/R1_1478 to R1_1486 is appended. This sequence of files should
encompass a correlation reset request (from BRIKEN to AIDA)

There appear to be a number of anomalies

1) MBS correlation scalers observed with one missing 16-bit data value (bits 16-31 or bits 0-15)
   across approximately five files (~20 minutes)

2) The first (presumably) MBS correlation scaler following reset corresponds to a time ~300s (25MHz clock)   
  637   Tue Jun 6 07:44:19 2017 TDPreamplifier waveforms
Preamplifier waveforms sampled by 50MSPS, 14 bit ADC


attachment 1 - nnaida13 ASIC settings (+ input)
attachment 2 - nnaida16 ASIC settings (- input)

beam on
-------

attachments 3-5 - three successive waveform updates nnaida13 asic #1 - 1000ch = 20us
attachments 6-8 - three successive waveform updates nnaida13 asic #1 - 1000ch = 2ms


attachments 9-11 - three successive waveform updates nnaida16 asic #1 - 1000ch = 20us
attachments 12-14 - three successive waveform updates nnaida16 asic #1 - 1000ch = 2ms

beam off
--------

attachments 15-18 - three successive waveform updates nnaida13 asic #1 - 1000ch = 2ms
attachments 19-21 - three successive waveform updates nnaida16 asic #1 - 1000ch = 2ms


attachments 22-23 - three successive waveform updates nnaida16 asic #1 - 1000ch = 2ms, y-scale expanded
attachments 24-25 - three successive waveform updates nnaida13 asic #1 - 1000ch = 2ms, y-scale expanded
  648   Fri Jun 9 04:53:24 2017 TDWaveforms
x-scale 1000ch = 2ms
y-scale expanded

p+n junction strips

DSSSD #1 .. #6 attachments 1-12

n+n ohmic strips

DSSSD #1 .. #6 attachments 13-24

DSSSD #4 pulser off, attachments 25-28
  658   Mon Jun 12 08:02:01 2017 TDMonday 12 June 16.00-24.00
16.01 DAQ continues

16.01 Detector biases & leakage currents OK - see attachments 1 & 2
      FEE64 Temperatures OK - see attachment 3
      good events statistics stable & OK - see attachment 4 
      No warnings/errors reported by merger since start of RIBF148/R3  - see attachment 5
      *all* system wide checks OK *except* nnaida13 fails ADC calibration

      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range 2.6Hz
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39

      147Sm BigRIPS setting - 0mm Al (at F7) + 2mm variable + 0mm Al fixed (at F11) = 2mm Al

16.08 *all* online histograms zero'd 

16.31 DAQ stop
      AIDA file RIBF148/R3_84

16.31 DAQ start
      AIDA file RIBF148/R3_84
      BigRIPS file 2504
      BRIKEN file 32

17.33 DAQ stop
      AIDA file RIBF148/R3_104

17.34 DAQ start
      AIDA file RIBF148/R3_104
      BigRIPS file 2505
      BRIKEN file 33

18.01 Detector biases & leakage currents OK - see attachments 6 & 7
      FEE64 Temperatures OK - see attachment 8
      good events statistics stable & OK - see attachment 9
      No warnings/errors reported by merger since start of RIBF148/R3  - see attachment 10
      *all* system wide checks OK *except* nnaida13 fails ADC calibration

18.03 online analysis file RIBF148/R3_113 - deadtime < 0.5%, 2 MBS & 0 ts timewarps - see attachment 11

18.04 DAQ stop
      AIDA file RIBF148/R3_114

18.05 DAQ start
      AIDA file RIBF148/R3_114
      BigRIPS file 2506
      BRIKEN file 34

      147Sm BigRIPS setting - 0mm Al (at F7) + 0mm variable + 0mm Al fixed (at F11) = 0mm Al

19.14 DAQ stop
      AIDA file RIBF148/R3_135

19.15 DAQ start
      AIDA file RIBF148/R3_135
      BigRIPS file 2507
      BRIKEN file 35

20.01 Detector biases & leakage currents OK - see attachments 12 & 13
      FEE64 Temperatures OK - see attachment 14
      good events statistics stable & OK - see attachment 15
      No warnings/errors reported by merger since start of RIBF148/R3  - see attachment 16
      *all* system wide checks OK *except* nnaida13 fails ADC calibration

20.03 online analysis file RIBF148/R3_150 - deadtime < 0.5%, 0 MBS & ts timewarps - see attachment 17

20.25 DAQ stop
      AIDA file RIBF148/R3_157

20.26 DAQ start
      AIDA file RIBF148/R3_158
      BigRIPS file 2508
      BRIKEN file 36

21.30 DAQ stop
      AIDA file RIBF148/R3_178

21.31 DAQ start
      AIDA file RIBF148/R3_179
      BigRIPS file 2509
      BRIKEN file 37

22.06 Detector biases & leakage currents OK - see attachments 18 & 19
      FEE64 Temperatures OK - see attachment 20
      good events statistics stable & OK - see attachment 21
      No warnings/errors reported by merger since start of RIBF148/R3  - see attachment 22
      *all* system wide checks OK *except* nnaida13 fails ADC calibration

22.08 online analysis file RIBF148/R3_190 - deadtime < 1%, 1 MBS & 0 ts timewarps - see attachment 23

22.26 DAQ stop
      AIDA file RIBF148/R3_196

22.27 DAQ start
      AIDA file RIBF148/R3_197
      BigRIPS file 2510
      BRIKEN file 38

23.03 online analysis
      per DSSSD x versus y (HEC) - see attachment 24
      per DSSSD E_P versus E_n (HEC) - see attachment 25
      ADC & disc data synchronised - see attachment 26

23.16 beam lost
      AIDA file RIBF148/R3_212

23.16 beam back within 1 min
      AIDA file RIBF148/R3_213

23.31 DAQ stop
      AIDA file RIBF148/R3_217

22.32 DAQ start
      AIDA file RIBF148/R3_218
      BigRIPS file 2511
      BRIKEN file 39
ELOG V3.1.3-7933898