There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use

Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 

The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system

In no particular order

1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)

   observe +/- 6V, 12V, 24V lines with/without load 

   try new CAEN NIM bin and/or NIM bin of different type

2) Measure actual voltages at FEE64 power connector input

   OH suggests fab of power adaptor for safe observation - contact EW

3) rev B adaptor PCB

   invert 125 way ERNI - check for mech conflicts
   paired HV input (avoid Lemo-00 T pieces)
   consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
   straight jumpers
   shrouded Samtec headers - consider mech issues/consequences of using eject clips too
   re-visit HV filtering & separate trace ground

4) isolation transformer

   as practical matter may be necessary to operate all platform from isolation transformer
   consider hire of appropriate unit

   need method to measure isolation - will require permit to work or equiv

5) investigate S4 area ac mains

   NH discussing with GSI electricians

6) Systematic measurement of AIDA PSU noise

   Spec linear AIDA FEE64 PSU

7) Redesign of snout
   Return to 1mm welded box Al for lower stage of snout  for added rigidity

8) Revisit calculation of cable lengths. Particularly for the triple

9) 



   

All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 

11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?

Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after

00:07 Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_896 - attachment 1
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6 & 7

00:15 Check ASIC control all FEE64s, all ASICs


02:03

analysis of file S505/R5_914 - attachment 8
 zero timewarps
 deadtime all FEE64s << 1% 

per FEE64 1.8.H spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 94 ch FWHM

per FEE64 stat & rate spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today


03:41

analysis of file S505/R5_926 - attachment 19
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24


07:00

analysis of file S505/R5_954 - attachment 25
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

07:45 Spoke to David and the beam has been gone since about 05:30
      Reason for the loss of beam is a vacuum issue before the FRS
      They are waiting for the experts

08:31 Statistics ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage currrents ok - attachment 3
      ASIC clock check ok
	
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2
	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  

      Currently on file R5_771
      Analysis of file R5_770 (No beam) - attachment 4
      Around 0.25% deadtime on AIDA02 rest even less

09:02 Current free HDD space 965 GB
      Current tape server rate 4979 kB/s
      Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours

11:41 Statistics ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7
      ASIC clock check ok
	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
      Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently?

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change

09:51 Taken over from Tom following the night shift
      Experiment is still running smoothly
      Compression of the files is complete up to the start of R5.
      Have started compression of files in R5 which should run up to R5_499
      Currently on R5_528

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      System wide checks:
          		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc594 : 	 87  
      aida08 fault 	 0xf1be : 	 0xf271 : 	 179  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 	Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Current merger rate is 2E6-4E6 events per second
      Current tapeserver rate is 5800 kB/s
      Free HDD space is 1.1 TB
      At current rates will last 54 hours which should see out the experiment
      Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
      
      Analysis of R5_528 - attachment 4
      Deadtime of AIDA02 currently around 15%

11:56 Statistics ok - attachment 5
      Temps ok - attachment 6
      Bias and leakage currents ok - attachment 7
      System wide checks:
      Clocks all ok 
      
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59c : 	 95  
      aida08 fault 	 0xf1be : 	 0xf27a : 	 188  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of file R5_458 - attachment 8
      Deadtime in AIDA02 only 11.5% in this file
      TapeServer rate still 5.5 MB/s

14:36 Has been no beam for the last while or so.
      With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
      Stats- attachment 9
      Temp - attachment 10
      Bias and leakage currents ok - attachment 11
      System wide checks:
      Clock ok
      	
        		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59d : 	 96  
      aida08 fault 	 0xf1be : 	 0xf27c : 	 190  
      White Rabbit error counter test result: Passed 6, Failed 2
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3c : 	 18  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

13:30 Beam taken to change an ion source

14:57 Beam back

16:21 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14

      System wide checks - ASIC clocks ok 
	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5a3 : 	 102  
      aida08 fault 	 0xf1be : 	 0xf285 : 	 199  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3e : 	 20  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Currently on file R5_592
      Analysis of R5_591 - Attachment 15
      Deadtime of AIDA02 sitting at 17%

[NH taking over for OH so he can get home]

18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
        Statitics ok - attachement 16
        Temps ok - attachement 17
        Bias & leakage ok - attachement 18

        System wide checks - 
        Clocks OK
        ADC Calibration N/A 
	WR Decoder -
        		 Base 		Current 	Difference
        aida07 fault 	 0xc53d : 	 0xc5ac : 	 111  
        aida08 fault 	 0xf1be : 	 0xf28c : 	 206  
        White Rabbit error counter test result: Passed 6, Failed 2
        FPGA -
         Base 		Current 		Difference
        aida07 fault 	 0x2a : 	 0x3f : 	 21  
        PLL OK

        Currently on file R5_621
        Analysis of R5_620 - Attachement 19
        aida02 deadtime only 5%, all others negligible 

20:20 Stats ok - attachment 20
      Temps ok - attachment 21
      Bias and leakage currents ok - attachment 22
      System wide checks:
      Clock ok
    	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ad : 	 112  
      aida08 fault 	 0xf1be : 	 0xf293 : 	 213  
      White Rabbit error counter test result: Passed 6, Failed 2

     	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3f : 	 21  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R6_637 - attachment 23

22:13 Statistics ok - attachment 24
      Temperatures ok - attachment 25
      Bias and leakage currents ok - attachment 26
      ASIC clock check ok

	
		         Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ba : 	 125  
      aida08 fault 	 0xf1be : 	 0xf2a2 : 	 228  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x40 : 	 22  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_658 - attachment 27

23.43 Check ASIC control
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

All system wide checks OK *except* WR & FPGA errors - attachments 1 & 2

DSSSD bias & leakage currents OK - attachment 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6

analysis of file S505/R5_415 - attachment 7
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 1%



02:36

analysis of file S505/R5_449 - attachment 8
 zero timewarps
 max deadtime aida02 c. 15%, all other FEE64s < 1%

per FEE64 rate & stat spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 99 ch FWHM

per FEE64 1.8.H spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


05:38

analysis of file S505/R5_483 - attachment 19
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24



07:27

analysis of file S505/R5_502 - attachment 25
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

Took over the night shift from Magda

18:00 : attachments 1-3

20:00 : attachments 4-6

22:00 : attachments 7-9

00:00 : attachments 10-12

Took over the night shift from Tom.

7:00 The beam is back.

8:00 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

14:00 : attachments 10-12

16:00 : attachments 13-15

23:36 Check ASIC contorol
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_154 - attachment 1
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

stats - attachments 2-6

per FEE64 1.8.H spectra - attachments 7 & 8

per p+n FEE64 1.8.L spectra - attachment 9
 aida01 pulser peak width 101 ch FWHM

per FEE64 stat & rate spectra - attachments 10 & 11

adc data item stats - attachment 12

FEE64 temps OK - attachment 13

DSSSD bias & leakage currents OK - attachments 14 & 15

00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17


03:03

analysis of file S505/R5_189 - attachment 18
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20

adc data item stats - attachment 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage currents OK - attachment 23

03:30

per FEE64 1.8.H spectra - attcahments 24 & 25


06:09 no beam 

analysis of file S505/R5_221 - attachment 26
 zero timewarps
 all dead times << 1%

all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28

adc data item stats - attachment 29

FEE64 temps OK - attachment 30

DSSSD bias & leakage currents OK - attachment 31

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good