AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 31 of 35  ELOG logo
ID Date Authordown Subject
  140   Tue Mar 10 23:17:31 2020 DK, LS, PW, TD, CAWednesday 11 March 2020
00:15 Shift change near AIDA R6_475

  We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.

  All system-wide checks passed successfully
  Temperatures ok (aida01 still a bit hot) - see attachment #1
  Stats as normal (SSD3 still noisy) - see attachment #2
  Bias and pulser settings remain sane - see attachment #3
  Writing data to disk at ~26 MB/s as yesterday - see attachment #4
  Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
  Rates look not unreasonable for expected implantation in short time window - see attachment #6

2:26
   We are near R6_574
   All system checks are fine
   Temps are fine - attachment #7
   Stats look as before - attachment #8
   Biases and leaks typical - attachment #9
   Merger fine, 3 x 10^6 items / sec (skip screencap)
   TapeServer still sits near 25 MB/s (skip screencap)
   Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10

2:44 R6_587 or R6_588
   The spills seem to have stopped based on the beeping scalars.  Confirmed with other group there are no scint counts.
   See attachment of AIDA rates where no implants can be seen.  See attachment #11

2:46 Spill beeps heard briefly, but they cut again.  R6_590

2:49 Beam seems to be back R6_592
  Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants.  No problem.  See attachment #12 for implants.

4:35 
  System checks all pass (except expected error on master clock)
  Temps okay - attachment #13
  Biases and leak currents normal - attachment #14
  Stats as before - attachment #15
  Merger going, 3 mega items per sec
  TapeServer going, 25 MB/s

5:59 Beam is gone again, near R6_738.  As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
   See attachment #16 which shows no implants in the rate.


   

Contact phones for Control Room: 2222

We called, they are aware of a problem with the magnets and they are working to resolve it.  So we will await them.

Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.  

We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off

7:14  Beam is still off, but I make some notes
   System wide checks all okay
   Temps as normal (see attachment #17)
   Biases and leak currents okay (see attachment #18)
   Stats as normal (see attachment #19)
   Merger going at 3 mega cps
   TapeServer on, around 25 MB/sec
   Implants not seen in histogram rate as expected (see attachment #20)

7:32 We called the operators to confirm that they don't change anything before calling us first.
     Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.

8.00 shift change at file R6_823
     beam is back on, no settings changed.
     particles per spill is similar to before (2k~3k per spill)
     implants now being seen in rate histogram (attachment 21)

8.32 system wide checks all okay
     FEE temperatures are normal (attachment 22)
     leak currents are okay and have been recorded to sheet (attachment 23)
     good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
     merger rate is running around 3M items/sec
     tape service rate is running around 26MB/sec

9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
     problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
     remaking each histogram and saving layout6 seems to be fixed (attachment 27)

Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
    invoked from within
"SpecDetails 1.8.h"
    (in namespace inscope "::urn:SpectrumService" script line 1)
    invoked from within
====

10.05 beam is being stopped to change some thresholds

10.34 system wide checks all okay
      FEE temperatures are normal (attachment 28)
      leak currents are okay and have been recorded to spreadsheet (attachment 29)
      good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24MB/sec
      implants not seen in histogram rate as expected as beam still down (see attachment 30)

10.42 beam still down but taking data, currently file finishing R6_946
      as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
      have zeroed all histograms so will take screenshots at 11.00

11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks

11.25 beam back file S480/R6_975

12.00 analysis file S480/R6_1001 - attachment 33
      FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
      all HEC data rate c. 1kHz
      all ADC data rate c. 1.2MHz

12.23 system wide checks all okay
      FEE temperatures are normal (attachment 34)
      leak currents are okay and have been recorded to spreadsheet (attachment 35)
      good event stats are okay (attachment 36)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 26MB/sec

13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
      seems to be intermittent. file R6_1051
      
13.10 beam back on, no one was called. currently on file R6_1061

13.22 beam off as going to higher intensity beam. file R6_1069

14.42 system wide checks all okay
      FEE temperatures are normal (attachment 38)
      leak currents are okay and have been recorded to spreadsheet (attachment 39)
      good event stats are okay (attachment 40)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24M/sec
14.48 on file R6_1131

16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
      file R6_1193

16:35 compression of files R6_[200-999] commenced (nice -20)

16:40 system wide checks all okay
      FEE temperatures are normal (attachment 41)
      leak currents are okay and have been recorded to spreadsheet (attachment 42)
      good event stats are okay (attachment 43)
Attachment 1: daid-11-tempsA.png
daid-11-tempsA.png
Attachment 2: daid-11-statsA.png
daid-11-statsA.png
Attachment 3: daid-11-biasA.png
daid-11-biasA.png
Attachment 4: daid-11-TSA.png
daid-11-TSA.png
Attachment 5: daid-11-mergerA.png
daid-11-mergerA.png
Attachment 6: daid-11-implantsA.png
daid-11-implantsA.png
Attachment 7: daid-11-tempB.png
daid-11-tempB.png
Attachment 8: daid-11-statsB.png
daid-11-statsB.png
Attachment 9: daid-11-biasB.png
daid-11-biasB.png
Attachment 10: daid-11-histoB.png
daid-11-histoB.png
Attachment 11: daid-11-histos-beamdrop.png
daid-11-histos-beamdrop.png
Attachment 12: daid-11-beamreturn.png
daid-11-beamreturn.png
Attachment 13: daid-11-tempC.png
daid-11-tempC.png
Attachment 14: daid-11-biasC.png
daid-11-biasC.png
Attachment 15: daid-11-statsC.png
daid-11-statsC.png
Attachment 16: daid-beamstop2.png
daid-beamstop2.png
Attachment 17: daid-11-tempD.png
daid-11-tempD.png
Attachment 18: daid-11-biasD.png
daid-11-biasD.png
Attachment 19: daid-11-statsD.png
daid-11-statsD.png
Attachment 20: daid-11-histosD.png
daid-11-histosD.png
Attachment 21: 200311_804_rateshistograms.png
200311_804_rateshistograms.png
Attachment 22: 200311_28_feetemps.png
200311_28_feetemps.png
Attachment 23: 200311_831_leakcurrents.png
200311_831_leakcurrents.png
Attachment 24: 200311_829_goodevents.png
200311_829_goodevents.png
Attachment 25: 200311_845_highEevenfees.png
200311_845_highEevenfees.png
Attachment 26: daid-11-histos-lowgaineven.png
daid-11-histos-lowgaineven.png
Attachment 27: 200311_919_highEevenfixed.png
200311_919_highEevenfixed.png
Attachment 28: 200311_1030_feetemps.png
200311_1030_feetemps.png
Attachment 29: 200311_1032_leakcurrents.png
200311_1032_leakcurrents.png
Attachment 30: 200311_1031_goodevents.png
200311_1031_goodevents.png
Attachment 31: 200311_1040_ratehistogram.png
200311_1040_ratehistogram.png
Attachment 32: 200311_1107_rates.png
200311_1107_rates.png
Attachment 33: R6_1001
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   87080536 ( 1162937.9 Hz)
        Other data format:  174903340 ( 2335788.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         31 (       0.4 Hz)
                               RESUME:         31 (       0.4 Hz)
                              SYNC100:   87327938 ( 1166241.9 Hz)
                              WR48-63:   87327938 ( 1166241.9 Hz)
                           FEE64 disc:     247402 (    3304.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      73144 (     976.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       74.880 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.144                0.000
  4                0.000                0.000
  5                0.033                0.000
  6                0.000                0.000
  7                0.033                0.000
  8                4.160                3.444
  9                0.111                0.000
 10                0.116                0.000
 11                0.111                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2537909    5321817          0          0          1          1    2619908    2619908      81999          0          0      13572
  1   12374388   24792837          0          0          0          0   12389075   12389075      14687          0          0       7144
  2    5995747   12118373          0          0          0          0    6038040    6038040      42293          0          0       8968
  3    3245929    6571235          0          0          3          3    3272386    3272386      26457          0          0      13205
  4    1018810    2094536          0          0          0          0    1037782    1037782      18972          0          0       5843
  5    3175371    6374045          0          0          1          1    3183138    3183138       7767          0          0       3654
  6    1851746    3764473          0          0          0          0    1872073    1872073      20327          0          0       9037
  7    2503081    5045008          0          0          1          1    2516029    2516029      12948          0          0       6580
  8   19164914   38353829          0          0         20         20   19172901   19172901       7987          0          0       1389
  9   15009332   30028700          0          0          2          2   15012676   15012676       3344          0          0        894
 10    5573107   11166435          0          0          2          2    5579846    5579846       6739          0          0       1608
 11   14630202   29272052          0          0          1          1   14634084   14634084       3882          0          0       1250
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:43185.973s (    0.741 blocks/s,   0.046 Mb/s)
Attachment 34: 200311_1219_feetemps.png
200311_1219_feetemps.png
Attachment 35: 200311_1221_leakcurrents.png
200311_1221_leakcurrents.png
Attachment 36: 200311_1220_goodevents.png
200311_1220_goodevents.png
Attachment 37: 200311_1300_rateshistogram.png
200311_1300_rateshistogram.png
Attachment 38: 200311_1440_FEETEMPS.png
200311_1440_FEETEMPS.png
Attachment 39: 200311_1442_leakcurrents.png
200311_1442_leakcurrents.png
Attachment 40: 200311_1442_GOODEVENTS.png
200311_1442_GOODEVENTS.png
Attachment 41: 200311_1638_temp.png
200311_1638_temp.png
Attachment 42: 200311_1639_bias.png
200311_1639_bias.png
Attachment 43: 200311_1639_stats.png
200311_1639_stats.png
  147   Thu Mar 12 23:44:48 2020 DKPID Plots
Labelled PID plots.  Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.

There was an error in the printouts for Cd isotopes, but the below attachments are corrected.

The Z-depth (color, not nuclear charge) is log scale.
Attachment 1: pid.png
pid.png
Attachment 2: pid.png
pid.png
Attachment 3: pid.pdf
pid.pdf
  148   Fri Mar 13 08:01:53 2020 DKFriday 13th 8:00 to 16:00 shift
8:00 - See previous shift's elog entry.  All good.

9:00
    All system wide checks passed
    Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
    Tape Server okay - 27 MB/s
    Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
    Stats as usual - attachment #2
    Temps normal - attachment #3
    Biases and leaks fine -  attachment #4

Disk usage projection
    Space remaining: ~2 TB
    Fill rate: ~2 GB / 1 minute
    -> 1000 minutes =~ 17 hours

Still gzipping from R6_823

10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
     	Kbytes Written:  -1986382400	Kbytes/sec:  26816
        Rate is fixed at that value on update.
        However, actual data files are being updated so data has been coming.
        DK and NH consulted and believe the situation is fine.
        We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.

10:25 Now we are on R10 after start and stop the TapeServer.  Problem is resolved.

11:30
      Implants: attachment #5
      Stats: attachment #6
      Biases / leaks: attachment #7
      Temps: attachment #8
      System wide checks performed, situation normal.
Attachment 1: daid-13-histosA.png
daid-13-histosA.png
Attachment 2: daid-13-statsA.png
daid-13-statsA.png
Attachment 3: daid-13-tempsA.png
daid-13-tempsA.png
Attachment 4: daid-13-biasA.png
daid-13-biasA.png
Attachment 5: daid-13-implantsB.png
daid-13-implantsB.png
Attachment 6: daid-13-statsB.png
daid-13-statsB.png
Attachment 7: daid-13-biasesB.png
daid-13-biasesB.png
Attachment 8: daid-13-tempsB.png
daid-13-tempsB.png
  236   Sat Apr 17 19:39:56 2021 DJ, TDSaturday 17 April 20.00-00.00
19.37 per FEE64 rate spectra - attachments 1 & 2
      1.8.L spectra - attachments 3 & 4
      1.8.H spectra - attachments 5-8
      1.8.W spectra - attachments 9 & 10

20.42 DAQ contrinues file NULL/R30_233



		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a55 : 	 3  
aida06 fault 	 0x4f3e : 	 0x4f41 : 	 3  
aida07 fault 	 0x3bcd : 	 0x3bd7 : 	 10  
aida08 fault 	 0xc7c7 : 	 0xc7ca : 	 3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-------
23:48
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a55 : 	 3  
aida06 fault 	 0x4f3e : 	 0x4f41 : 	 3  
aida07 fault 	 0x3bcd : 	 0x3bde : 	 17  
aida08 fault 	 0xc7c7 : 	 0xc7ca : 	 3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x7 : 	 7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26      5      6      2      1      3      1      3      3      3      6   : 36336
aida02 :      7      9     10      3      2      2      2      4      2      3      6   : 36068
aida03 :     26      7      8      4      3      4      2      4      2      3      6   : 36448
aida04 :     18      8      2      1      2      2      3      4      2      3      6   : 36168
aida05 :     24      7      9      4      3      1      3      2      2      4      6   : 37352
aida06 :     17     10      3      1      3      5      1      3      3      3      6   : 36644
aida07 :      7     11      2      2      3      2      3      2      4      3      6   : 37268
aida08 :     23      4      1      5      1      1      2      3      3      3      6   : 36332
aida09 :     14      6      3      2      1      3      2      3      3      3      6   : 36504
aida10 :      2      2      2      2      1      2      2      2      3      3      6   : 35768
aida11 :      2      2      1      2      2      2      2      4      2      3      6   : 35816
aida12 :     26      8      5      5      4      3      1      3      3      3      6   : 36632
----
Attachment 1: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(5).png
Attachment 7: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(6).png
Attachment 8: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(7).png
Attachment 9: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(8).png
Attachment 10: Screenshot_2021-04-17_Statistics_aidas-gsi(4).png
Screenshot_2021-04-17_Statistics_aidas-gsi(4).png
Attachment 11: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 12: 50.png
50.png
Attachment 13: 51.png
51.png
  238   Sun Apr 18 07:30:20 2021 DJ, TDSunday 18th April 08:00-12:00

---- 09-26
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

---


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

--


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

 

08.45 TD resets baseline for WR and FPGA errors

 

Attachment 1: Screenshot_2021-04-18_Statistics_aidas-gsi.png
Screenshot_2021-04-18_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 52.png
52.png
Attachment 4: 53.png
53.png
Attachment 5: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 6: 54.png
54.png
Attachment 7: Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
  263   Thu Apr 22 23:17:46 2021 DJ Friday 23rd April 00:00-08:00

Took over from TD ... All is well.

00:45 System Wide Checks

Clock: 12 passed, 0 failed.

ADC Calibration: Passed 12, 0 failed.

White Rabbit Decoder: Passed 4,failed 8. - See screenshot.

FPGA Passed 11, failed 1. - See screenshot.

FEE64 Linux: See screenshot.

-----

02:49 run closed. Checking beam.

03:00 File 55 opened.

03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.

File 56 opened at 4:24.

---------

04:30 System Wide Checks (See screenshots)

White Rabbit Decoder: Passed 12, failed 0.

FPGA: page not loading properly.

----

05:37 System wide checks. See screenshots

Clock: Status: 11 passed. 1 fail.

ADC calibration: Passed 9, failed 3.

White Rabbit decoder: Passed 12, failed 0.

-----

07:32 System wide checks (See screenshots).

 

 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi.png
Screenshot_2021-04-23_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot-CAEN_HV_Supply.png
Screenshot-CAEN_HV_Supply.png
Attachment 4: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Attachment 5: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 60.png
60.png
Attachment 10: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Attachment 12: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Attachment 13: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Attachment 17: 61.png
61.png
Attachment 18: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Attachment 19: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Attachment 20: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Attachment 21: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 22: Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Attachment 23: 63.png
63.png
Attachment 24: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Attachment 25: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Attachment 26: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Attachment 27: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
  695   Fri Jan 31 09:37:47 2025 CC, TD, MPFriday 31 January
Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000

N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.

n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted

BB7 aida10 (asics #1-2 p+n, #3-4 n+n)

BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream) 

10.30 FEE64 power ON

      DSSSD bias ON (BB18 only) - attachment 1
       leakage current OK

      ASIC settings 2024Dec13-17.02.45 restored
       p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf

      Attachments 2-7 WR timestamp aida05 & aida12, system wide check
       aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
       WR decoder status aida02

      per FEE64 Rate spectra - attachment 8
       aida10 high rate - BB7 not biased
       rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
       BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments

      ADC data item stats - attachment 9
       rates generally OK
       aida10 high - BB7 not biased
       aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
       all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!

      BNC PB-5 settings - attachment 10

      WR timestamps OK - attachment 11

      FEE64 temperatures OK - attachment 12

12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet

      ADC data item stats - attachment 13
       all rates higher cf. attachment 9

      per FEE64 Rate spectra - attachment 14

      per p+n FEE64 1.8.L spectra - attachment 15
       aida09 pulser peak width 69 ch FWHM = 49keV FWHM 
       aida14 pulser peak width 50 ch FWHM = 35keV FWHM

      per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17

      per n+n FEE64 1.8W spectra - 20us FSR - attachment 18

      DSSSD bias & leakage current OK - attachment 19
       ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%

      FEE64 temperatures OK - attachment 20

      DSSSD bias volatge & leakage current OK - attachment 21
       CAEN N1419ET ch#0 BB18, ch#1 BB7

      Install of bPlas drivers, cabling, grounds complete

      ADC data item stats - attachment 22
       
      per FEE64 Rate spectra - attachment 23
       all BB18 p+n FEE64s *except* aida05 show very low rates of noise

      per p+n FEE64 1.8.L spectra - attachment 24
       aida09 pulser peak width 62 ch FWHM = 42keV FWHM
       BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
Attachment 1: Screenshot_from_2025-01-31_10-39-16.png
Screenshot_from_2025-01-31_10-39-16.png
Attachment 2: Screenshot_from_2025-01-31_10-58-48.png
Screenshot_from_2025-01-31_10-58-48.png
Attachment 3: Screenshot_from_2025-01-31_10-58-59.png
Screenshot_from_2025-01-31_10-58-59.png
Attachment 4: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 5: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 6: Screenshot_from_2025-01-31_10-59-52.png
Screenshot_from_2025-01-31_10-59-52.png
Attachment 7: Screenshot_from_2025-01-31_11-00-02.png
Screenshot_from_2025-01-31_11-00-02.png
Attachment 8: Screenshot_from_2025-01-31_11-17-59.png
Screenshot_from_2025-01-31_11-17-59.png
Attachment 9: Screenshot_from_2025-01-31_11-18-43.png
Screenshot_from_2025-01-31_11-18-43.png
Attachment 10: Screenshot_from_2025-01-31_11-18-50.png
Screenshot_from_2025-01-31_11-18-50.png
Attachment 11: Screenshot_from_2025-01-31_11-21-05.png
Screenshot_from_2025-01-31_11-21-05.png
Attachment 12: Screenshot_from_2025-01-31_11-21-29.png
Screenshot_from_2025-01-31_11-21-29.png
Attachment 13: Screenshot_from_2025-01-31_12-21-09.png
Screenshot_from_2025-01-31_12-21-09.png
Attachment 14: Screenshot_from_2025-01-31_12-21-55.png
Screenshot_from_2025-01-31_12-21-55.png
Attachment 15: Screenshot_from_2025-01-31_12-26-26.png
Screenshot_from_2025-01-31_12-26-26.png
Attachment 16: Screenshot_from_2025-01-31_12-30-13.png
Screenshot_from_2025-01-31_12-30-13.png
Attachment 17: Screenshot_from_2025-01-31_12-30-34.png
Screenshot_from_2025-01-31_12-30-34.png
Attachment 18: Screenshot_from_2025-01-31_12-31-25.png
Screenshot_from_2025-01-31_12-31-25.png
Attachment 19: Screenshot_from_2025-01-31_12-41-33.png
Screenshot_from_2025-01-31_12-41-33.png
Attachment 20: Screenshot_from_2025-01-31_12-42-08.png
Screenshot_from_2025-01-31_12-42-08.png
Attachment 21: Screenshot_from_2025-01-31_15-59-26.png
Screenshot_from_2025-01-31_15-59-26.png
Attachment 22: Screenshot_from_2025-01-31_16-07-21.png
Screenshot_from_2025-01-31_16-07-21.png
Attachment 23: Screenshot_from_2025-01-31_16-07-31.png
Screenshot_from_2025-01-31_16-07-31.png
Attachment 24: Screenshot_from_2025-01-31_16-08-39.png
Screenshot_from_2025-01-31_16-08-39.png
Attachment 25: Screenshot_from_2025-01-31_16-10-48.png
Screenshot_from_2025-01-31_16-10-48.png
Attachment 26: Screenshot_from_2025-01-31_16-11-30.png
Screenshot_from_2025-01-31_16-11-30.png
  697   Fri Jan 31 17:26:47 2025 CC, TD, MPFriday 31 January contd.
18.18 bPlas and BB7 installs complete 
      Restart AIDA DAQ, Merger, Tape Server and re-test

      per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 69 ch FWHM = 42keV FWHM

      per FEE64 Rate Spectra - attachment 2
       BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
       BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK

      per FEE64 1.8.W spectra - 20us FSR - attachments 3-5

      ADC data item stats - attachment 6
       aida01  aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
       BB7 aida10 19k

      WR timestamps OK - attachment 7

      Merger, TapeSever etc - attachments 8-9
       disk directpry /TapeData/FEB25
       working OK
     
      DSSSD bias & leakage current OK - attachment 10

      System wide checks OK - attachment 11-15
       note global clock status 6 errors reported earlier today have now gone

      FEE64 temperatures OK - attachment 16

18.38 Transition to safe state

      DAQ STOP
      disable data transfer 1
      detector bias OFF
      FEE64 power OFF


      Can restart as follows

1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
     sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps

If all OK can re-connect to Merger/TapeServer as follows

1) DAQ STOP
2) enable data transfer 1
3) DAQ GO


Can disconnect from Merger/TapeServer as follows

1) DAQ STOP
2) disable data transfer 1
   
Attachment 1: Screenshot_from_2025-01-31_18-18-17.png
Screenshot_from_2025-01-31_18-18-17.png
Attachment 2: Screenshot_from_2025-01-31_18-19-19.png
Screenshot_from_2025-01-31_18-19-19.png
Attachment 3: Screenshot_from_2025-01-31_18-19-37.png
Screenshot_from_2025-01-31_18-19-37.png
Attachment 4: Screenshot_from_2025-01-31_18-20-04.png
Screenshot_from_2025-01-31_18-20-04.png
Attachment 5: Screenshot_from_2025-01-31_18-20-30.png
Screenshot_from_2025-01-31_18-20-30.png
Attachment 6: Screenshot_from_2025-01-31_18-20-42.png
Screenshot_from_2025-01-31_18-20-42.png
Attachment 7: Screenshot_from_2025-01-31_18-21-01.png
Screenshot_from_2025-01-31_18-21-01.png
Attachment 8: Screenshot_from_2025-01-31_18-25-58.png
Screenshot_from_2025-01-31_18-25-58.png
Attachment 9: Screenshot_from_2025-01-31_18-26-13.png
Screenshot_from_2025-01-31_18-26-13.png
Attachment 10: Screenshot_from_2025-01-31_18-29-38.png
Screenshot_from_2025-01-31_18-29-38.png
Attachment 11: Screenshot_from_2025-01-31_18-29-52.png
Screenshot_from_2025-01-31_18-29-52.png
Attachment 12: Screenshot_from_2025-01-31_18-29-58.png
Screenshot_from_2025-01-31_18-29-58.png
Attachment 13: Screenshot_from_2025-01-31_18-30-06.png
Screenshot_from_2025-01-31_18-30-06.png
Attachment 14: Screenshot_from_2025-01-31_18-30-12.png
Screenshot_from_2025-01-31_18-30-12.png
Attachment 15: Screenshot_from_2025-01-31_18-30-30.png
Screenshot_from_2025-01-31_18-30-30.png
Attachment 16: Screenshot_from_2025-01-31_18-30-48.png
Screenshot_from_2025-01-31_18-30-48.png
  574   Mon Apr 15 09:23:43 2024 CC, JB, HMA, TDNoise in AIDA when connecting BB7

9:40 started up AIDA. TEMP OK. Noise condition same as Saturday. Attachments 1-3.

9:56 BB7 premaps connected but not powered. Attahcment 4.

10:20 BB7 Preamps powered. Attachment 5.

Noise increased marginally, but still 10/16 <20kHz, max 130k.

 

 

15.15  per FEE64 Rate spectra - attachment 6

             ADC data item stats - attachment 7

           increased noise observed - 6/16 < 20k, max 300k 

 

17.10  per FEE64 1.8.W spectra - 20us FSR - attachments 8-9

            per FEE64 Rate spectra - attachment 10

             ADC data item stats - attachment 11

           increased noise observed - 7/16 < 20k, max 180k 

20.30  per FEE64 Rate spectra - attachment 12

             ADC data item stats - attachment 13

           status quo ante - 10/16 < 20k, max 130k 

 

          FEE64 power OFF

          DSSSD bias OFF

 

 

Attachment 1: Screenshot_from_2024-04-15_09-55-15.png
Screenshot_from_2024-04-15_09-55-15.png
Attachment 2: Screenshot_from_2024-04-15_09-54-58.png
Screenshot_from_2024-04-15_09-54-58.png
Attachment 3: Screenshot_from_2024-04-15_09-54-17.png
Screenshot_from_2024-04-15_09-54-17.png
Attachment 4: Screenshot_from_2024-04-15_10-13-20.png
Screenshot_from_2024-04-15_10-13-20.png
Attachment 5: Screenshot_from_2024-04-15_10-24-39.png
Screenshot_from_2024-04-15_10-24-39.png
Attachment 6: Screenshot_from_2024-04-15_15-24-34.png
Screenshot_from_2024-04-15_15-24-34.png
Attachment 7: Screenshot_from_2024-04-15_15-23-41.png
Screenshot_from_2024-04-15_15-23-41.png
Attachment 8: Screenshot_from_2024-04-15_17-16-19.png
Screenshot_from_2024-04-15_17-16-19.png
Attachment 9: Screenshot_from_2024-04-15_17-15-27.png
Screenshot_from_2024-04-15_17-15-27.png
Attachment 10: Screenshot_from_2024-04-15_17-14-32.png
Screenshot_from_2024-04-15_17-14-32.png
Attachment 11: Screenshot_from_2024-04-15_17-13-37.png
Screenshot_from_2024-04-15_17-13-37.png
Attachment 12: Screenshot_from_2024-04-15_20-29-18.png
Screenshot_from_2024-04-15_20-29-18.png
Attachment 13: Screenshot_from_2024-04-15_20-28-48.png
Screenshot_from_2024-04-15_20-28-48.png
  41   Tue Apr 2 07:39:44 2019 CB, TD, CATuesday 2nd April 2019
08:15 beam tuning in progress

08.30 System wide checks 

- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- 'collect memory information from FEE64 Linux' still returns scripting error -> will not check again until update
- WR decoder test passes
- good event statistics OK, except aida 09, 10, 11, 12 as yesterday morning (see attach 1)
- FEE temperature OK (see attach 2)
- Detector OK (see attach 3)
- NewMerger. MergerHTTPd, newMerger Terminal, data relay MBS screenshots attached (4,5,6,7)

10:37 Issues with magnets due to the new control system. Beam tuning continues

AIDA continues running OK
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- WR decoder test passes
- good event statistics OK, except aida 09, 10 (see attach 8). Aida 9 & 12 rates now lower
- FEE temperature OK
- Detector OK
- Merger OK

12:21 Beam start
AIDA continues running OK
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- WR decoder test passes
-ASIC check load performed
-Writing to file R11 in directory 290319
-histograms zeroed
-good event statistics OK, except aida 09, 10 (attachment 9)
-FEE temperatures ok (attachment 10)
-detector bias ok (attachment 11)

-thick degrader was placed in front of snout during beam operation

12:40 beam stop
      TapeServer stopped, file R11_0 in 290319
      TapeServer restarted, not writing to file

      low energy spectra for aida09 (attachment 12)

      high energy spectra for aida09 (attachment 13)

      high energy spectra for aida11 (attachment 14)

      1D hit patterns (stat) in aida09,10,11,12 (attachment 15)

      ASIC control settings used during beam, including thresholds (attachment 16)

14.20 *NOTE degrader setting different from yesterday, silver beam still not observed, just unidentified light ions.

      AIDA continues running OK
      aida06 global clock failure (check clock status)
      aida02 , aida04, aida06 fail calibration (ADC calibration)
      SYNC error counter ok
      SYNC pulse check ok
      WR decoder test passes

14.37 good event statistics OK, except aida 10 (attachment 17)
      detector bias ok (attachment 18)
      FEE temperatures ok (attachment 19)

14.54 Degrader removed 

16:13 AIDA continues running OK
      aida06 global clock failure (check clock status)
      aida02 , aida04, aida06 fail calibration (ADC calibration)
      SYNC error counter ok
      SYNC pulse check ok
      WR decoder test passes
      FEE64 temperatures OK
      good event statistics OK, except aida 3, 9, 10 running hot (see attach 18)

17:00 Beam start - no degrader
      AIDA running OK
      Writing on file R13
      Beam is better focused (see attach 19). Appears to be off-centre in the bottom right part of the detector
      Rates OK (see attach 20)
      DSSD bias/current OK (see attach 21)

17:21 Beam re-centred. Histograms cleared - see attach 22
      AIDA continues OK
      Run continues

      High energy spectra in aida09 (attachment 25)

      Beam rate approx 5000 ions/s

      Beam being diffused
      TapeServer start stop - AIDA writing to file R14 in 290319

17:48 1D hit patterns - beam in process of being diffused (attachment 26)

17:57 leakage current back at 1.465 uA (attachment 27)

18:08 beam stopped
      TapeServer stopped and restarted - not writing to file

      updated hit patterns from beam diffusing (attachment 28)

      aida10, ADC3 high energy spectra (attachment 29)

18.11 calibrating tpc with masks

      TapeServer stop/start R16, 290319

18.49 leakage current briefly fluctuating between ~1.2 uA and ~2 uA, but has returned to stability at ~ 1.4 uA
(attachment 30)

18.55 beam stopped for tuning - TapeServer stop/start. Not writing to file.

18.58 leakage current still stable after beam off (attachment 31)

19.10 beam start. TapeServer stop/start writing to file R18 in 290319

19.15 leakage current still ok after beam activation

19.34 fee12 high energy spectra (attachment 32)

20.35 beam stop, tapeserver stopped

      histogramming link dropped on aida 12

20.38 beam start. TapeServer start R20
      
      
Attachment 1: 71.png
71.png
Attachment 2: 72.png
72.png
Attachment 3: 73.png
73.png
Attachment 4: 74.png
74.png
Attachment 5: 75.png
75.png
Attachment 6: 76.png
76.png
Attachment 7: 76.png
76.png
Attachment 8: 78.png
78.png
Attachment 9: 020419_1227_stats.png
020419_1227_stats.png
Attachment 10: 020419_1228_temp.png
020419_1228_temp.png
Attachment 11: 020419_1230_bias.png
020419_1230_bias.png
Attachment 12: 020419_1235_lowenergyspectraaida09.png
020419_1235_lowenergyspectraaida09.png
Attachment 13: 020419_HEspectra_09(2).png
020419_HEspectra_09(2).png
Attachment 14: 040219_1257_HEspectra11.png
040219_1257_HEspectra11.png
Attachment 15: highenergyhitpatters020419_1300.png
highenergyhitpatters020419_1300.png
Attachment 16: 020419_noon_asiccontrol.png
020419_noon_asiccontrol.png
Attachment 17: 020419_1434_stats.png
020419_1434_stats.png
Attachment 18: 020419_1436_bias.png
020419_1436_bias.png
Attachment 19: 040219_1454_temp.png
040219_1454_temp.png
Attachment 20: 79.png
79.png
Attachment 21: 80.png
80.png
Attachment 22: 81.png
81.png
Attachment 23: 82.png
82.png
Attachment 24: 84.png
84.png
Attachment 25: 020419_1737_HEspectra09.png
020419_1737_HEspectra09.png
Attachment 26: 020419_1748_hitpattern.png
020419_1748_hitpattern.png
Attachment 27: 020419_1756_bias.png
020419_1756_bias.png
Attachment 28: 020419_1803_hitpattern.png
020419_1803_hitpattern.png
Attachment 29: 020419_1807_315H.png
020419_1807_315H.png
Attachment 30: 020418_1851_bias.png
020418_1851_bias.png
Attachment 31: 020419_1859_bias.png
020419_1859_bias.png
Attachment 32: 040219_1932_22h_12.png
040219_1932_22h_12.png
Attachment 33: 020419_1954_hitpatter.png
020419_1954_hitpatter.png
Attachment 34: 020419_2003_hitpattern.png
020419_2003_hitpattern.png
Attachment 35: 020419_2006_waveforms.png
020419_2006_waveforms.png
Attachment 36: 020419_2035_bias.png
020419_2035_bias.png
  429   Wed May 11 19:08:41 2022 CB, PPWednesday 11th May - evening shift

20:19 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

 

22:12 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

Attachment 1: 16.png
16.png
Attachment 2: 22.png
22.png
Attachment 3: 13.png
13.png
Attachment 4: 14.png
14.png
  308   Wed May 12 15:03:10 2021 CB, OHWed 12 16:00-20:00
16:00 Shift change

Statistics OK (attach 1)
Temps OK (attach 2)
Bias OK (attach 3)

System wide checks
Clock, ADC - OK

WR
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36cb : 	 1  
White Rabbit error counter test result: Passed 15, Failed 1


FPGA

			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0x4b : 	 65  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      5      6      1      1      2      2      3      3      3      6   : 36388
aida02 :     21     10      2      2      2      3      2      2      3      3      6   : 36100
aida03 :     25      5      3      1      0      3      3      2      3      3      6   : 36188
aida04 :      5      2      0      3      2      3      2      3      3      3      6   : 36484
aida05 :     16      6      3      1      2      4      1      3      2      4      6   : 37440
aida06 :      2      2      2      2      2      3      1      3      3      3      6   : 36216
aida07 :     16      9      3      0      3      4      3      2      3      3      6   : 36472
aida08 :      0      7      1      1      2      2      3      3      2      3      6   : 35560
aida09 :     15      7      4      2      2      4      2      3      2      3      6   : 35700
aida10 :      3      7      3      4      1      3      1      4      1      4      6   : 36788
aida11 :      5      1      3      1      1      4      3      3      2      3      6   : 35756
aida12 :      5      5      0      3      2      3      2      4      2      3      6   : 35996
aida13 :      2      5      1      2      2      4      3      3      2      3      6   : 35840
aida14 :     19      5      3      1      1      2      1      3      3      3      6   : 36100
aida15 :     20      6      6      2      1      3      2      2      3      3      6   : 36064
aida16 :      7      9      4      1      3      4      2      3      2      3      6   : 35716

File size change
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 10 16:47:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


16:20 Beam is back. Currently on File R1_200
16:43 Beam is being optimised.

17:00 UCESB OK
Stats OK (attach 4)
Temps OK
Bias OK (attach 5)
System-wide checks as before

17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK

17:52 Started compressing R1_1*

18:00 UCESB ok (attach 5)
Stats OK (attach 6)
Temps OK
Bias OK (attach 7)
System wide checks as before

19:00 UCESB ok (as before)
Stats OK (attach 8)
Temps OK
Bias OK (attach 9)
Merger OK (attach 10)

19:32 UCESB ok
Stats OK (attach 11)
Temps OK (attach 12)
Bias OK (attach 13)
System wide checks as before

19:51 Shift handed over to James S
Attachment 1: Stat1.png
Stat1.png
Attachment 2: Temp1.png
Temp1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Stats2.png
Stats2.png
Attachment 5: Bias2.png
Bias2.png
Attachment 6: Stats3.png
Stats3.png
Attachment 7: Bias3.png
Bias3.png
Attachment 8: ucesb1.png
ucesb1.png
Attachment 9: Stats4.png
Stats4.png
Attachment 10: Bias4.png
Bias4.png
Attachment 11: Merger.png
Merger.png
Attachment 12: ucesb2.png
ucesb2.png
Attachment 13: Stats5.png
Stats5.png
Attachment 14: Temps2.png
Temps2.png
Attachment 15: Bias5.png
Bias5.png
  317   Fri May 14 14:57:32 2021 CB + OH, TD, NHFriday 14th May 16:00-20:00
16:00 Shift taken over from CA

16:00 
Temperatures OK. Attach 1. 
Upon reloading aida11 first returns "No respose" for all values. 
Upon refreshing, aida11 responds but aida 9 returns "No response". 
Upon refreshing again, attach 1.

Bias OK. Attach 2.

Statistics OK. Attach 3-7

System-wide checks
Clock OK
ADC - all fail
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     32     28     25      7      1      4      1      3      2      3      6   : 36112
aida02 :     43     28     29      5      4      2      3      4      1      3      6   : 36092
aida03 :     39     26     28      5      4      3      2      5      1      3      6   : 36428
aida04 :      9      5     21      8      5      4      4      4      2      2      6   : 35292
aida05 :     46     32     26      7      2      2      2      2      2      3      6   : 35768
aida06 :      4      5     16      1      3      4      2      4      1      3      6   : 35352
aida07 :     47     21     25      5      4      3      1      4      1      3      6   : 35604
aida08 :     62     33     32      6      0      2      3      3      3      2      6   : 35520
aida09 :     33     25     28      7      3      5      2      4      1      3      6   : 36140
aida10 :     47     29     22      6      3      2      2      2      2      3      6   : 35716
aida11 :     39     30     29      6      2      3      1      3      2      3      6   : 36124
aida12 :     38     36     20      6      3      2      3      3      3      2      6   : 35448
aida13 :     42     29     24      3      4      2      3      3      3      2      6   : 35440
aida14 :     31     28     18      7      3      3      4      4      2      2      6   : 35228
aida15 :     48     25     30      5      3      2      4      4      2      2      6   : 35272
aida16 :     37     16     28      6      3      3      2      2      2      3      6   : 35796

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021



16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time. 
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
      Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20



17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17

System-wide checks
Clock - all pass
ADC - all pass
WR
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     56     35     27      8      1      4      1      2      2      3      6   : 35816
aida02 :     43     31     27      5      3      1      3      3      2      3      6   : 36404
aida03 :     45     28     28     16      4      2      2      4      1      3      6   : 36180
aida04 :      2      2     18     12      4      3      3      4      2      2      6   : 34872
aida05 :     28     28     24     10      2      2      2      2      1      4      6   : 36752
aida06 :     28     11      9      5      2      3      3      3      1      3      6   : 35064
aida07 :     52     24     23      9      4      3      1      2      2      3      6   : 35744
aida08 :     21     26     27      8      0      2      3      3      3      2      6   : 35284
aida09 :     40     32     26     10      3      5      1      3      1      3      6   : 35520
aida10 :     46     23     27      7      3      1      1      3      3      3      6   : 36928
aida11 :     49     31     20     10      2      2      1      2      2      3      6   : 35516
aida12 :     39     35     17      9      2      2      3      4      3      2      6   : 35940
aida13 :     40     32     20      7      4      2      3      4      3      2      6   : 36032
aida14 :     33     33     16     10      3      4      3      4      3      2      6   : 36236
aida15 :     53     27     25     12      3      2      4      4      2      2      6   : 35452
aida16 :     35     22     32     15      3      4      3      3      1      3      6   : 36060

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24

System-wide checks
Clock - all pass
ADC - all pass
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     31     31      8      1      4      1      2      2      3      6   : 35788
aida02 :     42     31     29      5      3      2      4      4      1      3      6   : 36304
aida03 :     29     30     29     16      4      3      2      4      1      3      6   : 36276
aida04 :     28      4     20     12      4      3      4      4      2      2      6   : 35280
aida05 :     25     28     23     10      2      2      2      2      1      4      6   : 36724
aida06 :      5      3     10      5      2      4      2      2      2      3      6   : 35308
aida07 :     38     26     22      9      4      3      1      2      2      3      6   : 35688
aida08 :     52     30     28      8      0      2      2      3      3      2      6   : 35200
aida09 :     22     26     21     10      3      4      1      4      1      3      6   : 35704
aida10 :     37     28     25      7      3      1      1      3      3      3      6   : 36900
aida11 :     53     30     20     10      2      3      2      2      2      3      6   : 35908
aida12 :     37     34     17     10      2      2      3      4      3      2      6   : 35956
aida13 :     47     34     16      7      4      2      3      4      3      2      6   : 36012
aida14 :     37     29     17     10      3      4      3      4      3      2      6   : 36236
aida15 :     44     28     25     12      3      2      4      4      2      2      6   : 35424
aida16 :     40     21     33     15      3      4      1      3      1      3      6   : 35576

18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%


19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32

System-wide checks

Clock, ADC - all pass
WR

	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc4 : 	 12  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     33     30      8      1      4      1      2      2      3      6   : 35788
aida02 :     51     30     30      5      3      2      3      4      1      3      6   : 36092
aida03 :     52     31     28     16      4      2      2      5      1      3      6   : 36744
aida04 :      6      8     20     12      4      4      3      4      2      2      6   : 35096
aida05 :     30     31     22     10      2      2      2      2      1      4      6   : 36752
aida06 :     12      6     10      5      2      3      3      2      2      3      6   : 35488
aida07 :     53     24     21      9      4      3      1      3      2      3      6   : 36228
aida08 :     36     28     28      9      0      2      3      3      3      2      6   : 35408
aida09 :     27     28     25     10      3      4      2      3      1      3      6   : 35548
aida10 :     50     23     26      7      3      1      1      3      3      3      6   : 36928
aida11 :     42     28     22     10      3      3      2      2      2      3      6   : 35944
aida12 :     39     35     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     43     34     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     42     32     16     10      3      4      3      4      3      2      6   : 36264
aida15 :     36     30     25     12      3      2      4      3      3      2      6   : 35920
aida16 :     40     21     25     15      3      4      2      3      1      3      6   : 35704

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19:40 Analysed R5_86. Attached. Dead time as before.

19:57 Shift handed over to James S
Attachment 1: Temp1.png
Temp1.png
Attachment 2: Bias1.png
Bias1.png
Attachment 3: Stats1-1.png
Stats1-1.png
Attachment 4: Stats1-2.png
Stats1-2.png
Attachment 5: Stats1-3.png
Stats1-3.png
Attachment 6: Stats1-4.png
Stats1-4.png
Attachment 7: Stats1-5.png
Stats1-5.png
Attachment 8: Stats1-6.png
Stats1-6.png
Attachment 9: R5_45_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
Attachment 10: Temps2.png
Temps2.png
Attachment 11: Bias2.png
Bias2.png
Attachment 12: Stats2-1.png
Stats2-1.png
Attachment 13: Stats2-2.png
Stats2-2.png
Attachment 14: Stats2-3.png
Stats2-3.png
Attachment 15: Stats2-4.png
Stats2-4.png
Attachment 16: Stats2-5.png
Stats2-5.png
Attachment 17: Stats2-6.png
Stats2-6.png
Attachment 18: Temp3.png
Temp3.png
Attachment 19: Bias3.png
Bias3.png
Attachment 20: Stats3-1.png
Stats3-1.png
Attachment 21: Stats3-2.png
Stats3-2.png
Attachment 22: Stats3-3.png
Stats3-3.png
Attachment 23: Stats3-4.png
Stats3-4.png
Attachment 24: Stats3-5.png
Stats3-5.png
Attachment 25: R5_67_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255938817 ( 1305497.2 Hz)
        Other data format:    5981183 (   30508.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.6 Hz)
                               RESUME:        113 (       0.6 Hz)
                              SYNC100:      32730 (     166.9 Hz)
                              WR48-63:      32730 (     166.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5915496 (   30173.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     257806 (    1315.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      196.047 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.111              102.399
  2                0.513              105.076
  3                3.587               39.451
  4                0.000              182.105
  5                0.792              103.780
  6                0.167              115.991
  7                1.244               72.680
  8                0.805               56.828
  9                0.000              190.127
 10                2.629                6.967
 11                0.295               76.751
 12                0.222              114.996
 13                0.000              188.225
 14                0.031              124.690
 15                2.491               63.947
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8223770       3364          0          0          2          2       1092       1092          0       1176          0      13825
  1   16628095       4338          0          0         12         12       2157       2157          0          0          0      28636
  2   13173045    1831289          0          0          3          3       1864       1864          0    1827555          0      22667
  3   27251797    1807380          0          0         13         13       3668       3668          0    1800018          0      33190
  4    2564061     622927          0          0          0          0        404        404          0     622119          0       6108
  5   14810679     385418          0          0          6          6       1903       1903          0     381600          0      19053
  6   12067625     742516          0          0          6          6       1616       1616          0     739272          0      12600
  7   21266971     549224          0          0         11         11       2723       2723          0     543756          0      22659
  8   23290358       5786          0          0          5          5       2888       2888          0          0          0      28199
  9    1157906        324          0          0          0          0        162        162          0          0          0       2443
 10   45608185      11489          0          0         27         26       5718       5718          0          0          0      16139
 11   20146427       4876          0          0          6          6       2432       2432          0          0          0       5579
 12   13411417       3300          0          0          7          7       1643       1643          0          0          0      24922
 13    2681820        654          0          0          0          0        327        327          0          0          0       1925
 14   11884997       2904          0          0          1          1       1451       1451          0          0          0      15892
 15   21771664       5394          0          0         15         15       2682       2682          0          0          0       3969
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:67311.180s (    0.475 blocks/s,   0.030 Mb/s)
Attachment 26: Stats_RESUME.png
Stats_RESUME.png
Attachment 27: Stats-CORRELATION.png
Stats-CORRELATION.png
Attachment 28: Bias4.png
Bias4.png
Attachment 29: Stats-good.png
Stats-good.png
Attachment 30: Stats-AIDAADC.png
Stats-AIDAADC.png
Attachment 31: Stats_DISC.png
Stats_DISC.png
Attachment 32: Stats_PAUSE.png
Stats_PAUSE.png
Attachment 33: R5_86_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256871525 ( 1549964.4 Hz)
        Other data format:    5048475 (   30462.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        133 (       0.8 Hz)
                               RESUME:        132 (       0.8 Hz)
                              SYNC100:      32617 (     196.8 Hz)
                              WR48-63:      32617 (     196.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4982976 (   30067.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     217412 (    1311.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      165.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.819               66.425
  2                0.365               78.785
  3                3.792               29.131
  4                0.000              149.995
  5                0.868               74.751
  6                0.080               85.583
  7                2.138               33.988
  8                2.899               39.410
  9                0.000              164.566
 10                6.499               11.006
 11                1.139               43.564
 12                0.944               80.601
 13                0.000              152.925
 14                0.102               87.597
 15                2.546               36.680
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8622288       3316          0          0          4          3       1158       1158          0        993          0      11812
  1   17129008       4510          0          0         12         12       2243       2243          0          0          0      24429
  2   13474006    1553973          0          0          6          6       1869       1869          0    1550223          0      19028
  3   26129234    1524704          0          0         20         20       3374       3374          0    1517916          0      27835
  4    2168475     521045          0          0          0          0        352        352          0     520341          0       5247
  5   15823578     326080          0          0          4          4       1963       1963          0     322146          0      16074
  6   12515224     622754          0          0          2          2       1625       1625          0     619500          0      10516
  7   23990039     458079          0          0         12         12       3099       3099          0     451857          0      18924
  8   22108276       5660          0          0         13         13       2817       2817          0          0          0      24206
  9    1062355        236          0          0          0          0        118        118          0          0          0       1949
 10   41223894      10244          0          0         40         40       5082       5082          0          0          0      13136
 11   20572413       4974          0          0          7          7       2480       2480          0          0          0       4577
 12   14036482       3446          0          0          3          3       1720       1720          0          0          0      21370
 13    3191210        818          0          0          0          0        409        409          0          0          0       1573
 14   12949121       3198          0          0          3          3       1596       1596          0          0          0      13450
 15   21875922       5438          0          0          7          7       2712       2712          0          0          0       3286
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:70812.570s (    0.452 blocks/s,   0.028 Mb/s)
  249   Mon Apr 19 19:19:18 2021 CBMonday 19 April 20:00-24:00 shift
20:19 
Statistics OK (attach 1)
No faults found
Ucesb OK (attach 2)
Temps OK (attach 3)
Bias OK (attach 4)

System-wide checks - as per last check
aida09 fails clock (OK)
aida07 and 09 fail calibration (OK)

		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05f : 	 13  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x29 : 	 41  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      0      3      2      2      2      5      2      3      2      3      6   : 35704
aida02 :     19      4      1      1      1      2      2      3      3      3      6   : 36316
aida03 :     12      9      2      1      2      2      2      3      3      3      6   : 36408
aida04 :     16      3      5      4      2      4      2      3      3      3      6   : 36776
aida05 :     41     10     10      3      3      3      3      3      3      3      6   : 37172
aida06 :     25     21      9      3      2      3      1      4      3      3      6   : 37116
aida07 :     10      7      1      2      1      3      2      2      3      3      6   : 35952
aida08 :     33      7      5      1      1      2      2      3      3      3      6   : 36460
aida09 :     12      7      4      2      2      2      2      2      2      4      6   : 36968
aida10 :     16      6      2      1      1      4      3      3      2      3      6   : 35824
aida11 :      2      0      1      2      2      2      2      4      2      3      6   : 35800
aida12 :     15      6      2      0      2      4      2      2      3      3      6   : 36108



20:48 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[...]_*

21:00 Stats OK, UCESB OK, No faults

21:07 Go4 spectra erased by remote shifters.

21:15 Local shifters report correlations lost. Beam may be intermittently removed for checks.

21:30 Still no beam. DAQ issues locally. AIDA MBS link had to be restarted due to operator mistake.




21:37 Beam back. Stats OK, UCESB OK, No faults found.

System wide checks:
Clock - aida09 fails

All modules now fail ADC calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd061 : 	 15  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x36 : 	 54  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      3      1      2      4      2      4      2      3      6   : 36104
aida02 :      2      3      0      2      2      4      2      2      3      3      6   : 36064
aida03 :      6      3      3      3      2      2      2      2      4      3      6   : 36928
aida04 :     15      5      8      4      3      3      2      3      3      3      6   : 36772
aida05 :     10      5      8      4      4      4      3      2      3      3      6   : 36688
aida06 :     14     17     10      3      3      3      1      4      3      3      6   : 37120
aida07 :      3      4      1      1      1      2      2      2      3      3      6   : 35740
aida08 :     12      5      7      2      2      4      1      2      3      3      6   : 35976
aida09 :      0      6      2      0      2      2      2      2      2      4      6   : 36816
aida10 :      4      2      5      2      1      4      2      4      2      3      6   : 36080
aida11 :     14      3      2      3      1      2      2      4      2      3      6   : 35856
aida12 :      1      2      0      2      1      4      1      3      3      3      6   : 36244

22:05 Stats OK (attach 5)
Temps OK (attach 6)
Bias OK (attach 7)

System wide checks as before - see above.

22:13 ADC calibration - Only 06,07 and 09 now fail
(TD re-calibrated remotely)

22:53 Beam off to allow access to S4 to reboot VME crate (akash).


22:58 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[1-2][0-9]_* R[3][0-2]_*
aka everything *except* R33_*


23:11: Beam is back. Stats and UCESB as before.

23:24: Stats OK (attach 8)
Temps OK (attach 9)
Bias OK (attach 10)
UCESB OK (attach 11)
System-wide checks
Clock: 06 fails
Calibration: 06,07,09 fail

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      2      2      3      4      3      3      2      3      6   : 35996
aida02 :      8     12      4      1      2      4      3      2      3      3      6   : 36448
aida03 :      2      4      2      4      3      3      1      3      3      3      6   : 36360
aida04 :      7      8      4      4      2      4      3      3      3      3      6   : 37020
aida05 :      3      5      7      4      3      3      3      3      3      3      6   : 36964
aida06 :      2      6      8      2      3      4      2      3      3      3      6   : 36792
aida07 :      8      4      5      2      0      3      1      2      3      4      6   : 37712
aida08 :     26     11      3      1      1      3      1      3      3      3      6   : 36304
aida09 :     22      4      2      2      3      4      1      3      3      3      6   : 36504
aida10 :     21      6      6      2      0      4      2      3      3      3      6   : 36644
aida11 :     22      7      2      3      2      1      2      4      2      3      6   : 35856
aida12 :     16      9      2      2      3      3      2      2      3      3      6   : 36136
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Ucesb1.png
Ucesb1.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temp2.png
Temp2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: Temps3.png
Temps3.png
Attachment 10: Bias3.png
Bias3.png
Attachment 11: Ucesb2.png
Ucesb2.png
  252   Tue Apr 20 10:54:41 2021 CBApril 20 12:00 - 16:00 shift
11:55 CA got AIDA back online

12:00 Stats OK (attach 1)
No faults found
UCESB OK - no beam atm (attach 2)
Temps OK (running hotter than yesterday, but not worrying yet) - attach 3
Bias OK (attach 4)

System wide checks
Clock - *all* pass
ADC - 06 fails
WR - *all* pass
FPGA *all* pass

Memory info
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      4      1      3      2      1      2      4      2      3      7   : 39876
aida02 :     23      6      1      2      2      2      1      4      2      3      7   : 39772
aida03 :     24      4      4      1      2      3      1      4      2      3      7   : 39904
aida04 :     10      3      1      1      1      3      1      4      2      3      7   : 39728
aida05 :     12      2      3      2      1      4      2      3      1      4      7   : 40688
aida06 :     10      5      4      2      2      3      3      4      2      3      7   : 40400
aida07 :     22      6      1      3      1      3      1      4      2      3      7   : 39864
aida08 :     20      4      4      1      2      3      2      4      2      3      7   : 40144
aida09 :      1      2      2      3      1      2      1      3      3      3      7   : 40148
aida10 :      3      3      1      2      2      2      2      3      2      3      7   : 39412
aida11 :     25      7      0      1      0      3      2      3      2      3      7   : 39484
aida12 :     21      7      3      1      3      2      2      3      2      3      7   : 39580


Waiting for beam.

12:42 Stats OK, UCESB ok, No faults. No beam.
13:21 Stats OK, UCESB ok, No faults. No beam.
13:24 Implants in DSSD#3. ASIC check. Disappeared. No beam.

14:00 No beam.
Stats OK (attach 5)
Temps OK (attach 6)
UCESB OK
Bias  OK (attach 7)

System wide checks
Clock OK
ADC calibration 06 fails
WR OK
FPGA OK

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      1      3      2      3      3      2      3      7   : 39848
aida02 :     15      8      2      2      2      3      2      3      2      3      7   : 39644
aida03 :      8      1      2      3      4      3      2      3      2      3      7   : 39720
aida04 :     25     12      3      1      1      3      1      4      2      3      7   : 39892
aida05 :     13      4      4      4      0      3      2      3      1      4      7   : 40596
aida06 :     21      8      3      3      0      3      3      4      2      3      7   : 40356
aida07 :     17     11      0      1      1      3      1      4      2      3      7   : 39804
aida08 :     16      8      2      3      2      3      1      4      2      3      7   : 39936
aida09 :     17      6      3      2      1      2      1      4      2      3      7   : 39716
aida10 :      3      3      2      2      4      1      2      2      3      3      7   : 39940
aida11 :      4      4      4      1      1      3      3      2      2      3      7   : 39248
aida12 :      2      4      1      3      1      2      2      3      2      3      7   : 39384

14:30 Stats OK, UCESB ok, No faults. No beam.
15:00 Stats OK, UCESB ok, No faults. No beam.


15:24 Beam is back!
Stats OK (attach 8), UCESB ok (attach 9), No faults.

15:50 Shift handed over to Liam
Attachment 1: Stats1.png
Stats1.png
Attachment 2: ucesb_nobeam.png
ucesb_nobeam.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temps2.png
Temps2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: UCESB2.png
UCESB2.png
  324   Tue May 18 14:57:43 2021 CBTuesday 18th May 16:00 - 20:00
15:58 Took shift over from OH and PP

Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x510 : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     22      8      7      4      1      4      2      2      2      2      7   : 37320
aida02 :     12      9      4      2      4      2      3      3      3      3      6   : 36856
aida03 :     19      6      7      2      2      2      2      4      3      3      6   : 37036
aida04 :     18      9     10      6      1      4      1      4      3      3      6   : 37168
aida05 :     17     12      2      1      2      3      3      3      3      3      6   : 36836
aida06 :     19      8      7      7      2      4      2      4      3      3      6   : 37468
aida07 :     28      5      4      1      2      4      2      3      2      2      7   : 37752
aida08 :     17      5      4      2      1      2      2      3      3      3      6   : 36396
aida09 :     21     10      4      5      4      4      3      3      3      3      6   : 37252
aida10 :     15     13      4      2      2      2      3      2      2      4      6   : 37284
aida11 :     22     13      3      3      4      3      1      4      3      3      6   : 37072
aida12 :      2      8      1      3      1      3      2      3      3      3      6   : 36472
aida13 :     23      5      4      4      1      4      3      2      3      3      6   : 36484
aida14 :     19      7      4      3      1      3      2      4      2      4      6   : 38116
aida15 :     20     10      2      2      3      2      3      2      3      3      6   : 36288
aida16 :     29      6      1      2      0      3      1      4      3      3      6   : 36724

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

Analysed R14_342. Attach 4. Dead time <5%



17:00

Stats OK - attach 5
Temps OK - as before
Bias  increasing quite quickly - attach 6

System-wide checks

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x512 : 	 18  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     28      7      8      4      1      4      2      2      2      2      7   : 37352
aida02 :     24     11      6      4      3      2      3      2      3      3      6   : 36440
aida03 :     17     11      5      2      1      2      2      3      3      3      6   : 36460
aida04 :     25     13     10      6      2      4      2      3      3      3      6   : 37036
aida05 :     23      7      3      2      2      3      3      3      3      3      6   : 36868
aida06 :      1     12      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     22     12      1      1      2      4      2      2      2      2      7   : 37224
aida08 :     21      8      7      2      1      3      3      2      3      3      6   : 36356
aida09 :     23      7      4      6      4      3      2      4      3      3      6   : 37396
aida10 :     27      7      2      2      2      2      3      3      2      4      6   : 37764
aida11 :     12      5      3      5      4      3      1      4      3      3      6   : 37032
aida12 :     23      9      4      1      2      3      2      3      3      3      6   : 36612
aida13 :     10     12      6      3      1      4      3      2      3      3      6   : 36488
aida14 :     23      7      3      3      1      3      2      3      2      4      6   : 37604
aida15 :     22      6      6      4      2      2      3      2      3      3      6   : 36328
aida16 :     27      7      3      1      0      3      1      4      3      3      6   : 36724



18:00

Stats OK - attach 7
Temps OK - as before
Bias - attach 8

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x517 : 	 23  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_371. Attach 9. Dead time below 5%.



18:35 MBS DAQ crashed



18:40
Stats OK - attach 9
Temps OK - as before
Bias - attach 10

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51a : 	 26  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:42 MBS DAQ restarted - new file being opened.


19:36
Stats OK - attach 11
Temps OK - attach 12
Bias - attach 13

System-wide checks
Clock OK
ADC all fail

	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51e : 	 30  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA OK

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     24     11      5      4      1      4      2      3      2      2      7   : 37832
aida02 :      0      5      6      3      4      2      3      2      3      3      6   : 36328
aida03 :     15      8      5      3      2      2      1      3      4      3      6   : 37292
aida04 :     14      8      7      5      3      4      2      3      3      3      6   : 36936
aida05 :     31      6      1      1      2      3      3      3      2      4      6   : 37852
aida06 :      9      8      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     21      7      2      1      2      4      1      3      2      2      7   : 37452
aida08 :      8      4      5      3      1      3      2      3      3      3      6   : 36528
aida09 :     24      7      3      5      4      3      2      4      3      3      6   : 37352
aida10 :     19      9      7      1      2      2      3      2      2      4      6   : 37284
aida11 :     27      5      0      5      4      3      1      4      3      3      6   : 37044
aida12 :     31      9      2      1      2      3      2      3      3      3      6   : 36612
aida13 :     21     10      4      3      1      4      3      2      3      3      6   : 36484
aida14 :     22      6      4      2      1      3      2      4      2      4      6   : 38088
aida15 :     21      9      3      4      2      2      3      2      3      3      6   : 36300
aida16 :     23     11      4      2      1      4      2      3      3      3      6   : 36724


19:43 Shift handed over to LS
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Temps1.png
Temps1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Analysis_R14_342
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252065796 ( 1015168.3 Hz)
        Other data format:    9854206 (   39686.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        242 (       1.0 Hz)
                               RESUME:        241 (       1.0 Hz)
                              SYNC100:      32924 (     132.6 Hz)
                              WR48-63:      32924 (     132.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    9787875 (   39419.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     499592 (    2012.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      248.300 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.028               16.674
  1                4.476                0.000
  2                0.095                0.000
  3                4.784                0.000
  4                0.046               31.798
  5                8.905                0.000
  6                0.093                3.062
  7                8.121                0.000
  8                0.937                0.000
  9                0.000               59.989
 10                0.965                0.000
 11                0.025                0.000
 12                0.042                0.000
 13                0.000               63.078
 14                0.061                0.000
 15                0.217                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5702434       2900          0          0          3          3        703        703          0       1488          0      26833
  1   25200522       6341          0          0         31         30       3140       3140          0          0          0      55809
  2   10737118    2562866          0          0          3          3       1687       1687          0    2559486          0      44132
  3   32448674    2519917          0          0         37         37       4558       4558          0    2510727          0      63444
  4    3679860    1335111          0          0          5          5        577        577          0    1333947          0      11808
  5   37244520     759044          0          0         62         62       4940       4940          0     749040          0      34314
  6    9560033    1568630          0          0          3          3       1348       1348          0    1565928          0      23903
  7   40864979    1077667          0          0         66         66       5138       5138          0    1067259          0      40193
  8   17664728       4388          0          0          6          6       2188       2188          0          0          0      56741
  9    2360528        644          0          0          0          0        322        322          0          0          0       4285
 10   20213561       5020          0          0         12         12       2498       2498          0          0          0      34696
 11   11474187       2786          0          0          1          1       1392       1392          0          0          0      10298
 12    8797735       2188          0          0          1          1       1093       1093          0          0          0      49042
 13    2441760        604          0          0          0          0        302        302          0          0          0       3463
 14    7875814       1932          0          0          4          4        962        962          0          0          0      33004
 15   15799343       4168          0          0          8          8       2076       2076          0          0          0       7627
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.352s (  931.544 blocks/s,  58.222 Mb/s)
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Bias2.png
Bias2.png
Attachment 7: Stats3.png
Stats3.png
Attachment 8: Bias3.png
Bias3.png
Attachment 9: Analysis_R14_371
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252182561 ( 1038396.9 Hz)
        Other data format:    9737439 (   40095.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        234 (       1.0 Hz)
                               RESUME:        234 (       1.0 Hz)
                              SYNC100:      32904 (     135.5 Hz)
                              WR48-63:      32904 (     135.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    9671163 (   39822.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     493534 (    2032.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      242.858 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.060               15.876
  1                5.312                0.000
  2                0.091                0.000
  3                3.590                0.000
  4                0.163               29.634
  5                9.407                0.000
  6                0.049                0.000
  7                6.820                0.000
  8                2.387                0.000
  9                0.000               29.724
 10                0.676                0.000
 11                0.043                0.000
 12                0.046                0.000
 13                0.000               57.696
 14                0.016                3.102
 15                0.214                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5708701       2832          0          0          4          4        686        686          0       1452          0      26548
  1   24896226       6554          0          0         36         36       3241       3241          0          0          0      54251
  2   10734245    2512200          0          0          3          3       1683       1683          0    2508828          0      43373
  3   32202526    2481239          0          0         32         32       4391       4391          0    2472393          0      63125
  4    3649395    1322612          0          0         11         11        611        611          0    1321368          0      11610
  5   36874871     743171          0          0         64         64       4722       4722          0     733599          0      33565
  6    9551266    1557362          0          0          1          1       1434       1434          0    1554492          0      23558
  7   41439213    1089893          0          0         57         57       5374       5374          0    1079031          0      40478
  8   17460031       4238          0          0          7          7       2112       2112          0          0          0      55845
  9    2360192        582          0          0          0          0        291        291          0          0          0       4290
 10   20593873       5182          0          0          6          6       2585       2585          0          0          0      34440
 11   11697470       2942          0          0          2          2       1469       1469          0          0          0      10244
 12    8815559       2192          0          0          1          1       1095       1095          0          0          0      48502
 13    2454200        586          0          0          0          0        293        293          0          0          0       3481
 14    7942567       1924          0          0          2          2        960        960          0          0          0      32639
 15   15802226       3930          0          0          8          8       1957       1957          0          0          0       7585
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.879s (  944.541 blocks/s,  59.034 Mb/s)
Attachment 10: Stats4.png
Stats4.png
Attachment 11: Bias4.png
Bias4.png
  332   Wed May 19 14:54:36 2021 CBWednesday 19 May 16:00 - 20:00
15:55 Took over from OH

16:00
Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System wide checks
All OK, except ADC all fail (as before) and WR

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_691. Attach 4. Dead time <5%

16:34 MBS changes file.


17:00
Stats OK - attach 5
Temps OK - as before
System wide checks as before
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


17:55
Stats OK - attach 6
Temps OK - as before

Leakage current approaching 10 uA - attach 7

System-wide checks - as before except
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x568 : 	 104  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



18:01 MBS switches to new file



18:52
Stats OK - attach 8
Temps OK - as before

System-wide checks as before

Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9

19:36 Stats OK - attach 10
Bias - attach 11

19:48 Shift handed over to LS
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Temps1.png
Temps1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Analysis_R14_691
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251761346 (  995725.1 Hz)
        Other data format:   10158654 (   40177.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        199 (       0.8 Hz)
                               RESUME:        199 (       0.8 Hz)
                              SYNC100:      32942 (     130.3 Hz)
                              WR48-63:      32942 (     130.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10092372 (   39915.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     503210 (    1990.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      252.842 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.053               17.526
  1                1.803                0.000
  2                0.604                2.918
  3                3.773                0.000
  4                0.070               17.512
  5                9.601                0.000
  6                0.278                0.000
  7                9.149                0.000
  8                0.114                0.000
  9                0.000               64.964
 10                0.536                0.000
 11                0.070                0.000
 12                0.024                6.415
 13                0.000               66.391
 14                0.036                7.395
 15                0.097                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6023981       3019          0          0          3          3        749        749          0       1515          0      26768
  1   22620294       5924          0          0         14         14       2948       2948          0          0          0      55394
  2   11082857    2607083          0          0          5          5       1799       1799          0    2603475          0      43681
  3   30564809    2580290          0          0         26         26       4295       4295          0    2571648          0      65104
  4    3958862    1393424          0          0          7          7        678        678          0    1392054          0      11940
  5   37388332     780379          0          0         62         62       4752       4752          0     770751          0      33878
  6   10191568    1631763          0          0          8          8       1480       1480          0    1628787          0      23955
  7   39960208    1134610          0          0         55         55       5179       5179          0    1124142          0      42303
  8   17144158       4230          0          0          3          3       2112       2112          0          0          0      56568
  9    2458065        662          0          0          0          0        331        331          0          0          0       4321
 10   20331742       4810          0          0          6          6       2399       2399          0          0          0      34970
 11   11809904       2968          0          0          4          4       1480       1480          0          0          0      10406
 12    9134013       2250          0          0          1          1       1124       1124          0          0          0      49240
 13    2508032        620          0          0          0          0        310        310          0          0          0       3587
 14    8541585       2182          0          0          2          2       1089       1089          0          0          0      33394
 15   18042936       4440          0          0          3          3       2217       2217          0          0          0       7701
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.672s (  950.348 blocks/s,  59.397 Mb/s)
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Stats3.png
Stats3.png
Attachment 7: Bias3.png
Bias3.png
Attachment 8: Analysis_R14_741
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253670855 ( 1280798.3 Hz)
        Other data format:    8249145 (   41650.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        236 (       1.2 Hz)
                               RESUME:        236 (       1.2 Hz)
                              SYNC100:      32737 (     165.3 Hz)
                              WR48-63:      32737 (     165.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8183199 (   41317.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     413031 (    2085.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      198.057 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.011                2.771
  1                4.649                0.000
  2                0.028                0.000
  3                4.808                0.000
  4                0.111                7.488
  5                7.093                0.000
  6                0.128                0.000
  7               10.340                0.000
  8                0.971                0.000
  9                0.000               24.260
 10                0.713                0.000
 11                0.245                0.000
 12                0.122                0.000
 13                0.000               21.524
 14                0.232                0.000
 15                1.605                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5709345       2725          0          0          1          1        769        769          0       1185          0      22625
  1   23799538       6034          0          0         27         27       2990       2990          0          0          0      44162
  2   11722022    2092139          0          0          1          1       1767       1767          0    2088603          0      36240
  3   29654318    2044332          0          0         38         38       3940       3940          0    2036376          0      53125
  4    3438629    1158241          0          0          8          8        579        579          0    1157067          0       9655
  5   33876189     636041          0          0         60         60       4396       4396          0     627129          0      27488
  6   10663472    1364583          0          0          5          5       1426       1426          0    1361721          0      20341
  7   38937063     921272          0          0         54         54       5023       5023          0     911118          0      34159
  8   16044621       3966          0          0          9          9       1974       1974          0          0          0      46740
  9    2236927        610          0          0          0          0        305        305          0          0          0       3583
 10   23087850       5792          0          0          6          6       2890       2890          0          0          0      28946
 11   13996775       3362          0          0          8          8       1673       1673          0          0          0       8557
 12    9568717       2316          0          0          4          4       1154       1154          0          0          0      40620
 13    2488330        626          0          0          0          0        313        313          0          0          0       2902
 14    9604847       2392          0          0          6          6       1190       1190          0          0          0      27522
 15   18842212       4714          0          0          9          9       2348       2348          0          0          0       6366
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.094s (  938.588 blocks/s,  58.662 Mb/s)
Attachment 9: Stats4.png
Stats4.png
Attachment 10: Stats6.png
Stats6.png
Attachment 11: Bias6.png
Bias6.png
  443   Fri May 13 15:00:23 2022 CB13 March - 16:00-24:00 shift
16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before
Attachment 1: 1305_1600_ucesb.png
1305_1600_ucesb.png
Attachment 2: 1305_1600_StatsTemps.png
1305_1600_StatsTemps.png
Attachment 3: 1305_1600_Grafana.png
1305_1600_Grafana.png
Attachment 4: 1305_1800_StatsLeakage.png
1305_1800_StatsLeakage.png
Attachment 5: 1305_2030_Stats.png
1305_2030_Stats.png
Attachment 6: 1305_2030_ucesb.png
1305_2030_ucesb.png
Attachment 7: 1305_2030_Grafana.png
1305_2030_Grafana.png
Attachment 8: 1305_2300_StatsLeakage.png
1305_2300_StatsLeakage.png
Attachment 9: 1305_2300_ucesb.png
1305_2300_ucesb.png
Attachment 10: 1305_2300_Grafana.png
1305_2300_Grafana.png
  142   Wed Mar 11 17:21:33 2020 CA, TD, OHWednesday 11 March 16.00-00.00
18.25 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - OK

WR timestamp control - attachments 4 & 5

merger & TapeServer OK - attachments 6-8

rate spectra - attachment 9

1.8.L spectra - attachments 10 & 11
  aida03 72 ch FWHM

20:36 DSSSD biases & leakage currents OK - attachment 12
      FEE64 temps OK - attachment 13
      good events statistics OK - attachment 14

      all system wide checks ok

21.50 beam starts back up
      DAQ running, TapeServer now writing to file R7_0
      data rate ~ 25 Mb/s
      data forwarding to MBS ok
      merger ok, data item rate ~ 2.7 million items/sec
      spill time length is now 3 seconds
      1500 count rate per spill, ~ 300 Hz implant rate in AIDA

22.35 FEE64 temps OK - attachment 15
      good event statistics ok - attachment 16
      detector bias/ leakage currents ok - attachment 17

      all system wide checks ok

22.47 attachments 18 & 19 - high energy spectra for all FEE64 (1.8.H)

      attachment 20 - hit rate spectra for all FEE64

23.02 beam down
      on file R7_55

23.11 beam back online
      on file R7_61

23.15 system wide checks all ok

      detector bias/ leakage currents ok - attachment 21

      good event statistics ok - attachment 22

      FEE64 temperatures ok - attachment 23      
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 10.png
10.png
Attachment 10: 11.png
11.png
Attachment 11: 12.png
12.png
Attachment 12: 200311_2035_bias.png
200311_2035_bias.png
Attachment 13: 200311_2034_temps.png
200311_2034_temps.png
Attachment 14: 200311_2034_stats.png
200311_2034_stats.png
Attachment 15: 200311_2233_temp.png
200311_2233_temp.png
Attachment 16: 200311_2234_stats.png
200311_2234_stats.png
Attachment 17: 200311_2235_bias.png
200311_2235_bias.png
Attachment 18: 200311_18h_odd.png
200311_18h_odd.png
Attachment 19: 200311_2245_18h_even.png
200311_2245_18h_even.png
Attachment 20: 200311_2242_rates.png
200311_2242_rates.png
Attachment 21: 200311_2314_bias.png
200311_2314_bias.png
Attachment 22: 200311_2314_stats.png
200311_2314_stats.png
Attachment 23: 201103_2313_temp.png
201103_2313_temp.png
  15   Wed Jan 23 08:12:01 2019 CA, TD, NH, VPJanuary 22nd 2019
Attachment 1 - 1.8.W waveform spectra with pulser signal

Attachment 2 - 1.8.L pulser peak spectra

widths: aida01 - 16.06
       
        aida04 - 20.68

        aida05 - 14.03

        aida08 - 73.02

        aida09 - 669.29

        aida11 - 405.35

        aida12 - 397.6

Attachment 3 - good event statistics
               fast comparator threshold (LEC) at 0xff
               FEEs 8,9,11,12 running hot

Attachment 4 - FEE64 temperatures - all ok

System wide checks ok, except aida02 fails ADC calibration

Leak found in seal on cooling pipe, switched power off
Attachment 1: 220119_18W.png
220119_18W.png
Attachment 2: 220119_pulserpeaks.png
220119_pulserpeaks.png
Attachment 3: 220119_stats.png
220119_stats.png
Attachment 4: 220119_Temp.png
220119_Temp.png
ELOG V3.1.4-unknown