03:09   General check

Rates, temptature, voltages are OK attached 1, 2, 3, 4

****Clock Ckeck******

OK
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

****** White Rabbit check******

     Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c2c :      48  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

*****FPGA check ******

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x10 :      6  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

*****Memory check*****

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6     12     11      7      2      3      2      4      2      3      6   : 36360
aida02 :     21     15      7      3      2      3      1      4      2      3      6   : 35996
aida03 :      9     10     10      2      4      3      3      3      2      3      6   : 36052
aida04 :     17     17     17      4      0      2      3      4      2      3      6   : 36444
aida05 :     25      7      3      1      1      1      2      3      2      4      6   : 37292
aida06 :     29     15      7      2      3      4      1      3      3      3      6   : 36700
aida07 :     17     17      6      5      2      3      2      3      3      3      6   : 36812
aida08 :     22      6     10      5      3      1      3      4      2      3      6   : 36360
aida09 :     18     28     14      4      2      3      3      3      2      3      6   : 36232
aida10 :     26     16      4      3      3      1      3      4      2      3      6   : 36296
aida11 :     16     10      4      0      3      3      1      4      2      3      6   : 35856
aida12 :     19     14      4      2      4      4      1      3      3      3      6   : 36668

04:15      General Check

Rates, Tempratures, Voltages are ok, attached 5,6,7,8

******Clock Check*******

OK

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

******ADC******

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

******White Rabbit check******

         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c39 :      61  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

******FPGA Check******

     Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x11 :      7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

****** Memorey check******

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     21     10      7      1      3      2      4      2      3      6   : 36436
aida02 :     28     21      3      3      1      3      1      4      2      3      6   : 35944
aida03 :     15      8     11      3      3      3      3      3      2      3      6   : 36044
aida04 :     25     19     14      4      1      3      4      3      2      3      6   : 36380
aida05 :     12      6      5      1      0      2      2      3      2      4      6   : 37328
aida06 :     29     21      4      2      3      4      1      3      3      3      6   : 36700
aida07 :     19     14      7      6      2      3      2      2      3      3      6   : 36332
aida08 :     11      8      8      6      3      1      3      4      2      3      6   : 36332
aida09 :     16     25     11      3      1      2      2      4      2      3      6   : 36184
aida10 :     14     13      4      2      3      2      2      4      2      3      6   : 36064
aida11 :      4      9      4      0      3      3      1      4      2      3      6   : 35800
aida12 :     38     17      4      2      4      4      1      3      3      3      6   : 36768

16:00 Take over from Magda mrning shift

Beam was off of sometime for tunning to Mercurey setting.

17:00 attached 1-3

19:00 attached 4-6

21:00 attached 7-9

23:30 10-12

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

   Shift file starting R9_639

   system wide checks okay
   FEE temperatures okay (attachment 1)
   leakage currents normal and recorded to spreadsheet (attachment 2)
   good event stats are normal (attachment 3)
   merger running at 3.5 M items/sec
   tape running at 26 MB/sec
   
   rates histogram (attachment 4)
   low energy histograms (attachment 5 and 6)
   high energy histograms (attachment 7 and 8)

02.02 
   system wide checks okay
   FEE temperatures okay (attachment 9)
   leakage currents normal and recorded to spreadsheet (attachment 10)
   good event stats are normal (attachment 11)
   merger running at 3.5 M items/sec
   tape running at 27 MB/sec

04.09 
   system wide checks okay
   FEE temperatures okay (attachment 12)
   leakage currents normal and recorded to spreadsheet (attachment 13)
   good event stats are normal (attachment 14)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
   writing to file R9_840

06.07 
   system wide checks okay
   FEE temperatures okay (attachment 15)
   leakage currents normal and recorded to spreadsheet (attachment 16)
   good event stats are normal (attachment 17)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
 
07.59 
   system wide checks okay
   FEE temperatures okay (attachment 18)
   leakage currents normal and recorded to spreadsheet (attachment 19)
   good event stats are normal (attachment 20)
   merger running at 3 M items/sec
   tape running at 26 MB/sec

as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space

14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?

15:06 Systems checks fine.
      Master clock failed (no master clock)
      all checks passed
      memory checks all around 38k as normal

      Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees

      Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k

      Attachment 4: leakage currents seem normal at expected values

15:16 Merger working for all FEEs
      tape server is on but no storage
      data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)

ASIC controls checked and are all OK

15.48 low energy pulsar peak FWHM

      fee   FHWM
      1     89.08
      2     132.06
      3     70.73
      4     81.84
      5     62.86
      6     79.92
      7     122.13
      8     73.31
      9     198.39
      10    115.53
      11    100.55
      12    144.04

ALL FEEs in downstream SSD are high

16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)

16:25 included rates, note not to common scale in y (attachment 9)

ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
  Did not seem completely reproducible?
  We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad

Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)


19:15 Update

There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.  
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems

One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space 
  This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)

Now we have brought back up the AIDA systems and should run through the checks

Most system checks look good (except some calibration errors)

Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool

Bias and leak currents attached as #10

Temps as #11

Stats as #12

19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.

21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
      A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
      Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET

21:51 AIDA running to no storage again

11.00(Germany) System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment1)
      Spectra rates (attachment2)
      FEE temps (attachment3)
      Leakage currents, written to google sheet (attachment4)  
      Merger~ 4.9M items/s
      Tapeserver ~17MB/s 

      In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
      AIDA has been taken out of the timesorter due to the high data rate, buffers were full
      AIDA cannot be seen in ucesb or Go4.

13.00 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment5)
      Spectra rates (attachment6)
      FEE temps (attachment7)
      Leakage currents, written to google sheet (attachment8)  
      Merger~ 5.1M items/s
      Tapeserver ~18MB/s 

      No timestamp related errors this shift

13.37 AIDA MBS control restarted R33_388
      System wide checks same as previous time

13.54 beam stopped for access,  file R33_396

13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
      Analysed R33_395, 396, 397, 398 no timewarps
      
14.29 beam back R33_415

14.54 see more bad timestamps in new merger terminal (attachment10)

14.55 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment11)
      Spectra rates (attachment12)
      FEE temps (attachment13)
      Leakage currents, written to google sheet (attachment14)  
      Merger~ 5.0M items/s
      Tapeserver ~17MB/s 

16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463

16.15 CA takes over until 18:00

17:11 System wide checks:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0xcdd6 : 	 0xcdd7 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

17:13 FEE64 temps ok - attachment 15
      Statistics ok - attachment 16
      bias and leakage currents ok - attachment 17

20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s

21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)

21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)

22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4e7 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.4M items/s
Tape server 6.3MB/s

Ran R14_119 through analyser, dead times all good (Fig 11)

22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)

23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)

23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)

24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4eb : 	 17  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.5M items/s
Tape server 6.5MB/s

Ran R14_142 through analyser, dead times all good (Fig 21)

19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x520 : 	 32  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~1.8M items/s
Tape server ~7MB/s

Analysed R14_399, deadtimes are okay (Fig 4)

20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)

21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)

21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)

22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x524 : 	 36  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.3M items/s
Tape server ~8MB/s

Analysed R14_430, deadtimes are okay (Fig 14)

22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%

23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)

23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)

24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x529 : 	 41  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

The difference has increased over time (not sure if an issue)

Merger rate ~3M items/s
Tape server ~8MB/s

Analysed R14_462, deadtimes are okay (Fig 24)

20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)

20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)

21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.8M items/s
Tape server ~10MB/s

Analysed R14_781 deadtimes are okay (Fig 9)

21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)

22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)

22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)

23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~3M items/s
Tape server ~10MB/s

Analysed R14_819 deadtimes are okay (Fig 19)

23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21)

16:30

All System checks okay, except for:

FPGA Timestamp error:

   
             Base         Current         Difference
aida12 fault      0x0 :      0x1b :      27  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Temperature Checks: See attachment 1

Statistics Checks: See attachment 2

Bias & Leakage Currents: See attachment 3

18:09

All system checks okay, except for:

FPGA Timestamp error:

   
             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Temperature checks: See attachment 4

Statistics Checks: See attachment 5

Bias & Leakage Currents: See attachment 6

19:40

All system checks okay, except for:

FPGA Timestamp error:

             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Temperature checks: See attachment 7

Statistics Checks: See attachment 8

Bias & Leakage Currents: See attachment 9

*Beam being tuned to another experiment (R3B) - all DSSD implants fluctuating back and forth to 0 Hz for around the last hour or so*

22:30

General check

Rates, Voltages, Temperatures, Ucesb are attached 10,11,12,13

Its been noticed that aida04 temp. is incresing to about 65 (red), still below 70.

the system wide check are all ok except FPGA

             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

System Checks @ ~18:10 :


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd05a : 	 8  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x23 : 	 35  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      6      4      0      1      5      2      3      2      3      6   : 35636
aida02 :     27      3      2      0      2      3      2      3      3      3      6   : 36516
aida03 :     33      9      2      1      1      3      2      3      3      3      6   : 36556
aida04 :     11      6      5      4      2      4      3      3      3      3      6   : 37036
aida05 :     15      8      7      4      3      4      4      2      3      3      6   : 36908
aida06 :     13      9      7      3      3      4      2      3      3      3      6   : 36876
aida07 :      1      3      3      2      2      3      1      3      3      3      6   : 36236
aida08 :     29      5      2      1      1      2      2      3      3      3      6   : 36380
aida09 :      3      8      2      0      2      3      2      2      2      4      6   : 36972
aida10 :      9      5      3      3      1      4      2      4      2      3      6   : 36124
aida11 :      5      7      3      2      2      2      2      4      2      3      6   : 35900
aida12 :     24      6      3      1      2      3      2      2      3      3      6   : 36064


System Checks @ ~19:55

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05e : 	 12  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x27 : 	 39  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18     10      2      2      1      5      2      3      2      3      6   : 35768
aida02 :     21      9      2      1      2      2      2      3      3      3      6   : 36444
aida03 :     13      5      6      1      2      2      2      3      3      3      6   : 36444
aida04 :     30     11      4      3      2      4      3      3      3      3      6   : 37104
aida05 :     10      9      8      4      4      4      4      2      3      3      6   : 36976
aida06 :     22     17     10      3      2      3      1      4      3      3      6   : 37088
aida07 :     20      8      4      1      2      4      1      2      3      3      6   : 35952
aida08 :     14      4      3      0      1      2      2      3      3      3      6   : 36296
aida09 :     24      8      4      0      1      3      3      3      3      3      6   : 36768
aida10 :     19      8      4      1      1      4      2      3      3      3      6   : 36652
aida11 :      6      6      1      2      3      1      2      4      2      3      6   : 35800
aida12 :     15      7      1      1      2      4      1      2      3      3      6   : 35876

08:30

System Checks

Clock Status error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 1

Temperatures - See attachment 2

Bias & Leakage Currents - See attachment 3

10:02

System Checks:

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration Error:

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 4

Temperatures - See attachment 5

Bias & Leakage Currents - See attachment 6

10:45

System Checks

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration check Error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 7

Temperatures - See attachment 8

Bias & Leakage Currents - See attachment 9

16:25

Informed at shift handover that there has been no beam.

Statistics  - See attachment 1

Temperatures - See attachment 2

Bias & Leakage currents - See attachment 3

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit decoder status:

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4dd :      3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:00

Usecb scalars now showing DSSD implant rates.

Statistics  - See attachment 4

Temperatures - See attachment 5

Bias & Leakage currents - See attachment 6

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx

19:45

Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.

Statistics  - See attachment 7

Temperatures - See attachment 8

Bias & Leakage currents - See attachment 9

System Wide checks - All passed, EXCEPT:

ADC Calibration:

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit decoder status:

     Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes

19:55 - Beam now back again - DSSD implants showing

		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bce : 	 1  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

13:55 CEST

Statistics : ok elog:234/4
Temp : ok  elog:234/5
Bias : ok  elog:234/6
ucesb : ok
DB: No faults found

ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and 
Control browser page to rerun calibration for that module

White Rabbit Check:
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

FPGA check:
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

14:05 no beam
14:10 beam back
14:15 no beam, beam current being optimised, going to thicker degrader when beam returns


14:37 CEST

Statistics : ok
Temp : ok
Bias : ok   (ch3 now over 6uA)
ucesb : ok
DB: No faults found

15:03 CEST

Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found

15:31 CEST

Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found

15:50

Statistics : ok elog:234/7
Temp : ok  elog:234/8
Bias : ok  elog:234/9
ucesb : ok elog:234/10
DB: No faults found

ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Check:
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check:
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

20:00  

Taking over from Liam, all systems good.

20:30  

DB No faults.
Stats good
ucesb ok

20:58 

DB No faults.
Stats good
Temps good
ucesb ok

21:31 

DB No faults.
Stats good
Temps good
ucesb ok

22:02

DB No faults.
Stats good elog:254/3
Temps good elog:254/4
ucesb ok elog:254/1
Current ok elog:254/2

ADC Calibration check:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1

22:06 ucesb - DAQ Error - Onsite team checking daq crash
Tape server icon stopped spinning
Stats Rates 0, Counter aida01 zeroed but others normal. elog:254/5

Called Tom. Tried resetting in midas, no luck. Full reboot of DAQ and midas required.
According to log at 22:07 aida01 reset
Data taking stopped at 22:04 file R45_193

22:50
After a full power cycle all appear to be up and running well. Starting full checks.

23:00

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

Check FPGA is having a page error

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	
4k
aida01 :      4      8      2      2      1      3      1      4      2      3      7   : 
39792
aida02 :     13      9      2      4      2      3      2      3      2      3      7   : 
39708
aida03 :     19      5      4      3      1      3      1      4      2      3      7   : 
39892
aida04 :     14      5      2      2      0      5      2      3      2      3      7   : 
39744
aida05 :     16     11      3      2      2      4      3      2      1      4      7   : 
40584
aida06 :     15      9      5      2      2      2      1      4      2      3      7   : 
39828
aida07 :     15      9      3      3      2      2      1      4      2      3      7   : 
39828
aida08 :     21      4      3      4      2      2      1      4      2      3      7   : 
39844
aida09 :     11      6      1      2      3      4      2      3      2      3      7   : 
39788
aida10 :      0      6      4      2      0      3      2      3      2      3      7   : 
39472
aida11 :     16      7      2      0      1      3      2      3      2      3      7   : 
39512
aida12 :      3      7      4      1      3      2      2      3      2      3      7   : 
39524

DB No faults.
Stats good elog:254/6
Temps good elog:254/7 aida01 had been high during reboot issue, >70, but now is back down
Current ok elog:254/8
ucesb ok

23:25 

DB No faults.
Stats good
Temps good
ucesb ok

23:42 No Beam
23:48 Beam Back

23:57

DB No faults.
Stats good elog:254/11
Temps good elog:254/12
ucesb ok elog:254/9
Current ok elog:254/10

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
	
*FPGA check page error*

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      9      5      2      1      2      1      4      2      3      7   : 39812
aida02 :     13      9      5      2      2      2      1      4      2      3      7   : 39820
aida03 :     21      8      3      2      2      2      1      4      2      3      7   : 39812
aida04 :     13      4      3      1      1      5      2      3      2      3      7   : 39780
aida05 :     24      5      3      2      2      4      2      3      2      3      7   : 39800
aida06 :     16      5      5      2      2      2      1      4      2      3      7   : 39800
aida07 :     22      6      2      2      1      3      1      4      2      3      7   : 39848
aida08 :     21      8      2      3      2      2      1      4      2      3      7   : 39828
aida09 :     18      7      1      1      3      4      2      3      2      3      7   : 39792
aida10 :     19      3      3      1      0      3      2      3      2      3      7   : 39476
aida11 :     17      4      4      0      1      3      2      3      2      3      7   : 39524
aida12 :     13      4      3      2      3      1      2      3      2      3      7   : 39428

00:04 Handing over to Corrigan

12:00 Taking over from Liam, all is well.

12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz

12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they 
need to check and will need apprx. 15 min.")

12:30
ucesb ok (no beam)
DB No faults
Stats ok.

12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.

13:00
We are taking intermittent beam while they work on the FRS

DB No faults
ucesb ok  elog:261/1
Current ok elog:261/2
Stats ok  elog:261/3
Temps ok elog:261/4

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x46 : 	 70  
FPGA Timestamp error counter test result: Passed 11, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      6      5      3      2      2      2      3      2      3      7   : 39580
aida02 :     21      7      4      2      2      3      2      3      2      3      7   : 39692
aida03 :     20     10      3      3      0      3      2      3      2      3      7   : 39600
aida04 :      8      7      4      5      2      4      3      2      2      3      7   : 39608
aida05 :     17     14      1      2      3      3      4      2      2      3      7   : 39748
aida06 :     17     10      2      3      2      2      4      2      2      3      7   : 39572
aida07 :     27     10      8      7      2      3      3      2      2      3      7   : 39708
aida08 :      9      6      2      3      2      1      3      3      2      3      7   : 39636
aida09 :      1      2      1      4      0      3      3      2      2      3      7   : 39204
aida10 :      5      8      4      4      2      2      1      3      2      3      7   : 39316
aida11 :      6      4      5      3      2      3      2      2      2      3      7   : 39144
aida12 :     15      5      8      1      0      4      1      3      2      3      7   : 39428


13:32
ucesb ok
DB No faults
Stats ok

14:01
ucesb ok
DB No faults
Stats ok

14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
      cat /var/log/messages | grep /MIDAS - attachments 6 & 7
      Grafana DSSSD bias & leakage currents for last 7 days - attachment 8

14:55
DB No faults 
ucesb ok
Current ok elog:261/9
Stats ok  elog:261/10
Temps ok elog:261/11

FPGA Timestamp Check Error:
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 11, Failed 1


15:30
ucesb ok
DB No faults
Stats ok

15:59
ucesb ok elog:261/12
DB No faults
Stats ok

19:51 Shift change

Stats ok
ucesb ok, Max implants ~1500 Hz 

20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)

20:51 
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.

20:59 

Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36ce : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0xb6 : 	 172  
FPGA Timestamp error counter test result: Passed 15, Failed 1

--- both errors above same module as earlier shift
	 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      3      2      0      1      3      3      2      3      3      6   : 36092
aida02 :      6      4      4      2      2      3      1      3      3      3      6   : 36280
aida03 :      6      3      2      0      0      3      3      2      3      3      6   : 36048
aida04 :     10      3      1      1      3      3      1      3      3      3      6   : 36272
aida05 :     19      8      2      0      2      3      1      3      3      3      6   : 36268
aida06 :      2      2      0      2      3      4      2      2      3      3      6   : 36120
aida07 :      6      7      3      0      3      4      3      2      3      3      6   : 36416
aida08 :      7      5      4      0      3      2      2      3      3      3      6   : 36420
aida09 :     21      6      3      2      2      4      2      3      2      3      6   : 35700
aida10 :     13      3      1      1      3      3      1      4      2      3      6   : 35772
aida11 :     25      6      4      2      1      4      2      4      2      3      6   : 36180
aida12 :      0      4      0      3      2      3      3      3      2      3      6   : 35712
aida13 :     18      5      2      3      2      3      2      4      2      3      6   : 36080
aida14 :      2      1      1      1      2      3      2      4      2      3      6   : 35904
aida15 :      0      1      1      3      0      4      2      2      3      3      6   : 35960
aida16 :      4      5      3      2      3      4      1      3      2      3      6   : 35432

ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4

21:36
ucesb ok
Stats ok 

22:01
ucesb ok 1500 Hz peak implant
Stats ok 
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.

22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler

23:40 FRS magnet issue, possibly power, no beam

00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah