AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 26 of 36  ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
Entry  Sun Jun 26 23:04:30 2022, Marc, new shift - Monday 27 June 0:00 to 8:00 12x

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change

Entry  Thu Apr 25 15:02:50 2024, Marc, 16:00 - 0:00 Thursday 25 April 8x
16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

 

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

Entry  Fri Apr 26 23:04:51 2024, Marc, 00:00-08:00 Saturday 27 April 12x

It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!

0.15 Checks:
      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage current dropped a bit since the evening shift ~22:00.

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

      ucesb - attachment 7

2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12

3:30 Still running smoothly. 

5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.

        DAQ restarted OK. Main DAQ informed.

 

Entry  Sat Apr 27 23:04:42 2024, Marc, 00:00-08:00 Sunday 28 April 12x

Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C

0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.

After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.

I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.

I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.

We are now back in business with AIDA back in main DAQ at  01:40

01:45 full checks:

           Bias: AIDA-Graphana - Attachment 1

               Leakage current ramping up again ! S4 temperaturenot really going down either: 25C

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5

      ucesb - attachment 6

03:00 full checks

          No change to be highlighted. All pretty much the same as during previous full checks.

05:00 full checks

Expt still running very smoothly.

           Bias: AIDA-Graphana - Attachment 7-8

               Leakage current going down slightly

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
      per FEE64 Rate spectra - attachment 11

      Merger etc - attachment 12

      ucesb - attachment 13

07:15 all running smoothly.

07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.

07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.

 

Entry  Sun Apr 28 22:52:06 2024, Marc, 0:00-08:00 Monday 29 April  7x

Just another manic magic Monday, ....

Taking over Betool's AIDA shift. All is good.

This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.

01:15: AIDA04 is down again. DAQ shift crew informed.

01:27: AIDA04 is back and shift crew has been informed.

01:30 Checks:

     Bias and leakage current - Attachment 1-2

     Temperatures - Attachment 3

     ADC Data item - Attachment 4

     Rate Spectra - Attachment 5

     UCESB - Attachment  6

     Timestamp checks - Attachement 7

06:00 It has run very smoothly.   

 

Entry  Mon Jun 10 15:07:24 2024, Marc, 16:00-00:00 shift Monday 10/06/24 31x

Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7
7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

 

Entry  Wed Jun 12 23:00:16 2024, Marc, 00:00-08:00 Thursday 13 June 23x

Starting a new night shift - All good

1am full check:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

2:30am: AIDA02 FEE stopped working. DESPEC main DAQ was alerted and remobe AIDA from timestitcher while I was restarting AIDA DAQ.

2:45am: AIDA02  FEE is back and AIDA has been added to the timestitcher again in the main DAQ.

The AIDA runs at the time of the crash to look at are R7_348, 349.

All spectra were zeroed at this time and checked.

4:00 am full check:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - attachement # 11
Tape service - attachement # 12
Spectra - attachement # 13
ucesb - attachement # 14

6:00am full check:
DSSSD bias & leakage current  ok - attachment # 15-16
FEE64 temperatures  ok - attachment # 17
ADC data item stats - attachment # 18
Merger ok - attachement # 19
Tape service - attachement # 20
Spectra - attachement # 21
ucesb - attachement # 22

6:30 the nearline/offline analysis team reported possible noise issue in DSSD#2 and asked if the threshold were changed at any point. I can't find any mention of changes in the elog. 
I've not done a setup of the all DAQ when AIDA02 fee crashes earlier this morning. I just did a setup of the merger. Thus I don't think the threshold have changed when I stopeed and restarted the DAQ.
I can see that currently slow comparator threshod are set to 0xa for most of the DSSD2 fees except for AIDA06 which is set to 0xf. I'm not sure this is the thershold we would need to increase. Something to check during the next shift.

In the UCESB window I can see that the number of decay in DSSD2 is double the number of decay in DSSD1 while the number of implant is similar. Looking at other shift it looks that this was alsready the case in prvious shifts.

 

07.06 analysis of data file S181/R7_457 - attachment 23

 max deadtime 5.8% aida08, all other FEE64s < 1.4%

LEC data rate 1.788M, HEC data rate 4.3k

 

Entry  Sun Apr 28 06:57:59 2024, Magda Satrazani, 08:00-16:00 Sunday 28 April 36x

Took over the shift from Marc.

 

 

10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19

          Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD

10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).

         Grafana (AIDA) is now reporting data correctly too (attachment 21).

          Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )

                             Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.

                             Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.

 

14.45 analysis data file R21_500 - attachment 36

         max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%

         no timewarps

         HEC data item rate 2.7kHz

 

Entry  Thu Jun 13 06:49:54 2024, Magda Satrazani, 08:00 - 16:00 Thursday 13 June 2024 40x

Took over the shift from Marc.

08:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

10:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12

12:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 13-14
FEE64 temperatures  ok - attachment # 15
ADC data item stats - attachment # 16
Merger ok - Attachement # 17
Tape service - attachement # 18

 

 

12.34 analysis data file S181/R7_610 - attachment 19

max deadtime 8.7% (aida08), all other FEE64s < 2.7%

LEC data rate 2.000M, HEC data rate 4.7k

 

12.52 per p+n FEE64 1.8.L spectra - attachment 20

 aida09 pulser peak width 63 ch FWHM

per FEE64 1.8.H spectra - attachments 21-22

per FEE64 1.8.W spectra - 20us FSR - attachments 23-24

 

13.00 all histograms zero'd

 

14:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 19-20
FEE64 temperatures  ok - attachment # 21
ADC data item stats - attachment # 22
Merger ok - Attachement # 23
Tape service - attachement # 24

 

13.01 Beam OFF for S4 access

ADC data item stats - attachment 25

per FEE64 Rate spectra - attachment 26

14.12 analysis data file R7_654 - attachment 27

max deadtime 3.9% (aida08), all other FEE64s < 3%

LEC data rate 1.817M, HEC data rate 2.5k

14.00 data file R7_649 - attachment 28

max deadtime 4.5% (aida08), all other FEE64s < 3%

LEC data rate 1.756M, HEC data rate 2.5k

 

 

16:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 25-26
FEE64 temperatures  ok - attachment # 27
ADC data item stats - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30

Entry  Sat May 14 06:57:21 2022, MS, OH, Saturday 14 May 
08:00 Took over the shift from Philippos
      In system wide checks aida09 fails ASIC clock check but it is bit 6 which is ok
Entry  Sat May 14 09:00:21 2022, MS, OH, Saturday 14 May Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.pngScreenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.pngScreenshot_from_2022-05-14_10-02-51.png

10:00

Entry  Sat May 14 10:59:52 2022, MS, OH, Saturday 14 May Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.pngScreenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.pngScreenshot_from_2022-05-14_12-01-54.png

12:00

Entry  Sat May 14 13:01:58 2022, MS, OH, Saturday 14 May Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.pngScreenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.pngScreenshot_from_2022-05-14_14-03-54.png

14:00

Entry  Mon May 16 07:03:16 2022, MS, OH, Monday 16th May 08:00-16:00 17x

08:00 Took over the night shift from Tom (Attachments 1-3)

10:00: Attachement 4-6

around 10:30: Beam went off (Attachment 7)

11:00: Beam is back

12:00 Attachments 8-10

14:00 Attachements 11-13

15:50 Attachments 14-17

Entry  Wed May 19 09:09:27 2021, MS, JS, OH, May 16 08:00-24:00 21x
16th May 08:00 - 12:00 shift
Author: MS


08:00


	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7163 : 	 23  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      7      2      3      2      3      2      4      2      3      7   : 40228
aida02 :      3      8      3      2      2      2      2      4      2      3      7   : 39996
aida03 :     23      9      6      1      0      3      2      4      2      3      7   : 40100
aida04 :     34     27     17      7      2      4      3      3      2      2      7   : 38608
aida05 :     19      9      5      2      2      2      3      3      2      3      7   : 39844
aida06 :      5      5      2      1      0      4      2      4      2      3      7   : 40060
aida07 :     28      2     10      3      2      4      1      4      2      3      7   : 40192
aida08 :     19      5      5      1      2      5      1      3      2      3      7   : 39652
aida09 :     20      8      3      3      1      3      2      3      2      3      7   : 39648
aida10 :     27     10      0      2      2      2      2      3      1      4      7   : 40572
aida11 :      3      3      2      1      2      4      2      3      2      3      7   : 39652
aida12 :     16      7     11      2      3      4      2      2      2      3      7   : 39464
aida13 :     14     10      4      3      1      5      2      2      2      3      7   : 39400
aida14 :     21      9     10      1      1      2      2      3      1      4      7   : 40604
aida15 :     19      8      2      3      0      4      3      2      2      3      7   : 39436
aida16 :     24      5      8      3      3      4      2      2      2      3      7   : 39464







 *** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000






10:00


	
FEE64 module aida06 global clocks failed, 6
 FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7166 : 	 26  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      7      6      2      2      3      2      3      2      3      7   : 39716
aida02 :      8      4      3      2      2      3      2      3      2      3      7   : 39600
aida03 :     23     11      5      1      0      4      2      3      1      4      7   : 40740
aida04 :     42     25     16      7      2      4      4      3      2      2      7   : 38864
aida05 :     24      5      5      1      2      2      3      3      1      4      7   : 40824
aida06 :     11      4      4      1      1      4      2      4      2      3      7   : 40172
aida07 :     21      8      5      1      2      3      2      4      2      3      7   : 40196
aida08 :     15      5      6      1      1      4      2      4      2      3      7   : 40228
aida09 :     15     10      4      1      2      3      2      3      2      3      7   : 39660
aida10 :     23      8      2      2      2      2      2      3      1      4      7   : 40572
aida11 :      6      4      4      2      2      4      2      3      2      3      7   : 39736
aida12 :     18      8      8      1      4      4      3      2      2      3      7   : 39720
aida13 :     23      6      2      3      2      5      2      2      2      3      7   : 39436
aida14 :     29      3     11      1      1      2      2      3      1      4      7   : 40604
aida15 :     29      7      2      2      0      4      3      3      2      3      7   : 39948
aida16 :      2      3      6      2      3      4      2      2      2      3      7   : 39296






*** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

12:00-16:00
16th May 12:00 - 16:00 shift
Author: JS

11:57 Taking over from Magda. Running full checks.
usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75%
Current ok  06.410 uA      006.835 uA
Stats good 1Statistics aidas-gsi(6).png
Temps ok 1Temperature and status scan aidas-gsi(6).png
Analysis ok R7_385. Dead time FEE1 a little hight 6%

PAUSE: 166  RESUME: 166

 *** Timestamp elapsed time:      196.305 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.044                0.000
  1               12.405                0.000
  2                0.807                0.000
  3                7.260                0.000
  4                0.014                0.000
  5                0.458                0.000
  6                0.027                0.000
  7                0.497                0.000
  8                0.723                0.000
  9                0.000               88.857
 10                6.189                0.000
 11                1.443                0.000
 12                0.474                0.000
 13                0.000               35.565
 14                0.000                0.000
 15                0.147                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x716e : 	 34  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      7      4      2      2      3      2      3      1      4      7   : 40712
aida02 :     25     10      7      1      1      4      1      3      2      3      7   : 39556
aida03 :     22     10      5      1      0      4      2      4      2      3      7   : 40216
aida04 :     40     24     17      7      2      5      3      3      2      2      7   : 38736
aida05 :      4      8      3      0      1      3      3      3      1      4      7   : 40768
aida06 :     21      6      6      2      2      3      2      4      2      3      7   : 40228
aida07 :     25      4      6      3      2      3      2      4      2      3      7   : 40260
aida08 :     19      9      4      1      1      4      2      4      2      3      7   : 40244
aida09 :     16      9      4      2      2      3      2      3      2      3      7   : 39688
aida10 :     19     10      5      1      2      2      2      4      1      4      7   : 41100
aida11 :     21     10      2      1      2      3      3      3      2      3      7   : 39908
aida12 :     25      7      7      2      2      3      2      3      2      3      7   : 39756
aida13 :     13      7      3      4      2      5      2      3      2      3      7   : 39964
aida14 :     21      7      9      2      1      2      2      4      1      4      7   : 41116
aida15 :     23      6      2      3      0      4      3      3      2      3      7   : 39948
aida16 :      9     11     10      2      3      4      2      2      2      3      7   : 39452

Tom says Aida09 clock fail is ok as its status bit is "6".
The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken.

12:35 -
usbec ok.
Current ok 
Stats good 
Temps ok 
Analysis ok R7_395. Dead time FEE1 still high 6.6%

13:33 -
usbec ok. Max implants ~ 1.8kHz
Current ok    006.850 uA      007.250 uA 
Stats - Aida11 runing low < 5k was ~20k overnight
Temps ok 
Analysis R7_415. Dead time FEE1 & FEE10 high 10%

14:00
usbec ok - ucesb1.png
Current ok - bias1.png
Stast -  Aida11 low - 

14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n.

15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool.

18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00.
      During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise.
      The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs.
      We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy.
      During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips.

18:08 System wide checks all ok - bar some ADC but waveforms disabled
      Statistics ok - 210516_1809_Stats
      Temperatures ok - 210516_1809_Temp
      Bias and leakage ok - 210516_1810_Bias

18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time
18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side.

19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves.
      Now have around 4.2TB left which will provide around 80 hours of writing

20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not.

20:57 System wide checks. Clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1556 : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics ok - 210516_2056_Stats
      Temp ok -  210516_2058_Temp
      Bias and leakage current ok -  210516_2058_Bias

23:16 System wide checks:
      Clock still ok 
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155a : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

       Statistics - 210516_2315_Stats
       Temperature - 210516_2316_Temp
       Bias and leakage current ok - 210516_231
Entry  Sat Apr 17 15:29:27 2021, MS, Saturday 17 April 16:00-20:00 2021 6x
16:21


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021




18:29
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bd1 : 	 4  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last




	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      3      4      4      1      3      1      3      3      3      6   : 36332
aida02 :      4      3     13      2      2      3      2      4      2      3      6   : 36152
aida03 :     12      7      5      2      2      3      2      4      2      3      6   : 36088
aida04 :     17      6      9      5      1      1      2      3      3      3      6   : 36452
aida05 :     26      8      0      3      1      1      2      3      3      3      6   : 36296
aida06 :     22      6      9      4      3      4      2      3      3      3      6   : 36952
aida07 :     17      7      3      1      2      3      3      2      3      3      6   : 36300
aida08 :     24      5      4      4      1      1      2      3      3      3      6   : 36360
aida09 :     15     11      3      2      1      3      1      3      3      3      6   : 36292
aida10 :      2      3      5      2      1      2      2      2      3      3      6   : 35824
aida11 :      2      2      0      0      3      2      2      4      2      3      6   : 35800
aida12 :     18     15      7      5      4      3      1      3      3      3      6   : 36688














	
Entry  Mon Apr 19 06:46:15 2021, MS, Monday 19 April 07:47-12:00 2021 9x

07:47

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c46 :      74  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

        
             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x35 :      43  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     18     10      7      1      3      2      4      2      3      6   : 36380
aida02 :     11     15      5      3      2      3      2      3      1      4      6   : 36692
aida03 :     15     11     13      3      4      3      3      3      2      3      6   : 36164
aida04 :     22     14     20      5      1      3      4      3      2      3      6   : 36456
aida05 :      7      7      5      2      1      2      3      4      2      3      6   : 36132
aida06 :     21     14      6      2      3      3      1      2      2      4      6   : 37028
aida07 :     11      8      7      4      2      3      2      2      4      3      6   : 37212
aida08 :     18     10      7      5      3      1      4      3      2      3      6   : 36072
aida09 :     24     23     18      3      2      3      2      4      2      3      6   : 36504
aida10 :      6     19      5      2      2      2      2      3      2      3      6   : 35520
aida11 :     22     15      7      2      1      3      2      3      2      3      6   : 35648
aida12 :     35     13      8      2      5      3      1      3      3      3      6   : 36724

 

At 8:30 DAQ and Temperature Scan stopped working properly.

 

 

11:00

It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence

TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.

Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.

In the waveforms large 100kHz transiets could be seen - attachment 4

A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.

The waveforms here are much improved though - attachment 5 and 6

 

 

11:45

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      1      2      2      1      2      4      2      4      2      3      7   : 40148
aida02 :      5      4      2      2      1      2      2      3      2      3      7   : 39380
aida03 :      3      2      1      1      1      3      1      4      2      3      7   : 39692
aida04 :      4      2      1      0      1      3      3      3      2      3      7   : 39664
aida05 :      2      7      2      0      2      2      3      4      2      3      7   : 40160
aida06 :      2      3      2      2      2      2      1      3      3      3      7   : 40192
aida07 :     23      7      2      0      0      3      2      4      2      3      7   : 39988
aida08 :     23     11      4      0      1      3      1      4      2      3      7   : 39860
aida09 :      1      3      1      1      2      4      1      3      1      4      7   : 40396
aida10 :      1      3      3      1      1      4      1      2      3      3      7   : 39852
aida11 :      3      0      2      3      1      2      3      2      2      3      7   : 39116
aida12 :      5      8      0      3      1      4      2      3      2      3      7   : 39668

 

 

 

 

 

 

 

 

Entry  Wed Apr 21 06:56:42 2021, MS, Wednesday 21 April 08:00-12:00 2021 10x

8:00


 

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

             Base         Current         Difference
aida12 fault      0x0 :      0x15 :      21  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      6      7      2      2      4      3      2      2      3      7   : 39596
aida02 :     25      3      2      1      1      3      2      3      2      3      7   : 39548
aida03 :     28      5      3      1      2      2      2      3      2      3      7   : 39528
aida04 :     27      6      2      2      2      4      2      4      2      3      7   : 40316
aida05 :      9      9      4      3      1      3      2      3      3      3      7   : 40652
aida06 :     21      7      6      1      2      2      4      2      2      3      7   : 39564
aida07 :     20      7      5      0      1      2      2      3      2      3      7   : 39448
aida08 :     21      5      4      1      1      2      3      3      2      3      7   : 39708
aida09 :      1      3      1      3      0      2      1      4      2      3      7   : 39564
aida10 :     20      6      3      2      1      2      3      3      2      3      7   : 39728
aida11 :     18      4      2      1      2      2      2      2      2      3      7   : 38952
aida12 :     19      8      7      1      1      2      3      3      2      3      7   : 39772

 

From the merger statistics during the night there have been 1355757 TS errors in aida12. Corrigan said they have been quiet since around 3am though.



10:00

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 1, Failed 11

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

    
             Base         Current         Difference
aida12 fault      0x0 :      0x15 :      21  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     15      7      6      2      1      4      3      2      2      3      7   : 39508
aida02 :      1      2      1      0      1      2      2      4      2      3      7   : 39780
aida03 :     23      8      2      0      1      3      2      3      2      3      7   : 39548
aida04 :     23      9      1      2      3      3      2      4      2      3      7   : 40244
aida05 :     14      7      2      2      4      3      2      3      1      4      7   : 40784
aida06 :     18      5      5      2      2      2      3      3      2      3      7   : 39808
aida07 :     27      4      3      2      2      3      3      2      2      3      7   : 39420
aida08 :     23      6      3      2      2      2      3      3      2      3      7   : 39804
aida09 :     18      4      6      3      2      4      3      2      2      3      7   : 39592
aida10 :      6      9      5      3      2      3      2      3      2      3      7   : 39696
aida11 :     16      4      1      2      3      4      1      3      2      3      7   : 39536
aida12 :     18      5      7      1      0      3      3      3      2      3      7   : 39808

 

10:22

NH Noticed that the ASIC clocks had not been synchronised following the last AIDA restart. - attachment 7
As the beam is having issues they are not currently recording
I have now synchronised the ASIC clocks.
All ADC are now calibrated

 

11:06

Used the beam off time to change the drive writing to. Now writing to /media/ThirdDrive/TapeData/S460/R50

11:10

Beam is back they will start writing to file again


12:00

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

    
             Base         Current         Difference
aida12 fault      0x0 :      0x17 :      23  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      5      5      1      1      4      2      3      2      3      7   : 39720
aida02 :     16      8      4      1      0      2      1      4      2      3      7   : 39648
aida03 :      9      7      3      2      3      3      2      3      2      3      7   : 39692
aida04 :      8      7      5      2      3      5      3      2      2      3      7   : 39720
aida05 :     20      5      3      4      2      3      3      3      2      3      7   : 39976
aida06 :     25      8      0      3      1      2      3      3      2      3      7   : 39748
aida07 :     24      8      4      2      3      3      2      3      2      3      7   : 39776
aida08 :     22      9      0      2      2      2      3      3      2      3      7   : 39776
aida09 :      1      4      4      3      2      3      2      3      2      3      7   : 39620
aida10 :     13      4      4      3      2      3      1      3      2      3      7   : 39412
aida11 :      2      3      1      1      3      4      2      2      2      3      7   : 39184
aida12 :     16     10      7      2      1      3      1      3      2      3      7   : 39424

 

Entry  Wed Jun 22 07:12:46 2022, MS, Wednesday 22 June 8:00-16:00 16x

Took over the night shift from Tom.

8:10 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.

12:19 OH Disabled all discriminators to reduce the data rate.

13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10

14:00 : attachments 11-13

16:00 : attachments 14-16

 

Entry  Sat Jun 25 06:55:07 2022, MS, Saturday 25 June 2022 8:00-16:00 15x

Took over the night shift from Tom.

7:00 The beam is back.

8:00 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

14:00 : attachments 10-12

16:00 : attachments 13-15

 

 

ELOG V3.1.4-unknown