Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

19:24 pm screenshots

and

17:18 pm screenshots

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13

9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20

11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27

Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.

However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).

Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.

Another set of sceenshots taken at around 23:00 - attachments 19 - 26.

16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

14:50 Checks

Leakage current  is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1

It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.

18:15 Checks

Everything appears to be going smoothly 

Grafana - DSSSD bias  and leakage current - attachments 2 and 3

FEE64 temperatures  - It appears okay, nothing strange , attachment 4

ADC data item stats - attachment 5

Per FEE64 Rate spectra - attachment 6

 ucesb - attachment 7

Merger Link Data Rates  - attachment 8

18:16

The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.

18:20  it came back to work normally

............................

20:00 Checks

Nothing new to report. All is well.

20:33 Aida02 showed as not reading - attachment 10

20:34 it resumed reading and returned to normal operation.

21:08 Aida02 was  not reading - attachment 11 , had to emailed Tom to fix it .

The shift crew have restarted ucesb and AIDA  is now showing OK - attachment 12

so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.

ADC data item stats and Merger etc  appear to be going smoothly - attachments 13 and 14

...........................................

22:00 Checks

Still running smoothly.

DSSD bias and  leakage current - ok

FEE64 temperatures - ok

Per FEE64 Rate spectra - ok

ADC data item stats - ok

Ucesb - ok

.................................

23:00 Checks

 Nothing new to report. Everything is going well , and I have captured screenshots of each one

Leakage currents dropped a bit See attachment 15

.............................

23:50 Checks

All seems smooth.

This appeared in the system log:

Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

These messages seem to appear sporadically.

18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?

18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.

aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters

Thanks Nic!

22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)

22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.

AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.

17:00

DSSSD bias & leakage current - attachment 2

FEE64 temperatures OK - attachment 3

Statistics  attachment 4

17:39

Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.

23:19 Flange removed. Starting to take beam.

16:00 aida04 not producing data

          power cycle all FEE64s to recover DAQ

16.20 analysis data file R21_668

         max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%

         no timewarps

         HEC data item rate 1.9kHz

21:30 screenshots

and

23:24 screenshots

First checks of shift. All looks ok.

Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.

Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)

checks at 16.00 - merger has crashed - aida01 dropped out  - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09 

Reset 01, restarted DAQ, and restarted as R16.

16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.

Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms

RUN17 STARTED 16.51|

 HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.

20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift

21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps

22.48 - things apear ok. (Attachments 23-26) 

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :     25     10      4      3      1      2      2      3      3      3      6   : 36500
aida03 :     27      8      5      2      3      4      3      3      2      3      6   : 36092
aida04 :     30      4      4      5      0      2      4      2      3      3      6   : 36472
aida05 :     19      5      7      1      3      2      2      2      2      4      6   : 37060
aida06 :     18     13      1      2      1      3      2      3      3      3      6   : 36544
aida07 :     24      7      3      2      2      3      1      3      2      4      6   : 37384
aida08 :     15     11      2      5      2      4      1      4      3      3      6   : 37076
aida09 :      7      3      4      1      5      6      5      4      3      2      6   : 36308
aida10 :      8      3     13      4      3      3      1      4      2      3      6   : 36040
aida11 :     18     11     17      7      4      4      4      3      2      3      6   : 36752
aida12 :     12      7      7      7      4      4      2      3      2      3      6   : 36024



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021



17:27.  Checked system stats, temps, leakage etc. looks ok 


18:49. Tape server writing to disk /NULL/R21_9+
       /TapeData points to /media/SecondDrive/ - 3.6Tb free

19.15. checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9     16      2      2      2      1      3      3      3      6   : 36424
aida02 :      1      3      5      4      1      3      3      2      3      3      6   : 36268
aida03 :     24     13     11      6      3      3      2      4      2      3      6   : 36472
aida04 :      5      4      4      4      3      3      3      2      3      3      6   : 36404
aida05 :     17      8      4      2      2      2      2      2      2      4      6   : 36996
aida06 :     26     11      3      2      0      3      2      3      3      3      6   : 36528
aida07 :     24      7      2      1      1      1      2      3      2      4      6   : 37272
aida08 :     13     12      6      3      3      4      1      4      3      3      6   : 37140
aida09 :     22     15     19      2      6      5      4      4      3      2      6   : 36416
aida10 :     16     11      6      4      3      3      1      4      2      3      6   : 36024
aida11 :     12      4      5      4      2      3      3      3      2      3      6   : 35872
aida12 :     18     12      9      4      4      4      2      3      2      3      6   : 36024


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


19.53.  Checked system stats, temps, leakage etc. looks ok 

20:30.  Checked system stats, temps, leakage etc. looks ok 

20:59.  Checked system stats, temps, leakage etc. looks ok 



21:31 System checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      9     17      3      2      2      1      3      3      3      6   : 36492
aida02 :      5      1      1      6      0      2      2      3      3      3      6   : 36332
aida03 :      2     10     11      6      4      4      3      3      2      3      6   : 36296
aida04 :     27     12      3      4      0      2      3      2      3      3      6   : 36220
aida05 :     19      9      7      2      1      2      2      2      2      4      6   : 36996
aida06 :     28      5      0      4      0      3      1      3      3      3      6   : 36248
aida07 :     19     13     10      6      2      2      1      3      2      4      6   : 37524
aida08 :     18      9      6      0      1      4      1      3      3      3      6   : 36400
aida09 :     17     19     12      3      6      5      4      4      3      2      6   : 36348
aida10 :     23      9      7      4      2      3      1      4      2      3      6   : 35988
aida11 :     19     10      3      3      2      3      3      3      2      3      6   : 35884
aida12 :     12      6      0      3      2      4      2      3      2      3      6   : 35648


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


22.06.  Checked system stats, temps, leakage etc. looks ok 

22.18CEST BNC PB-5 amplitude changed from 1V to 2V
          fioler NULL/R21_67

22.23CEST All histograms zero'd

22.43.  Checked system stats, temps, leakage etc. looks ok 
23.21.  Checked system stats, temps, leakage etc. looks ok 


23.46 system checks
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     17     12      7      0      3      3      2      4      2      3      6   : 36180
aida02 :     35      3      5      2      0      2      1      3      3      3      6   : 36148
aida03 :     22      6     10      2      1      3      2      4      2      3      6   : 36136
aida04 :     27      9      3      5      2      4      3      3      2      3      6   : 36100
aida05 :     20      9      3      3      2      2      2      2      2      4      6   : 37032
aida06 :     25     11      1      2      0      3      1      3      3      3      6   : 36236
aida07 :     20      6      6      2      3      3      2      4      1      4      6   : 37216
aida08 :     25     10      0      2      1      3      1      3      3      3      6   : 36276
aida09 :     10      4      3      4      4      5      4      4      3      2      6   : 35960
aida10 :     26      9      3      0      1      3      1      4      2      3      6   : 35744
aida11 :      2      3      4      2      2      3      3      3      2      3      6   : 35744
aida12 :      4      9      7      2      2      4      2      3      2      3      6   : 35720
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC

All looks good.

Screenshots attached

10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   

16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before

15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).

          TEMP OK.

          Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.

          9/16 <20kHz, max 176k

15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.

          No change in noise observed.

          9/16 <20kHz, max 187k

15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.

          10/16 <20kHz, max 158k.

Current bPlast grounding scheme in attachments 8-9.

Everything is OK now.

Screenshots attached.

15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.

           Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.

           Histograms okay, some FEE64s now have hot channels. Attachment 2.

           9/16 aidas < 20kHz - max rate 162k

18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.

TO-DO (kicking the can down the road):

- Some work to do on noise, but we might have to accept the situation as it currently is.

- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.

- Try different grounding configuration:

    - Grounding drain wires to frame. Observe AIDA conditions.

    - Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.

TO-DO for 11.04.2024

- Try bringing bPlast drain wire ground back to the PSU ground for PN 300 and R&SRMP4040.

- Recheck the downstream detector bias and ground scheme.

-

9:27 Restarted AIDA and hit go at 9:54. Situation the same as it was in the evening of 10.04.2024. 10 out of 16 FEE64s in good condition. Waveforms of aida16,

        TEMP OK - HV OK. See attachments 1-6.

10:20 We grounded the output of the R&SRMP4040 to to the drain wire ground of the bPlast ribbon cables and the output of the PN 300 PSU that is powering the booster boards. Results given by attachments 7 -11. We only saw a marginal improvement in the overall noise condition.

10:32 We connected the ground of the frame to the drain wire ground, that is also mutually connected to the ground of the R&SRMP4040 output ground and PN 300 output ground. Results given by attachments ???. We did not see any improvement in the overal noise condition.

11.00 DSSSD HV OFF

          FEE64 power OFF

          Check seating of all adaptor PCBs and drain/ground wires secured - generally OK

          Restart

          1.8.W spectra - 20us FSR - attachments 15-16

          per FEE64 Rate spectra - attachment 17

           10/16 < 20k, max 110k

          ADC data item stats - attachment 18

          Incremental improvement.

11.45 DSSSD HV OFF

          FEE64 power OFF

          test - daisy chain removed

          Restart

          1.8.W spectra - 20us FSR - attachments 20-21

          per FEE64 Rate spectra - attachment 19

           10/16 < 20k, max 210k

          ADC data item stats - attachment 22

          overall somewhat worse

12.20 DSSSD HV OFF

          FEE64 power OFF

          Re-install test - daisy chain, Tighten aida04 DSSSD ribbon cable drain wire.

          Restart

          1.8.W spectra - 20us FSR - attachments 23-24

          per FEE64 Rate spectra - attachment 25

          ADC data item stats - attachment 26

           10/16 < 20k, max 110k

          Status quo ante

         aida04 & aida08 1*W & 2*W spectra - 200us FSR - attachments 27-31

           large transients observed for aida04 1*W - cable or ASIC fault?

         aida16 **W spectra - 200us FSR - attachments 32-35

           all channels apopear to be working but mix of high/low noise channels for asics #3-4, asics #1-2 all appear high noise

14.00 DSSSD HV OFF

          FEE64 power OFF

          Disconnect ribbon cables from aida04 adaptor PCB

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 37

          per FEE64 Rate spectra - attachment 36

          ADC data item stats - attachment 38

          Implies origin of large transients observed in aida04 asic#1 is downstream of the FEE64 adaptor PCB, i.e. ribbon cable or DSSSD.

14.40 DSSSD HV OFF

          FEE64 power OFF

          re-connect ribbon cables from aida04 adaptor PCB

          1x pin J2 slightly bent - straightended with screwdriver - FFSD connector insertion OK

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 40-41

          per FEE64 Rate spectra - attachment 42-43

          ADC data item stats - attachment 39

            10x < 20k, max 120k

           aida11 asic#4 1.8.W spectra - 200us FSR - attachment

             no large transients observed

           DSSSD #1 & #2 bias from -120V to -100V

            ADC data item stats - attachment 40

            9x < 20k, max 220k (aida08)

            per FEE64 Rate spectra - attachment 41

17.15 bPlas ON -

        Current bPlast ground configuration:

         bPlas current ground configuration - drainwires of all ribbon cables excluding short side (cont. with snout) are grounded back to the bPlast R&SRMP4040 PSU, on the 29 V output. 4 V PSU to booster boards are floating, output of PN 300 is grounded to the PSU. The mesytec PSU, that also powers bPlast SiPMs at

         29V is not grounded to anything but the frame. Snout is currently light tight, I suspect internal radiation from FATIMA.

         It is noted that bPlast current draw fluctuated significantly (+/- 300 mV) as a result of thresholds set to the detector. I tried to set the thresholds HIGH to stabilise the bPlast detector as the power draw is fluctuating greatly.         

          AIDA noise very good

            ADC data item stats - attachment 42

            12x < 20k, max 83k (aida04)

            per FEE64 Rate spectra - attachment 43

To Do list

- separate 29V (low current) and 4V (high current) return paths/ground refs

- The mesytec PSU that powers the bPlast detector  is running to frame, it might be an idea to ground ribbon c