AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40 MONNET
  DESPEC, Page 35 of 37  ELOG logo
  ID Date Author Subjectdown
  98   Tue Nov 19 21:22:06 2019 CA, CB, DK, NH19/20 November 2019 - overnight shift
19 November

Waiting for 40Ar beam to be delivered for tests.
Objective today is to assess degrader thickness suitable to stop 40Ar in 2nd DSSD layer.
Energy expected at 300 MeV/A, around 1E3-4 beam intensity - to be determined.

Tomorrow fragments could be expected.

DSSD / FEE64 pairing as follows

DSSD3 (most downstream): FEEs 9,10,11,12
DSSD2 (middle): FEEs 5,6,7,8
DSSD1 (most upstream): FEEs 1,2,3,4
+plastic scintillator furthest upstream

New AIDA Google spreadsheet created with leakage current, to monitor them during beamtime. First entry added. Bias attached.
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023

DESPEC platform has already been moved. No effect on rates - see attached. FEE1 is missing one channel. FEE9, 10 noisy as expected. See attached.


System wide checks *all OK* except
Master clock failed (normal)
ADC calibration fails on 6 and 12. Recalibrated via FADC Align. All OK now.

MBS transfer OK. See attached.

Good event statistics OK. See attached.

FEE64 temperatures OK. See attached.

Merger and data rate OK. See attached.

Waiting for beam.



20 November

01:07 - Still no beam. 

System wide checks OK. 
Stats OK. 
Temps OK. 
MBS OK. 
Merger and data rate OK.


02:35  - Beam is expected soon
Tried enabling writing data to disk. Merger crashed. FEE64 stopped responding to commands.
Reset Merger. FEE64 now respond to commands.
Tried enabling writing to disk again. Merger crashed once more. Reset merger.
Reset Tape Server interface. Now writing to disk.

No data currently being sent to MBS, but operators say they "are adding a new detector". May be normal.

DAQ & Merger going OK. Writing to Nov19/R1

No beam yet.

Temperatures OK
System wide checks OK
Stats OK. See attached.
MBS not working.
Merger and data rate OK. See attached.

~02:48 Beam on. 40Ar. 300 MeV/A, 500 pps. Degrader settings unclear.

03:54 Degrader settings have been changing between 4.5, 1 and 6 ug/cm2
Currently on 6 g/cm2.
AIDA seen through MBS shows no HE on detector 3, and Det 1 and 2 show no counts on positive y (FEE6 and 1). That is not the case from MIDAS.

04:30 The aim of the tests above was to have no 40Ar at all in DSSD3. It does not appear that 6 g/cm2 is sufficient, as we still see events. The reduction between 1 & 2 vs. 3 is about a factor of 2-10 depending on ADC channel. See attached.

~04:30 Moved to 6.2 g/cm2 degrader. See attached.

~04:45 Moved to 5.8 g/cm2. See attached.

~05:09 Moved to 6.4 g/cm2. See attached.

05:31 High energy channel spectra (1.8.H) for all FEE64 w/ 6.4 g/cm2 degrader - See attachments 13 & 14

05:38: Moved to 1.0 g/cm2. See attached. Difference is not huge which may indicate these spectra are misleading, as partly expected. A more detailed analysis will be carried out off line.

06:00: Beam stopped. ~R77. AIDA keeps running background over the day. Beam should restart around 11 pm later today.
Attachment 1: Bias2235.png
Bias2235.png
Attachment 2: Rates2236.png
Rates2236.png
Attachment 3: 19112241MBS.png
19112241MBS.png
Attachment 4: 19112242Stats.png
19112242Stats.png
Attachment 5: 19112244Temperatures.png
19112244Temperatures.png
Attachment 6: 19112246Merger.png
19112246Merger.png
Attachment 7: 20110239.png
20110239.png
Attachment 8: 20110240DataTransfer.png
20110240DataTransfer.png
Attachment 9: 6gdegrader.png
6gdegrader.png
Attachment 10: 6.2degrader.png
6.2degrader.png
Attachment 11: 5.8degrader.png
5.8degrader.png
Attachment 12: 6.4degrader.png
6.4degrader.png
Attachment 13: 18h_0530.png
18h_0530.png
Attachment 14: 18H_0528.png
18H_0528.png
Attachment 15: 1gcm2degrader.png
1gcm2degrader.png
  330   Wed May 19 09:12:52 2021 Muneerah, Betool, LS17th May 00:00-12:00
Auther: Muneerah and Betool
02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok
      Statistics ok - 210517_0202_Stats
      Temperatures ok - 210517_0202_Temp
      Bias and leakage ok - 210517_0200_Bias 
02:05 System wide checks:
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155d : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      8      3      3      3      5      2      2      2      3      7   : 39460
aida02 :     22      4      5      0      3      2      2      3      2      3      7   : 39560
aida03 :     21     10      2      1      1      3      2      4      2      3      7   : 40100
aida04 :     18      7      2      0      3      3      3      2      2      3      7   : 39392
aida05 :     16      6      5      1      2      4      3      2      2      3      7   : 39520
aida06 :     18      6      6      2      2      2      2      3      2      3      7   : 39576
aida07 :     12      8      6      1      2      3      1      3      2      3      7   : 39408
aida08 :     13     11      4      0      1      3      3      2      2      3      7   : 39308
aida09 :     10      8      4      2      3      4      2      3      2      3      7   : 39848
aida10 :     16      8      2      1      2      4      2      3      1      4      7   : 40768
aida11 :     26     10      2      2      1      2      2      3      2      3      7   : 39512
aida12 :     16      9      2      1      2      4      2      2      2      3      7   : 39240
aida13 :     15      9      3      2      2      4      2      2      2      3      7   : 39284
aida14 :     25     10      2      3      3      3      2      3      2      3      7   : 39796
aida15 :      4      2      2      1      1      3      2      2      2      3      7   : 38944
aida16 :     16      7      3      3      1      3      3      2      2      3      7   : 39368

04:00   Statistics ok - 210517_0403_Stats
        Temperatures ok - 210517_0403_Temp
        Bias and leakage ok - 210517_0400_Bias 
04:05 System wide checks:
      Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1560 : 	 14  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      7      5      2      3      5      2      2      2      3      7   : 39492
aida02 :     23      6      7      2      1      3      2      3      2      3      7   : 39676
aida03 :     17      9      3      2      0      3      2      3      2      3      7   : 39548
aida04 :     16      4      1      2      1      3      3      3      2      3      7   : 39792
aida05 :     22      7      3      3      1      4      3      3      2      3      7   : 40032
aida06 :     18      6      7      3      1      3      2      3      2      3      7   : 39688
aida07 :     18      7      4      1      2      4      1      4      2      3      7   : 40032
aida08 :     19      5      4      1      1      3      3      3      2      3      7   : 39828
aida09 :     10      6      7      1      3      4      2      2      2      3      7   : 39336
aida10 :     12     10      2      3      1      4      2      3      1      4      7   : 40768
aida11 :     10      9      4      3      2      1      2      3      2      3      7   : 39440
aida12 :      8      7      6      1      2      4      2      3      2      3      7   : 39768
aida13 :     11      2      6      1      2      3      2      3      2      3      7   : 39612
aida14 :     27      7      4      2      3      3      2      3      2      3      7   : 39780
aida15 :      4      2      3      1      1      3      2      2      2      3      7   : 38960
aida16 :     18      8      4      2      1      4      2      2      2      3      7   : 39240


06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA
06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again.
07:01 it is working fine.
07:40 Screen shots after the restart
 	Statistics ok - 210517_0742_Stats
        Temperatures ok - 210517_0743_Temp
        Bias and leakage ok - 210517_0740_Bias 

System wide checks

	
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0x61da : 	 0x61dd : 	 3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      6      3      1      1      3      1      3      3      3      7   : 40336
aida02 :     11      6      0      1      1      5      2      4      2      3      7   : 40252
aida03 :      0      5      2      3      3      3      3      4      2      3      7   : 40424
aida04 :      2      4      2      1      2      4      3      3      2      3      7   : 39912
aida05 :     11      7      5      1      2      2      4      2      3      3      7   : 40532
aida06 :     20      6      0      2      3      2      1      3      3      3      7   : 40320
aida07 :     15      5      1      2      2      3      2      2      3      3      7   : 40116
aida08 :      2      3      4      1      3      3      2      2      3      3      7   : 40128
aida09 :     18      5      1      3      2      4      2      2      3      3      7   : 40288
aida10 :     11      6      3      4      2      3      3      2      2      4      7   : 41484
aida11 :      4      4      5      2      2      2      3      3      2      3      7   : 39744
aida12 :     18      7      2      3      2      2      4      3      2      3      7   : 40064
aida13 :      2      2      1      3      0      3      2      4      2      3      7   : 39944
aida14 :     16      4      5      3      3      3      3      2      2      4      7   : 41552
aida15 :      9      3      1      1      2      3      2      3      3      3      7   : 40556
aida16 :     16      5      1      2      1      2      3      4      2      3      7   : 40184

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs


12:52 Check FPGA Timestamp errors gives following error message:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"
Tcl Call Trace
6: DocSubstSystemFile sock8 error 500 can't\ read\ &quot\;TS_Base_Collected&quot\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n&quot\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}&quot\;\n\ \ \ \ (procedure\ &quot\;do_TS_errors&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;do_TS_errors\ &quot\;\n\ \ \ \ (&quot\;TSERRORS&quot\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...&quot\;\n\ \ \ \ (procedure\ &quot\;do_click&quot\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n&quot\;do_click&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ &quot\;&quot\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ &quot\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...&quot\;\n\ \ \ \ (file\ &quot\;Check.tcl&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;source\ Check.tcl&quot\;\n\ \ \ \ (in\ namespace\ eval\ &quot\;::Check&quot\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n&quot\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n&lt\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n&gt\;...&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ #0\ [list\ subst\ \$script\]&quot\;\n\ \ \ \ (procedure\ &quot\;SubstFile&quot\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n&quot\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]&quot\;\n\ \ \ \ (procedure\ &quot\;Subst_File&quot\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n&quot\;Subst_File\ \$template\ \$interp&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_application/x-tcl-template&quot\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n&quot\;\$cmd\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_Handle&quot\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n&quot\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;DocDomain&quot\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n&quot\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml&quot\;\n\ \ \ \ (&quot\;eval&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]&quot\;
5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\"
4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE
3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {}
2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml}
1: HttpdReadPostGlobal sock8 Httpds

Dead time check:

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_24
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252995858 (  451517.9 Hz)
        Other data format:    8924143 (   15926.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.2 Hz)
                               RESUME:         85 (       0.2 Hz)
                              SYNC100:      34087 (      60.8 Hz)
                              WR48-63:      34087 (      60.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8855799 (   15804.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      560.323 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000              349.960
  1                0.002                0.000
  2                0.000                5.974
  3                1.105                0.000
  4                0.000              447.454
  5                0.000                2.809
  6                0.000               16.727
  7                0.137                0.000
  8                0.090                0.000
  9                0.000              429.897
 10                0.461                0.000
 11                0.018                0.000
 12                0.000                8.985
 13                0.000              466.311
 14                0.000               15.095
 15                0.346                0.000

13:24 Beam is off at the moment...

13:43 Changing spill structure from 1.5s/3.5s to 1s. 

13:50

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_29
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251763810 (  930342.1 Hz)
        Other data format:   10156190 (   37530.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        205 (       0.8 Hz)
                               RESUME:        205 (       0.8 Hz)
                              SYNC100:      33009 (     122.0 Hz)
                              WR48-63:      33009 (     122.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10089762 (   37284.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     527077 (    1947.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      270.614 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.034                6.803
  1                8.461                0.000
  2                0.569                0.000
  3               11.664                0.000
  4                0.024               26.416
  5                0.854                0.000
  6                0.069                0.000
  7                0.741                0.000
  8                1.336                0.000
  9                0.000               75.926
 10                0.353                0.000
 11                0.105                0.000
 12                0.070                0.000
 13                0.000               93.585
 14                0.017                0.000
 15                0.121                0.000

15:06 Spill structure back to 1.5 on 2s off.
Attachment 1: 210517_0200_Bias.png
210517_0200_Bias.png
Attachment 2: 210517_0202_stats.png
210517_0202_stats.png
Attachment 3: 210517_0202_temp.png
210517_0202_temp.png
Attachment 4: 210517_0400_Bias.png
210517_0400_Bias.png
Attachment 5: 210517_0403_stats.png
210517_0403_stats.png
Attachment 6: 210517_0403_temp.png
210517_0403_temp.png
Attachment 7: 210517_0740_Bias.png
210517_0740_Bias.png
Attachment 8: 210517_0742_Stats.png
210517_0742_Stats.png
Attachment 9: 210517_0743_Temp.png
210517_0743_Temp.png
Attachment 10: 0833_Stats.png
0833_Stats.png
Attachment 11: 0902_Stats.png
0902_Stats.png
Attachment 12: 1241_Stats.png
1241_Stats.png
Attachment 13: 1347_Stats.png
1347_Stats.png
Attachment 14: FEETemps_1042.png
FEETemps_1042.png
Attachment 15: FEETemps_1246.png
FEETemps_1246.png
Attachment 16: FEETemps_1350.png
FEETemps_1350.png
Attachment 17: leakage_currents_1302.png
leakage_currents_1302.png
  108   Wed Dec 18 02:44:27 2019 NH17.12.19, 18.12.19
Beam: 238U (850 MeV/u)
Fragment: 168Re

Tuning accelerator and FRS, believed since 3AM we are implanting on AIDA.

FEE issues:
One HEC in aida03 seems to be empty - LEC channel fills
aida09 has a lot of weird channels, check/load doesn't seem to fix
aida10 still no data after cable replaced

DSSD leakages OK - slightly higher now

Temps good


Fig 4: Rates 14:32

Leakages and temps good AS Before

Checking if we are implanting properly - possibly into the first DSSD at the moment

Fig 5 FRS PID
Fig 6 DSSD 
Fig 7 DSSD (PID gated)
Attachment 1: 1812-stats.png
1812-stats.png
Attachment 2: 1812-leakages.png
1812-leakages.png
Attachment 3: 1812-temps.png
1812-temps.png
Attachment 4: 1812v2-Rates.png
1812v2-Rates.png
Attachment 5: nh_1712_zaoq.png
nh_1712_zaoq.png
Attachment 6: nh_1712_dssd.png
nh_1712_dssd.png
Attachment 7: nh_1712_dssd_id.png
nh_1712_dssd_id.png
  602   Wed Apr 24 15:37:59 2024 PP16:00-00:00 shift Wednesday 24 April

All seems OK.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_16-38-45.png
Screenshot_from_2024-04-24_16-38-45.png
Attachment 2: Screenshot_from_2024-04-24_16-39-09.png
Screenshot_from_2024-04-24_16-39-09.png
Attachment 3: Screenshot_from_2024-04-24_16-39-38.png
Screenshot_from_2024-04-24_16-39-38.png
Attachment 4: Screenshot_from_2024-04-24_16-40-24.png
Screenshot_from_2024-04-24_16-40-24.png
Attachment 5: Screenshot_from_2024-04-24_16-41-15.png
Screenshot_from_2024-04-24_16-41-15.png
Attachment 6: Screenshot_from_2024-04-24_16-43-41.png
Screenshot_from_2024-04-24_16-43-41.png
Attachment 7: Screenshot_from_2024-04-24_16-44-15.png
Screenshot_from_2024-04-24_16-44-15.png
Attachment 8: Screenshot_from_2024-04-24_16-44-32.png
Screenshot_from_2024-04-24_16-44-32.png
  651   Tue Jun 11 15:01:07 2024 PP TD16:00-00:00 shift Tuesday 11/06/24
All seems OK

Screenshots attached.

17.33 per FEE64 1.8.W spectra - attachments 8-9
      per p+n FEE64 1.8.L spectra - attachment 10
       aida09 pulser peak width 66 ch FWHM
       cf 
       c. 11:00 today 57 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/650
       cf
       c. 00:10 today 53 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/649

17.48 ADC data item stats - attachment 11
      All histograms zero'd

17.51 Analysis data file S181/R6_96 - attachment 12
      max deadtime 5% (aida08)
      HEC Data items 3.7kHz, LEC data items 1.8MHz
      cf.
      c. 12.29 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/650
      c. 00.09 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/649
Attachment 1: Screenshot_from_2024-06-11_16-06-43.png
Screenshot_from_2024-06-11_16-06-43.png
Attachment 2: Screenshot_from_2024-06-11_16-07-53.png
Screenshot_from_2024-06-11_16-07-53.png
Attachment 3: Screenshot_from_2024-06-11_16-09-03.png
Screenshot_from_2024-06-11_16-09-03.png
Attachment 4: Screenshot_from_2024-06-11_16-12-05.png
Screenshot_from_2024-06-11_16-12-05.png
Attachment 5: Screenshot_from_2024-06-11_16-12-46.png
Screenshot_from_2024-06-11_16-12-46.png
Attachment 6: Screenshot_from_2024-06-11_16-13-40.png
Screenshot_from_2024-06-11_16-13-40.png
Attachment 7: Screenshot_from_2024-06-11_16-14-11.png
Screenshot_from_2024-06-11_16-14-11.png
Attachment 8: Screenshot_from_2024-06-11_16-17-21.png
Screenshot_from_2024-06-11_16-17-21.png
Attachment 9: Screenshot_from_2024-06-11_17-34-26.png
Screenshot_from_2024-06-11_17-34-26.png
Attachment 10: Screenshot_from_2024-06-11_17-33-33.png
Screenshot_from_2024-06-11_17-33-33.png
Attachment 11: Screenshot_from_2024-06-11_17-39-32.png
Screenshot_from_2024-06-11_17-39-32.png
Attachment 12: Screenshot_from_2024-06-11_17-48-43.png
Screenshot_from_2024-06-11_17-48-43.png
Attachment 13: R6_98
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258260025 ( 1785474.1 Hz)
        Other data format:    3659975 (   25303.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        939 (       6.5 Hz)
                               RESUME:        938 (       6.5 Hz)
                              SYNC100:      32538 (     225.0 Hz)
                              WR48-63:      32538 (     225.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3593022 (   24840.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     539819 (    3732.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      144.645 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.468                0.000
  1                0.015                0.000
  2                0.040                0.000
  3                0.872                0.000
  4                0.007                0.000
  5                0.312                0.000
  6                0.000                0.000
  7                7.299                0.000
  8                0.001                0.000
  9                3.731                0.000
 10                2.112                0.000
 11                0.008                0.000
 12                0.000                0.000
 13                0.367                0.000
 14                0.001                0.000
 15                1.636                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15794317       8531          0          0         49         49       2058       2058          0       4317          0      10696
  1    7121332       1752          0          0          3          3        873        873          0          0          0      24880
  2   15560192    1259782          0          0          6          6       2078       2078          0    1255614          0      14645
  3   17271150       4664          0          0        134        134       2198       2198          0          0          0      19004
  4    6664272     367237          0          0          2          2        840        840          0     365553          0       5226
  5   16529211       4220          0          0         48         48       2062       2062          0          0          0      23130
  6    5459932     300460          0          0          0          0        680        680          0     299100          0      14450
  7   45345978      11675          0          0        200        199       5638       5638          0          0          0     336191
  8    6229388       1534          0          0          1          1        766        766          0          0          0      16825
  9   38606423    1232596          0          0        155        155       5043       5043          0    1222200          0      15969
 10   19820101       5140          0          0        195        195       2375       2375          0          0          0       9560
 11    4555885     447544          0          0          2          2        651        651          0     446238          0      14419
 12    4015114        976          0          0          0          0        488        488          0          0          0       3706
 13   15417116       3872          0          0         46         46       1890       1890          0          0          0      14088
 14    5845488       1420          0          0          1          1        709        709          0          0          0      10117
 15   34024126       8572          0          0         97         97       4189       4189          0          0          0       6913
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.520s (  876.243 blocks/s,  54.765 Mb/s)
  648   Mon Jun 10 15:07:24 2024 Marc16:00-00:00 shift Monday 10/06/24

Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7
7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

 

Attachment 1: Screenshot_from_2024-06-10_16-54-55.png
Screenshot_from_2024-06-10_16-54-55.png
Attachment 2: Screenshot_from_2024-06-10_16-55-40.png
Screenshot_from_2024-06-10_16-55-40.png
Attachment 3: Screenshot_from_2024-06-10_17-01-34.png
Screenshot_from_2024-06-10_17-01-34.png
Attachment 4: Screenshot_from_2024-06-10_17-02-22.png
Screenshot_from_2024-06-10_17-02-22.png
Attachment 5: Screenshot_from_2024-06-10_17-05-47.png
Screenshot_from_2024-06-10_17-05-47.png
Attachment 6: Screenshot_from_2024-06-10_17-06-52.png
Screenshot_from_2024-06-10_17-06-52.png
Attachment 7: Screenshot_from_2024-06-10_17-13-25.png
Screenshot_from_2024-06-10_17-13-25.png
Attachment 8: Screenshot_from_2024-06-10_18-59-09.png
Screenshot_from_2024-06-10_18-59-09.png
Attachment 9: Screenshot_from_2024-06-10_18-59-42.png
Screenshot_from_2024-06-10_18-59-42.png
Attachment 10: Screenshot_from_2024-06-10_19-02-24.png
Screenshot_from_2024-06-10_19-02-24.png
Attachment 11: Screenshot_from_2024-06-10_19-01-43.png
Screenshot_from_2024-06-10_19-01-43.png
Attachment 12: Screenshot_from_2024-06-10_19-03-42.png
Screenshot_from_2024-06-10_19-03-42.png
Attachment 13: Screenshot_from_2024-06-10_19-04-24.png
Screenshot_from_2024-06-10_19-04-24.png
Attachment 14: Screenshot_from_2024-06-10_19-05-59.png
Screenshot_from_2024-06-10_19-05-59.png
Attachment 15: Screenshot_from_2024-06-10_19-07-37.png
Screenshot_from_2024-06-10_19-07-37.png
Attachment 16: Screenshot_from_2024-06-10_20-56-43.png
Screenshot_from_2024-06-10_20-56-43.png
Attachment 17: Screenshot_from_2024-06-10_20-57-19.png
Screenshot_from_2024-06-10_20-57-19.png
Attachment 18: Screenshot_from_2024-06-10_20-59-07.png
Screenshot_from_2024-06-10_20-59-07.png
Attachment 19: Screenshot_from_2024-06-10_20-59-34.png
Screenshot_from_2024-06-10_20-59-34.png
Attachment 20: Screenshot_from_2024-06-10_21-00-23.png
Screenshot_from_2024-06-10_21-00-23.png
Attachment 21: Screenshot_from_2024-06-10_21-02-53.png
Screenshot_from_2024-06-10_21-02-53.png
Attachment 22: Screenshot_from_2024-06-10_21-02-04.png
Screenshot_from_2024-06-10_21-02-04.png
Attachment 23: Screenshot_from_2024-06-10_21-08-56.png
Screenshot_from_2024-06-10_21-08-56.png
Attachment 24: Screenshot_from_2024-06-10_22-57-04.png
Screenshot_from_2024-06-10_22-57-04.png
Attachment 25: Screenshot_from_2024-06-10_22-57-32.png
Screenshot_from_2024-06-10_22-57-32.png
Attachment 26: Screenshot_from_2024-06-10_22-58-10.png
Screenshot_from_2024-06-10_22-58-10.png
Attachment 27: Screenshot_from_2024-06-10_22-58-48.png
Screenshot_from_2024-06-10_22-58-48.png
Attachment 28: Screenshot_from_2024-06-10_22-59-34.png
Screenshot_from_2024-06-10_22-59-34.png
Attachment 29: Screenshot_from_2024-06-10_23-00-43.png
Screenshot_from_2024-06-10_23-00-43.png
Attachment 30: Screenshot_from_2024-06-10_23-01-18.png
Screenshot_from_2024-06-10_23-01-18.png
Attachment 31: Screenshot_from_2024-06-10_23-03-00.png
Screenshot_from_2024-06-10_23-03-00.png
  619   Sun Apr 28 16:26:47 2024 Betool Alayed16:00-00:00 Sunday 28 April

19:24 pm screenshots

and

17:18 pm screenshots

Attachment 1: Screenshot_from_2024-04-28_17-18-38.png
Screenshot_from_2024-04-28_17-18-38.png
Attachment 2: Screenshot_from_2024-04-28_17-13-13.png
Screenshot_from_2024-04-28_17-13-13.png
Attachment 3: Screenshot_from_2024-04-28_17-11-48.png
Screenshot_from_2024-04-28_17-11-48.png
Attachment 4: Screenshot_from_2024-04-28_17-10-32.png
Screenshot_from_2024-04-28_17-10-32.png
Attachment 5: Screenshot_from_2024-04-28_17-04-10.png
Screenshot_from_2024-04-28_17-04-10.png
Attachment 6: Screenshot_from_2024-04-28_19-18-42.png
Screenshot_from_2024-04-28_19-18-42.png
Attachment 7: Screenshot_from_2024-04-28_19-20-10.png
Screenshot_from_2024-04-28_19-20-10.png
Attachment 8: Screenshot_from_2024-04-28_19-22-19.png
Screenshot_from_2024-04-28_19-22-19.png
Attachment 9: Screenshot_from_2024-04-28_19-23-19.png
Screenshot_from_2024-04-28_19-23-19.png
Attachment 10: Screenshot_from_2024-04-28_19-24-40.png
Screenshot_from_2024-04-28_19-24-40.png
  657   Wed Jun 12 15:41:07 2024 Betool Alayed16:00-00:00 Wed 12 Jun 24

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13

9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20

11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27

Attachment 1: Screenshot_from_2024-06-12_17-00-31.png
Screenshot_from_2024-06-12_17-00-31.png
Attachment 2: Screenshot_from_2024-06-12_17-01-10.png
Screenshot_from_2024-06-12_17-01-10.png
Attachment 3: Screenshot_from_2024-06-12_17-02-29.png
Screenshot_from_2024-06-12_17-02-29.png
Attachment 4: Screenshot_from_2024-06-12_17-03-14.png
Screenshot_from_2024-06-12_17-03-14.png
Attachment 5: Screenshot_from_2024-06-12_17-31-47.png
Screenshot_from_2024-06-12_17-31-47.png
Attachment 6: Screenshot_from_2024-06-12_17-31-13.png
Screenshot_from_2024-06-12_17-31-13.png
Attachment 7: Screenshot_from_2024-06-12_19-03-51.png
Screenshot_from_2024-06-12_19-03-51.png
Attachment 8: Screenshot_from_2024-06-12_19-04-39.png
Screenshot_from_2024-06-12_19-04-39.png
Attachment 9: Screenshot_from_2024-06-12_19-05-34.png
Screenshot_from_2024-06-12_19-05-34.png
Attachment 10: Screenshot_from_2024-06-12_19-06-05.png
Screenshot_from_2024-06-12_19-06-05.png
Attachment 11: Screenshot_from_2024-06-12_19-07-15.png
Screenshot_from_2024-06-12_19-07-15.png
Attachment 12: Screenshot_from_2024-06-12_19-08-02.png
Screenshot_from_2024-06-12_19-08-02.png
Attachment 13: Screenshot_from_2024-06-12_19-12-15.png
Screenshot_from_2024-06-12_19-12-15.png
Attachment 14: Screenshot_from_2024-06-12_21-01-54.png
Screenshot_from_2024-06-12_21-01-54.png
Attachment 15: Screenshot_from_2024-06-12_21-03-08.png
Screenshot_from_2024-06-12_21-03-08.png
Attachment 16: Screenshot_from_2024-06-12_21-03-52.png
Screenshot_from_2024-06-12_21-03-52.png
Attachment 17: Screenshot_from_2024-06-12_21-04-48.png
Screenshot_from_2024-06-12_21-04-48.png
Attachment 18: Screenshot_from_2024-06-12_21-06-37.png
Screenshot_from_2024-06-12_21-06-37.png
Attachment 19: Screenshot_from_2024-06-12_21-07-16.png
Screenshot_from_2024-06-12_21-07-16.png
Attachment 20: Screenshot_from_2024-06-12_21-08-36.png
Screenshot_from_2024-06-12_21-08-36.png
Attachment 21: Screenshot_from_2024-06-12_22-58-59.png
Screenshot_from_2024-06-12_22-58-59.png
Attachment 22: Screenshot_from_2024-06-12_22-59-43.png
Screenshot_from_2024-06-12_22-59-43.png
Attachment 23: Screenshot_from_2024-06-12_23-01-09.png
Screenshot_from_2024-06-12_23-01-09.png
Attachment 24: Screenshot_from_2024-06-12_23-02-14.png
Screenshot_from_2024-06-12_23-02-14.png
Attachment 25: Screenshot_from_2024-06-12_23-03-09.png
Screenshot_from_2024-06-12_23-03-09.png
Attachment 26: Screenshot_from_2024-06-12_23-03-48.png
Screenshot_from_2024-06-12_23-03-48.png
Attachment 27: Screenshot_from_2024-06-12_23-05-26.png
Screenshot_from_2024-06-12_23-05-26.png
  660   Thu Jun 13 21:06:44 2024 Robert Page16:00 to 00:00 shift on 13/6/2024

Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.

However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).

 

Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.

 

Another set of sceenshots taken at around 23:00 - attachments 19 - 26.

 

 

Attachment 1: Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Attachment 2: Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Attachment 5: Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Attachment 6: Screenshot_from_2024-06-13_16-04-29.png
Screenshot_from_2024-06-13_16-04-29.png
Attachment 7: Screenshot_from_2024-06-13_17-34-30.png
Screenshot_from_2024-06-13_17-34-30.png
Attachment 8: Screenshot_from_2024-06-13_18-02-57.png
Screenshot_from_2024-06-13_18-02-57.png
Attachment 9: Screenshot_2024-06-13_at_18-01-10_ucesb.png
Screenshot_2024-06-13_at_18-01-10_ucesb.png
Attachment 10: Screenshot_2024-06-13_at_18-05-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_18-05-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 11: Screenshot_2024-06-13_at_18-05-39_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_18-05-39_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2024-06-13_at_18-06-06_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_18-06-06_Statistics_aidas-gsi.png
Attachment 13: Screenshot_2024-06-13_at_18-20-26_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_18-20-26_AIDA_-_Grafana.png
Attachment 14: Screenshot_2024-06-13_at_20-55-49_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_20-55-49_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2024-06-13_at_20-56-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_20-56-12_Spectrum_Browser_aidas-gsi.png
Attachment 16: Screenshot_2024-06-13_at_20-56-34_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_20-56-34_Temperature_and_status_scan_aidas-gsi.png
Attachment 17: Screenshot_2024-06-13_at_20-57-39_ucesb.png
Screenshot_2024-06-13_at_20-57-39_ucesb.png
Attachment 18: Screenshot_from_2024-06-13_20-55-04.png
Screenshot_from_2024-06-13_20-55-04.png
Attachment 19: Screenshot_from_2024-06-13_22-52-59.png
Screenshot_from_2024-06-13_22-52-59.png
Attachment 20: Screenshot_2024-06-13_at_22-53-37_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_22-53-37_Statistics_aidas-gsi.png
Attachment 21: Screenshot_2024-06-13_at_22-54-01_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_22-54-01_Spectrum_Browser_aidas-gsi.png
Attachment 22: Screenshot_2024-06-13_at_22-54-24_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_22-54-24_Temperature_and_status_scan_aidas-gsi.png
Attachment 23: Screenshot_2024-06-13_at_22-55-49_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_22-55-49_NewMerger_Control_aidas-gsi.png
Attachment 24: Screenshot_2024-06-13_at_22-56-12_ucesb.png
Screenshot_2024-06-13_at_22-56-12_ucesb.png
Attachment 25: Screenshot_2024-06-13_at_22-56-26_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_22-56-26_AIDA_-_Grafana.png
Attachment 26: Screenshot_2024-06-13_at_22-55-18_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_22-55-18_Tape_Service_(Expert)_aidas-gsi.png
  606   Thu Apr 25 15:02:50 2024 Marc16:00 - 0:00 Thursday 25 April
16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

 

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

Attachment 1: Screenshot_from_2024-04-25_16-19-32.png
Screenshot_from_2024-04-25_16-19-32.png
Attachment 2: Screenshot_from_2024-04-25_16-45-19.png
Screenshot_from_2024-04-25_16-45-19.png
Attachment 3: Screenshot_from_2024-04-25_16-46-12.png
Screenshot_from_2024-04-25_16-46-12.png
Attachment 4: Screenshot_from_2024-04-25_16-46-41.png
Screenshot_from_2024-04-25_16-46-41.png
Attachment 5: Screenshot_from_2024-04-25_16-47-00.png
Screenshot_from_2024-04-25_16-47-00.png
Attachment 6: Screenshot_from_2024-04-25_16-52-18.png
Screenshot_from_2024-04-25_16-52-18.png
Attachment 7: Screenshot_from_2024-04-25_16-52-50.png
Screenshot_from_2024-04-25_16-52-50.png
Attachment 8: Screenshot_from_2024-04-25_17-23-36.png
Screenshot_from_2024-04-25_17-23-36.png
  613   Sat Apr 27 15:13:06 2024 Norah16:00 - 00:00 Saturday 27 April

14:50 Checks

Leakage current  is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1

It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.

18:15 Checks

Everything appears to be going smoothly 

Grafana - DSSSD bias  and leakage current - attachments 2 and 3

FEE64 temperatures  - It appears okay, nothing strange , attachment 4

ADC data item stats - attachment 5

Per FEE64 Rate spectra - attachment 6

 ucesb - attachment 7

Merger Link Data Rates  - attachment 8

 

18:16

The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.

18:20  it came back to work normally

............................

20:00 Checks

Nothing new to report. All is well.

20:33 Aida02 showed as not reading - attachment 10

20:34 it resumed reading and returned to normal operation.

21:08 Aida02 was  not reading - attachment 11 , had to emailed Tom to fix it .

The shift crew have restarted ucesb and AIDA  is now showing OK - attachment 12

so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.

ADC data item stats and Merger etc  appear to be going smoothly - attachments 13 and 14

...........................................

22:00 Checks

Still running smoothly.

DSSD bias and  leakage current - ok

FEE64 temperatures - ok

Per FEE64 Rate spectra - ok

ADC data item stats - ok

Ucesb - ok

.................................

23:00 Checks

 Nothing new to report. Everything is going well , and I have captured screenshots of each one

Leakage currents dropped a bit See attachment 15

.............................

23:50 Checks

All seems smooth.

Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
Screenshot_from_2024-04-27_16-54-13.png
Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
Screenshot_from_2024-04-27_18-15-47.png
Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
Screenshot_from_2024-04-27_18-09-13.png
Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
Screenshot_from_2024-04-27_18-07-50.png
Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
Screenshot_from_2024-04-27_18-06-46.png
Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
Screenshot_from_2024-04-27_18-07-19.png
Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
Screenshot_from_2024-04-27_18-08-53.png
Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
Screenshot_from_2024-04-27_18-09-57.png
Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
Screenshot_from_2024-04-27_19-18-03.png
Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
Screenshot_from_2024-04-27_20-33-01.png
Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
Screenshot_from_2024-04-27_20-52-28.png
Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
Screenshot_from_2024-04-27_21-22-51.png
Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
Screenshot_from_2024-04-27_21-44-02.png
Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
Screenshot_from_2024-04-27_21-45-02.png
Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
Screenshot_from_2024-04-27_23-26-06.png
  610   Fri Apr 26 15:34:00 2024 RDP16:00 - 00:00 Friday 26 April

This appeared in the system log:

Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

These messages seem to appear sporadically.

18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?

18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.

aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters

Thanks Nic!

22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)

22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.

 

Attachment 1: Screenshot_2024-04-26_at_16-40-03_ucesb.png
Screenshot_2024-04-26_at_16-40-03_ucesb.png
Attachment 2: Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
Attachment 3: Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
Attachment 4: Screenshot_2024-04-26_at_19-18-28_ucesb.png
Screenshot_2024-04-26_at_19-18-28_ucesb.png
Attachment 5: Screenshot_2024-04-26_at_22-12-06_ucesb.png
Screenshot_2024-04-26_at_22-12-06_ucesb.png
Attachment 6: Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
  645   Sun Jun 9 16:38:30 2024 Norah , Muneerah, JB16:0-00:00 9 June 2024

AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.

17:00

DSSSD bias & leakage current - attachment 2

FEE64 temperatures OK - attachment 3

Statistics  attachment 4

 

17:39

Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.

23:19 Flange removed. Starting to take beam.

 

 

 

 

Attachment 1: Screenshot_from_2024-06-09_17-24-13.png
Screenshot_from_2024-06-09_17-24-13.png
Attachment 2: Screenshot_from_2024-06-09_18-00-27.png
Screenshot_from_2024-06-09_18-00-27.png
Attachment 3: Screenshot_from_2024-06-09_18-01-49.png
Screenshot_from_2024-06-09_18-01-49.png
Attachment 4: Screenshot_from_2024-06-09_18-01-23.png
Screenshot_from_2024-06-09_18-01-23.png
  616   Sun Apr 28 15:09:34 2024 Betool Alayed16:0-00:00 28 April 2024

16:00 aida04 not producing data

          power cycle all FEE64s to recover DAQ

16.20 analysis data file R21_668

         max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%

         no timewarps

         HEC data item rate 1.9kHz

 

Attachment 1: R21_668.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256513573 ( 1160512.7 Hz)
        Other data format:    5406427 (   24459.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1079 (       4.9 Hz)
                               RESUME:       1078 (       4.9 Hz)
                              SYNC100:      32824 (     148.5 Hz)
                              WR48-63:      32824 (     148.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5338622 (   24152.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     440937 (    1994.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      221.035 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1               19.736                0.000
  2                0.053                0.000
  3               36.607                0.000
  4                0.854                0.000
  5                4.747                0.000
  6                0.000                0.000
  7                1.845                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.661                0.000
 11                0.020                0.000
 12                0.004                0.000
 13                0.000                0.000
 14                0.003                0.000
 15                1.247                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8138410       8624          0          0         12         12        985        985          0       6630          0      38979
  1   27142829     245404          0          0        242        242       3461       3461          0     237998          0      64179
  2   11096772    2350080          0          0          8          8       1768       1768          0    2346528          0      26862
  3   37888832    1835283          0          0        304        303       4762       4762          0    1825152          0      18340
  4    9329455     413600          0          0         28         28       1212       1212          0     411120          0      29227
  5   23846951     517930          0          0        167        167       3201       3201          0     511194          0      24385
  6    5635241       1420          0          0          0          0        710        710          0          0          0       8053
  7   40074524      10318          0          0         93         93       5066       5066          0          0          0     120554
  8    3088839        754          0          0          0          0        377        377          0          0          0      14672
  9    5102920       1254          0          0          0          0        627        627          0          0          0       6080
 10   25963054       6756          0          0        140        140       3238       3238          0          0          0       2640
 11    5980678       1458          0          0          5          5        724        724          0          0          0      52714
 12    5769255       1548          0          0          2          2        772        772          0          0          0       8398
 13    3930590        926          0          0          0          0        463        463          0          0          0       9544
 14    5399163       1328          0          0          1          1        663        663          0          0          0       5692
 15   38126060       9744          0          0         77         77       4795       4795          0          0          0      10618
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.004s (  941.068 blocks/s,  58.817 Mb/s)
  620   Sun Apr 28 20:35:57 2024 Betool Alayed16:0-00:00 28 April 2024

21:30 screenshots

and

23:24 screenshots

Attachment 1: Screenshot_from_2024-04-28_21-30-19.png
Screenshot_from_2024-04-28_21-30-19.png
Attachment 2: Screenshot_from_2024-04-28_21-31-11.png
Screenshot_from_2024-04-28_21-31-11.png
Attachment 3: Screenshot_from_2024-04-28_21-32-07.png
Screenshot_from_2024-04-28_21-32-07.png
Attachment 4: Screenshot_from_2024-04-28_21-32-51.png
Screenshot_from_2024-04-28_21-32-51.png
Attachment 5: Screenshot_from_2024-04-28_21-35-10.png
Screenshot_from_2024-04-28_21-35-10.png
Attachment 6: Screenshot_from_2024-04-28_23-35-25.png
Screenshot_from_2024-04-28_23-35-25.png
Attachment 7: Screenshot_from_2024-04-28_23-34-31.png
Screenshot_from_2024-04-28_23-34-31.png
Attachment 8: Screenshot_from_2024-04-28_23-33-42.png
Screenshot_from_2024-04-28_23-33-42.png
Attachment 9: Screenshot_from_2024-04-28_23-32-28.png
Screenshot_from_2024-04-28_23-32-28.png
Attachment 10: Screenshot_from_2024-04-28_23-31-51.png
Screenshot_from_2024-04-28_23-31-51.png
  588   Mon Apr 22 15:17:07 2024 DSJ16.00-0.00 22/04/24

First checks of shift. All looks ok.

Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.

Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)

 

checks at 16.00 - merger has crashed - aida01 dropped out  - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09 

Reset 01, restarted DAQ, and restarted as R16.

16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.

Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms

RUN17 STARTED 16.51|

 HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.

 

20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift

21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps

22.48 - things apear ok. (Attachments 23-26) 

Attachment 1: Screenshot_from_2024-04-22_16-16-30.png
Screenshot_from_2024-04-22_16-16-30.png
Attachment 2: Screenshot_from_2024-04-22_16-14-31.png
Screenshot_from_2024-04-22_16-14-31.png
Attachment 3: Screenshot_from_2024-04-22_16-09-16.png
Screenshot_from_2024-04-22_16-09-16.png
Attachment 4: Screenshot_from_2024-04-22_16-05-22.png
Screenshot_from_2024-04-22_16-05-22.png
Attachment 5: Screenshot_from_2024-04-22_16-33-34.png
Screenshot_from_2024-04-22_16-33-34.png
Attachment 6: Screenshot_from_2024-04-22_16-32-54.png
Screenshot_from_2024-04-22_16-32-54.png
Attachment 7: Screenshot_from_2024-04-22_16-32-04.png
Screenshot_from_2024-04-22_16-32-04.png
Attachment 8: Screenshot_from_2024-04-22_16-31-03.png
Screenshot_from_2024-04-22_16-31-03.png
Attachment 9: Screenshot_from_2024-04-22_16-30-00.png
Screenshot_from_2024-04-22_16-30-00.png
Attachment 10: Screenshot_from_2024-04-22_16-29-03.png
Screenshot_from_2024-04-22_16-29-03.png
Attachment 11: Screenshot_from_2024-04-22_16-27-01.png
Screenshot_from_2024-04-22_16-27-01.png
Attachment 12: Screenshot_from_2024-04-22_16-25-30.png
Screenshot_from_2024-04-22_16-25-30.png
Attachment 13: Screenshot_from_2024-04-22_19-00-04.png
Screenshot_from_2024-04-22_19-00-04.png
Attachment 14: Screenshot_from_2024-04-22_18-59-25.png
Screenshot_from_2024-04-22_18-59-25.png
Attachment 15: Screenshot_from_2024-04-22_18-58-21.png
Screenshot_from_2024-04-22_18-58-21.png
Attachment 16: Screenshot_from_2024-04-22_18-56-34.png
Screenshot_from_2024-04-22_18-56-34.png
Attachment 17: Screenshot_from_2024-04-22_20-46-50.png
Screenshot_from_2024-04-22_20-46-50.png
Attachment 18: Screenshot_from_2024-04-22_20-45-57.png
Screenshot_from_2024-04-22_20-45-57.png
Attachment 19: Screenshot_from_2024-04-22_20-45-10.png
Screenshot_from_2024-04-22_20-45-10.png
Attachment 20: Screenshot_from_2024-04-22_20-44-41.png
Screenshot_from_2024-04-22_20-44-41.png
Attachment 21: Screenshot_from_2024-04-22_22-00-30.png
Screenshot_from_2024-04-22_22-00-30.png
Attachment 22: Screenshot_from_2024-04-22_22-53-11.png
Screenshot_from_2024-04-22_22-53-11.png
Attachment 23: Screenshot_from_2024-04-22_22-52-19.png
Screenshot_from_2024-04-22_22-52-19.png
Attachment 24: Screenshot_from_2024-04-22_22-50-52.png
Screenshot_from_2024-04-22_22-50-52.png
Attachment 25: Screenshot_from_2024-04-22_22-49-07.png
Screenshot_from_2024-04-22_22-49-07.png
Attachment 26: Screenshot_from_2024-04-22_22-48-31.png
Screenshot_from_2024-04-22_22-48-31.png
  231   Fri Apr 16 15:52:08 2021 DSJ 16 April 16.00 shift
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :     25     10      4      3      1      2      2      3      3      3      6   : 36500
aida03 :     27      8      5      2      3      4      3      3      2      3      6   : 36092
aida04 :     30      4      4      5      0      2      4      2      3      3      6   : 36472
aida05 :     19      5      7      1      3      2      2      2      2      4      6   : 37060
aida06 :     18     13      1      2      1      3      2      3      3      3      6   : 36544
aida07 :     24      7      3      2      2      3      1      3      2      4      6   : 37384
aida08 :     15     11      2      5      2      4      1      4      3      3      6   : 37076
aida09 :      7      3      4      1      5      6      5      4      3      2      6   : 36308
aida10 :      8      3     13      4      3      3      1      4      2      3      6   : 36040
aida11 :     18     11     17      7      4      4      4      3      2      3      6   : 36752
aida12 :     12      7      7      7      4      4      2      3      2      3      6   : 36024



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021



17:27.  Checked system stats, temps, leakage etc. looks ok 


18:49. Tape server writing to disk /NULL/R21_9+
       /TapeData points to /media/SecondDrive/ - 3.6Tb free

19.15. checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9     16      2      2      2      1      3      3      3      6   : 36424
aida02 :      1      3      5      4      1      3      3      2      3      3      6   : 36268
aida03 :     24     13     11      6      3      3      2      4      2      3      6   : 36472
aida04 :      5      4      4      4      3      3      3      2      3      3      6   : 36404
aida05 :     17      8      4      2      2      2      2      2      2      4      6   : 36996
aida06 :     26     11      3      2      0      3      2      3      3      3      6   : 36528
aida07 :     24      7      2      1      1      1      2      3      2      4      6   : 37272
aida08 :     13     12      6      3      3      4      1      4      3      3      6   : 37140
aida09 :     22     15     19      2      6      5      4      4      3      2      6   : 36416
aida10 :     16     11      6      4      3      3      1      4      2      3      6   : 36024
aida11 :     12      4      5      4      2      3      3      3      2      3      6   : 35872
aida12 :     18     12      9      4      4      4      2      3      2      3      6   : 36024


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


19.53.  Checked system stats, temps, leakage etc. looks ok 

20:30.  Checked system stats, temps, leakage etc. looks ok 

20:59.  Checked system stats, temps, leakage etc. looks ok 



21:31 System checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      9     17      3      2      2      1      3      3      3      6   : 36492
aida02 :      5      1      1      6      0      2      2      3      3      3      6   : 36332
aida03 :      2     10     11      6      4      4      3      3      2      3      6   : 36296
aida04 :     27     12      3      4      0      2      3      2      3      3      6   : 36220
aida05 :     19      9      7      2      1      2      2      2      2      4      6   : 36996
aida06 :     28      5      0      4      0      3      1      3      3      3      6   : 36248
aida07 :     19     13     10      6      2      2      1      3      2      4      6   : 37524
aida08 :     18      9      6      0      1      4      1      3      3      3      6   : 36400
aida09 :     17     19     12      3      6      5      4      4      3      2      6   : 36348
aida10 :     23      9      7      4      2      3      1      4      2      3      6   : 35988
aida11 :     19     10      3      3      2      3      3      3      2      3      6   : 35884
aida12 :     12      6      0      3      2      4      2      3      2      3      6   : 35648


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


22.06.  Checked system stats, temps, leakage etc. looks ok 

22.18CEST BNC PB-5 amplitude changed from 1V to 2V
          fioler NULL/R21_67

22.23CEST All histograms zero'd

22.43.  Checked system stats, temps, leakage etc. looks ok 
23.21.  Checked system stats, temps, leakage etc. looks ok 


23.46 system checks
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     17     12      7      0      3      3      2      4      2      3      6   : 36180
aida02 :     35      3      5      2      0      2      1      3      3      3      6   : 36148
aida03 :     22      6     10      2      1      3      2      4      2      3      6   : 36136
aida04 :     27      9      3      5      2      4      3      3      2      3      6   : 36100
aida05 :     20      9      3      3      2      2      2      2      2      4      6   : 37032
aida06 :     25     11      1      2      0      3      1      3      3      3      6   : 36236
aida07 :     20      6      6      2      3      3      2      4      1      4      6   : 37216
aida08 :     25     10      0      2      1      3      1      3      3      3      6   : 36276
aida09 :     10      4      3      4      4      5      4      4      3      2      6   : 35960
aida10 :     26      9      3      0      1      3      1      4      2      3      6   : 35744
aida11 :      2      3      4      2      2      3      3      3      2      3      6   : 35744
aida12 :      4      9      7      2      2      4      2      3      2      3      6   : 35720
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021
Attachment 1: Screenshot_2021-04-16_Statistics_aidas-gsi.png
Screenshot_2021-04-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 10.png
10.png
Attachment 4: Screenshot_2021-04-16_Statistics_aidas-gsi(1).png
Screenshot_2021-04-16_Statistics_aidas-gsi(1).png
Attachment 5: 11.png
11.png
Attachment 6: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-16_Statistics_aidas-gsi(2).png
Screenshot_2021-04-16_Statistics_aidas-gsi(2).png
Attachment 8: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 9: 12.png
12.png
Attachment 10: Screenshot_2021-04-16_Statistics_aidas-gsi(3).png
Screenshot_2021-04-16_Statistics_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 12: 13.png
13.png
  197   Sat Mar 13 23:08:08 2021 CA14th March 00:00-08:00
00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC
Attachment 1: Screenshot_from_2021-03-13_23-14-33.png
Screenshot_from_2021-03-13_23-14-33.png
Attachment 2: Screenshot_from_2021-03-13_23-14-57.png
Screenshot_from_2021-03-13_23-14-57.png
Attachment 3: Screenshot_from_2021-03-13_23-15-12.png
Screenshot_from_2021-03-13_23-15-12.png
Attachment 4: Screenshot_from_2021-03-14_01-22-56.png
Screenshot_from_2021-03-14_01-22-56.png
Attachment 5: Screenshot_from_2021-03-14_01-29-35.png
Screenshot_from_2021-03-14_01-29-35.png
Attachment 6: Screenshot_from_2021-03-14_01-29-52.png
Screenshot_from_2021-03-14_01-29-52.png
Attachment 7: Screenshot_from_2021-03-14_01-40-57.png
Screenshot_from_2021-03-14_01-40-57.png
Attachment 8: Screenshot_from_2021-03-14_02-16-28.png
Screenshot_from_2021-03-14_02-16-28.png
Attachment 9: Screenshot_from_2021-03-14_02-17-01.png
Screenshot_from_2021-03-14_02-17-01.png
Attachment 10: Screenshot_from_2021-03-14_02-17-32.png
Screenshot_from_2021-03-14_02-17-32.png
Attachment 11: Screenshot_from_2021-03-14_02-29-29.png
Screenshot_from_2021-03-14_02-29-29.png
Attachment 12: R56_35
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261409816 ( 1907248.6 Hz)
        Other data format:     510184 (    3722.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2488 (      18.2 Hz)
                               RESUME:       2487 (      18.1 Hz)
                              SYNC100:        511 (       3.7 Hz)
                              WR48-63:        511 (       3.7 Hz)
                           FEE64 disc:     503410 (    3672.9 Hz)
                             MBS info:        777 (       5.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      60911 (     444.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      137.061 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.725              132.339
  2                3.132              135.291
  3                6.180              132.876
  4               12.844              128.581
  5                8.386              132.070
  6                1.216              132.607
  7                1.093              115.159
  8               50.447              129.654
  9               19.940              134.755
 10                0.568              132.339
 11               51.844              133.681
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22917695      26526          0          0        878        878         65         65      23863        777          0       8590
  1   26593258      35084          0          0         13         13         51         51      34956          0          0      10875
  2   25634005     337148          0          0         16         16         76         76     336964          0          0       6926
  3   28548879       3343          0          0         34         34         27         27       3221          0          0       1214
  4    7607641      18967          0          0        663        663         14         14      17613          0          0       6303
  5   19366719      25239          0          0        305        304         24         24      24582          0          0       8037
  6   18601227      17453          0          0          3          3         18         18      17411          0          0       6563
  7   10076045       2592          0          0          4          4         24         24       2536          0          0        908
  8   30026924       8042          0          0        217        216         34         34       7541          0          0       2274
  9   30858440      18084          0          0        113        113         88         88      17682          0          0       4590
 10    9946552      15820          0          0          3          3         46         46      15722          0          0       4264
 11   31232431       1886          0          0        239        240         44         44       1319          0          0        367
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:13611.407s (    2.351 blocks/s,   0.147 Mb/s)
Attachment 13: Screenshot_from_2021-03-14_04-20-07.png
Screenshot_from_2021-03-14_04-20-07.png
Attachment 14: Screenshot_from_2021-03-14_04-20-34.png
Screenshot_from_2021-03-14_04-20-34.png
Attachment 15: Screenshot_from_2021-03-14_04-20-50.png
Screenshot_from_2021-03-14_04-20-50.png
Attachment 16: Screenshot_from_2021-03-14_04-25-41.png
Screenshot_from_2021-03-14_04-25-41.png
Attachment 17: Screenshot_from_2021-03-14_04-32-03.png
Screenshot_from_2021-03-14_04-32-03.png
Attachment 18: Screenshot_from_2021-03-14_04-32-32.png
Screenshot_from_2021-03-14_04-32-32.png
Attachment 19: Screenshot_from_2021-03-14_04-33-14.png
Screenshot_from_2021-03-14_04-33-14.png
Attachment 20: Screenshot_from_2021-03-14_06-10-39.png
Screenshot_from_2021-03-14_06-10-39.png
Attachment 21: Screenshot_from_2021-03-14_06-11-19.png
Screenshot_from_2021-03-14_06-11-19.png
Attachment 22: Screenshot_from_2021-03-14_06-11-38_-_1.png
Screenshot_from_2021-03-14_06-11-38_-_1.png
  597   Tue Apr 23 13:15:43 2024 PP14:00 checks

All looks good.

Screenshots attached

Attachment 1: Screenshot_from_2024-04-23_14-11-33.png
Screenshot_from_2024-04-23_14-11-33.png
Attachment 2: Screenshot_from_2024-04-23_14-04-26.png
Screenshot_from_2024-04-23_14-04-26.png
Attachment 3: Screenshot_from_2024-04-23_14-04-44.png
Screenshot_from_2024-04-23_14-04-44.png
Attachment 4: Screenshot_from_2024-04-23_14-06-02.png
Screenshot_from_2024-04-23_14-06-02.png
Attachment 5: Screenshot_from_2024-04-23_14-07-07.png
Screenshot_from_2024-04-23_14-07-07.png
Attachment 6: Screenshot_from_2024-04-23_14-09-26.png
Screenshot_from_2024-04-23_14-09-26.png
Attachment 7: Screenshot_from_2024-04-23_14-10-55.png
Screenshot_from_2024-04-23_14-10-55.png
Attachment 8: Screenshot_from_2024-04-23_14-11-16.png
Screenshot_from_2024-04-23_14-11-16.png
  96   Wed Nov 13 09:10:37 2019 CA, NH, OH13th November 2019
10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   
Attachment 1: 191113_pulserpeaksOdd.png
191113_pulserpeaksOdd.png
Attachment 2: 191113_pulserpeaksEven.png
191113_pulserpeaksEven.png
Attachment 3: 191113_goodevents.png
191113_goodevents.png
Attachment 4: 191113_bias.png
191113_bias.png
ELOG V3.1.3-7933898