AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 5 of 36  ELOG logo
ID Date Author Subjectdown
  307   Wed May 12 06:59:55 2021 OH, RDPWed 12 May 08:00-16:00
08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.

      They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
      They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID

08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2

09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
      Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
      Data rate 16.3MB/s
      ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.

09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage

09:47 Beam back turned off no storage. On file R1_10

11:04 Now that we have implants on the detector we get a better look at the strips
      From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
      This adaptor board corresponds to FEE11

12:17 Leakage currents screenshot 512.png

13:30 Beam off. Expected back at 15:00...

14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
      They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
      They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
      Rates are as before. No better, no worse.

15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...
  306   Wed May 12 02:07:59 2021 MA, BAWed 12 May 00:00-08:00

03:12 System Check

attachment 1 : Current

attachment 2 : Temperature

attachment 3 : Rates

 

   
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

   
Calibration test result: Passed 16, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida13 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      5      7      2      1      2      2      3      3      3      6   : 36464
aida02 :      9      8      3      3      1      4      1      2      4      3      6   : 36916
aida03 :      5      2      6      2      0      3      2      3      3      3      6   : 36420
aida04 :      6      5      2      3      4      4      2      3      3      3      6   : 36800
aida05 :     17      6      6      2      2      4      1      3      2      4      6   : 37524
aida06 :      7     12      3      4      3      3      1      3      3      3      6   : 36460
aida07 :     17     11      5      1      3      3      3      2      3      3      6   : 36428
aida08 :      3      5      0      1      4      2      2      4      2      3      6   : 35924
aida09 :     27      6      4      2      0      2      2      2      3      3      6   : 35868
aida10 :     16     11      8      0      2      2      1      3      2      4      6   : 37272
aida11 :     15      2      2      3      1      4      3      4      2      3      6   : 36364
aida12 :      1      6      4      3      1      3      2      4      2      3      6   : 35988
aida13 :     22     14     10      2      4      2      2      4      2      3      6   : 36264
aida14 :     26     10      4      3      2      1      1      3      3      3      6   : 36184
aida15 :     14      2      3      2      2      4      1      2      3      3      6   : 35896
aida16 :      7      5      4      0      2      4      2      3      2      3      6   : 35588

06:10 DSSD1 rate high!

Attached 4

Called OH and he woke up to fix it :)

It was a problem with one of the ASIC according to what he says:

One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35

 

07:03 System check

attachment 5 Spectrum rate

attachment 6 Voltages

attachment 7 Rates

attachment 8 Temperature

System wide clock are all ok except

White rabbit

 Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA

 Base         Current         Difference
aida13 fault      0xa :      0x14 :      10  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

  309   Wed May 12 18:53:38 2021 JSWed 12 20:00-00:00
19:51 Shift change

Stats ok
ucesb ok, Max implants ~1500 Hz 

20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)

20:51 
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.

20:59 

Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36ce : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0xb6 : 	 172  
FPGA Timestamp error counter test result: Passed 15, Failed 1

--- both errors above same module as earlier shift
	 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      3      2      0      1      3      3      2      3      3      6   : 36092
aida02 :      6      4      4      2      2      3      1      3      3      3      6   : 36280
aida03 :      6      3      2      0      0      3      3      2      3      3      6   : 36048
aida04 :     10      3      1      1      3      3      1      3      3      3      6   : 36272
aida05 :     19      8      2      0      2      3      1      3      3      3      6   : 36268
aida06 :      2      2      0      2      3      4      2      2      3      3      6   : 36120
aida07 :      6      7      3      0      3      4      3      2      3      3      6   : 36416
aida08 :      7      5      4      0      3      2      2      3      3      3      6   : 36420
aida09 :     21      6      3      2      2      4      2      3      2      3      6   : 35700
aida10 :     13      3      1      1      3      3      1      4      2      3      6   : 35772
aida11 :     25      6      4      2      1      4      2      4      2      3      6   : 36180
aida12 :      0      4      0      3      2      3      3      3      2      3      6   : 35712
aida13 :     18      5      2      3      2      3      2      4      2      3      6   : 36080
aida14 :      2      1      1      1      2      3      2      4      2      3      6   : 35904
aida15 :      0      1      1      3      0      4      2      2      3      3      6   : 35960
aida16 :      4      5      3      2      3      4      1      3      2      3      6   : 35432

ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4

21:36
ucesb ok
Stats ok 

22:01
ucesb ok 1500 Hz peak implant
Stats ok 
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.

22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler

23:40 FRS magnet issue, possibly power, no beam

00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah
  308   Wed May 12 15:03:10 2021 CB, OHWed 12 16:00-20:00
16:00 Shift change

Statistics OK (attach 1)
Temps OK (attach 2)
Bias OK (attach 3)

System wide checks
Clock, ADC - OK

WR
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36cb : 	 1  
White Rabbit error counter test result: Passed 15, Failed 1


FPGA

			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0x4b : 	 65  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      5      6      1      1      2      2      3      3      3      6   : 36388
aida02 :     21     10      2      2      2      3      2      2      3      3      6   : 36100
aida03 :     25      5      3      1      0      3      3      2      3      3      6   : 36188
aida04 :      5      2      0      3      2      3      2      3      3      3      6   : 36484
aida05 :     16      6      3      1      2      4      1      3      2      4      6   : 37440
aida06 :      2      2      2      2      2      3      1      3      3      3      6   : 36216
aida07 :     16      9      3      0      3      4      3      2      3      3      6   : 36472
aida08 :      0      7      1      1      2      2      3      3      2      3      6   : 35560
aida09 :     15      7      4      2      2      4      2      3      2      3      6   : 35700
aida10 :      3      7      3      4      1      3      1      4      1      4      6   : 36788
aida11 :      5      1      3      1      1      4      3      3      2      3      6   : 35756
aida12 :      5      5      0      3      2      3      2      4      2      3      6   : 35996
aida13 :      2      5      1      2      2      4      3      3      2      3      6   : 35840
aida14 :     19      5      3      1      1      2      1      3      3      3      6   : 36100
aida15 :     20      6      6      2      1      3      2      2      3      3      6   : 36064
aida16 :      7      9      4      1      3      4      2      3      2      3      6   : 35716

File size change
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 10 16:47:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


16:20 Beam is back. Currently on File R1_200
16:43 Beam is being optimised.

17:00 UCESB OK
Stats OK (attach 4)
Temps OK
Bias OK (attach 5)
System-wide checks as before

17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK

17:52 Started compressing R1_1*

18:00 UCESB ok (attach 5)
Stats OK (attach 6)
Temps OK
Bias OK (attach 7)
System wide checks as before

19:00 UCESB ok (as before)
Stats OK (attach 8)
Temps OK
Bias OK (attach 9)
Merger OK (attach 10)

19:32 UCESB ok
Stats OK (attach 11)
Temps OK (attach 12)
Bias OK (attach 13)
System wide checks as before

19:51 Shift handed over to James S
  162   Tue Nov 24 13:10:42 2020 NHWaveforms with frame grounded
AIDA Frame is now grounded to platform (will add photo later)

Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now)
  100   Fri Nov 22 08:55:31 2019 NHWaveforms
AIDA Waveforms immediately after beam at 21.11.19 8am or so.
  101   Mon Nov 25 13:25:51 2019 NHWaveforms
> AIDA Waveforms immediately after beam at 21.11.19 8am or so.

Update 25.11.19 with all FEE64 waveforms added

Things to note:
aida01, aida05, aida09 are all on TOP - show VHF noise, esp. aida01?
TOP is where HV is connected (-ve core)

Everything noisy in general though
  578   Thu Apr 18 13:59:55 2024 TD, JBWR timestamp OK, readout time not OK
WR timestamps appear to be OK but readout times fort aida03, aida04 and aida09 appear to be out of sequence. Attachment 1-3.

Reseated HDMI cable of aida09 and performed complete powercycle. Check the MACB HDMI port (appears to be loose) - still no change aida09 not collecting WR timestamps. Attachment 4-5.
  699   Thu Feb 20 17:06:42 2025 JB, CC, TD,MPWR error for aida 13,14,15,16

WR timestamp errors resolved after reseating the HDMI cables to the MACB

  74   Thu Oct 31 14:27:06 2019 NHWR Timestamps
All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.

HDMI cables in aida09 checked and good.
  78   Thu Oct 31 19:02:42 2019 PatrickWR Timestamps
> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
  26   Sun Feb 3 16:01:49 2019 TDUpdated analyses GSIJan19/R1_0 - R10_0
Summary statistics for WR test data files R1_0, R2_0, R3_0, R4_0, R7_0 and R10_0 (attachments 1-6)

Latest version of WR TDR data analyser (attachment 7)
  551   Tue Mar 26 18:56:03 2024 TDUSB-controlled ac mains relay interlock box - wiring
Sensor ( 4 pins )

Red  +24V
Blue   0V
Yellow } contact 
Green  } closure in ( short these two wires together for logic 1 )

N.B. yellow and green wires of unused inputs must be connected together


Output ( 3 pins ) to USB-controlled ac mains relay

Red    NC ( normally closed - with respect to COM with no power applied, single pole double throw relay out )
Blue   NO ( normally open - with respect to COM with no power applied, single pole double throw relay out )
Green COM
  182   Mon Mar 8 22:51:47 2021 OHTuesday 9th March
18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)

at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
          Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
          Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message

01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event

Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.


02:00 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

03:10 Problem with accelerator will be no beam for ~30 minutes


03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
      There are now also errors in all FEEs for WR events
	
		 Base 		Current 	Difference
aida01 fault 	 0xc45f : 	 0xc46a : 	 11  
aida02 fault 	 0xd250 : 	 0xd25b : 	 11  
aida03 fault 	 0x27b2 : 	 0x27bd : 	 11  
aida04 fault 	 0x714b : 	 0x7156 : 	 11  
aida05 fault 	 0x74f5 : 	 0x7500 : 	 11  
aida06 fault 	 0x558a : 	 0x5595 : 	 11  
aida07 fault 	 0x8d20 : 	 0x8d2b : 	 11  
aida08 fault 	 0x8131 : 	 0x813c : 	 11  
aida09 fault 	 0xf412 : 	 0xf41d : 	 11  
aida10 fault 	 0x82c2 : 	 0x82cd : 	 11  
aida11 fault 	 0x2845 : 	 0x2850 : 	 11  
aida12 fault 	 0x1bf6 : 	 0x1c01 : 	 11  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Also errors in FPGA
			 Base 		Current 		Difference
aida03 fault 	 0x0 : 	 0x1 : 	 1  
aida04 fault 	 0x0 : 	 0x1 : 	 1  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A8D22 3A5FDE66 , WR/10=>   23DDAE9D2A32FD7, Readout Time =>   23DDAE9D3C9C000 
aida02 : White Rabbit=>  166A8D22 4C6C3AEB , WR/10=>   23DDAE9D4713917, Readout Time =>   23DDAE9D5B94000 
aida03 : White Rabbit=>  166A8D22 5FB33A3C , WR/10=>   23DDAE9D65EB906, Readout Time =>   23DDAE9D769C000 
aida04 : White Rabbit=>  166A8D22 6FBA29DB , WR/10=>   23DDAE9D7F9042F, Readout Time =>   23DDAE9D8D14000 
aida05 : White Rabbit=>  166A8D22 7F71248D , WR/10=>   23DDAE9D98B5074, Readout Time =>   23DDAE9DA9D8000 
aida06 : White Rabbit=>  166A8D22 8F78B071 , WR/10=>   23DDAE9DB25AB3E, Readout Time =>   23DDAE9DC280000 
aida07 : White Rabbit=>  166A8D22 9F5A64B7 , WR/10=>   23DDAE9DCBC3D45, Readout Time =>   23DDAE9DDFF0000 
aida08 : White Rabbit=>  166A8D22 B1E91092 , WR/10=>   23DDAE9DE974E75, Readout Time =>   23DDAE9DFA08000 
aida09 : White Rabbit=>  166A8D22 C150D9ED , WR/10=>   23DDAE9E021AF64, Readout Time =>   23DDAE9E0FD8000 
aida10 : White Rabbit=>  166A8D22 D02BA583 , WR/10=>   23DDAE9E19DF6F3, Readout Time =>   23DDAE9E2D50000 
aida11 : White Rabbit=>  166A8D22 E287CFC2 , WR/10=>   23DDAE9E373FB2D, Readout Time =>   23DDAE9E4720000 
aida12 : White Rabbit=>  166A8D22 F2D4C911 , WR/10=>   23DDAE9E515474E, Readout Time =>   23DDAE9E1BCC000 

System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results

AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).

03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.

All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.

05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
      The problem was with the accelerator operators system. There was no beam.

06:20 Error in WR system wide checks. Rest all ok
      This time there is no bad merge items recorded in the merger terminal.
      There are also no timewarp events seen in ucesb which were observed with the issues at 3am.

		 Base 		Current 	Difference
aida01 fault 	 0xdbe9 : 	 0xdbea : 	 1  
aida02 fault 	 0xa93d : 	 0xa93e : 	 1  
aida03 fault 	 0x8fc4 : 	 0x8fc5 : 	 1  
aida04 fault 	 0x3e56 : 	 0x3e57 : 	 1  
aida05 fault 	 0xb04c : 	 0xb04d : 	 1  
aida06 fault 	 0x5576 : 	 0x5577 : 	 1  
aida07 fault 	 0x99ff : 	 0x9a00 : 	 1  
aida08 fault 	 0x8412 : 	 0x8413 : 	 1  
aida09 fault 	 0x2457 : 	 0x2458 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A9523 11478D1D , WR/10=>   23DDBB6B4ED8E1C, Readout Time =>   23DDBB6B6148000 
aida02 : White Rabbit=>  166A9523 236AA5E8 , WR/10=>   23DDBB6B6BDDD64, Readout Time =>   23DDBB6B8164000 
aida03 : White Rabbit=>  166A9523 375F235D , WR/10=>   23DDBB6B8BCB6BC, Readout Time =>   23DDBB6B9C3C000 
aida04 : White Rabbit=>  166A9523 47B8038C , WR/10=>   23DDBB6BA5F338E, Readout Time =>   23DDBB6BA77C000 
aida05 : White Rabbit=>  166A9523 599FC116 , WR/10=>   23DDBB6BC29934F, Readout Time =>   23DDBB6BD34C000 
aida06 : White Rabbit=>  166A9523 696EB0A7 , WR/10=>   23DDBB6BDBE44DD, Readout Time =>   23DDBB6BECB8000 
aida07 : White Rabbit=>  166A9523 7982236B , WR/10=>   23DDBB6BF59D057, Readout Time =>   23DDBB6C0418000 
aida08 : White Rabbit=>  166A9523 87C85008 , WR/10=>   23DDBB6C0C73B34, Readout Time =>   23DDBB6C1C7C000 
aida09 : White Rabbit=>  166A9523 97020497 , WR/10=>   23DDBB6C24D0075, Readout Time =>   23DDBB6C34B4000 
aida10 : White Rabbit=>  166A9523 A69B171D , WR/10=>   23DDBB6C3DC4F1C, Readout Time =>   23DDBB6C50A0000 
aida11 : White Rabbit=>  166A9523 B86C8D7D , WR/10=>   23DDBB6C5A4748C, Readout Time =>   23DDBB6C6C88000 
aida12 : White Rabbit=>  166A9523 CA1DD1B2 , WR/10=>   23DDBB6C76961C5, Readout Time =>   23DDBB6C5C40000 

Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s) 
Cannot check file as was not writing to file.

      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10

06:40 There are occasional seconds of beam. According to operators it comes and then it goes again

06:45 Beam fairly stable so we are running again.
      AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19

07:00 Beam stopped again
07:02 Beam returned
      There are apparently issues with MUSIC chamber one and its resolution

07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. 
  185   Tue Mar 9 20:04:20 2021 TDTuesday 9 March
21.04 DAQ continues OK - file R20_379

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      System wide checks all OK *except*

ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

Check FPGA Timestamp Errors
Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1

If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.2M data items/s
      TapeServer OK - 45 Mb/s

21.20 no merger server error/warning messages since 16.30UTC

21.35 p+n junction strip  HEC & LEC spectra, Rate spectra common y-scale 0-30000 & 0-10
      all spectra zero'd

22.05 no beam for 2-3h whilst accelerator generator is fixed
  567   Tue Apr 9 08:37:47 2024 TDTuesday 9 April
09.36 Cooling water temperature and flow OK

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      info code 4, correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 5-10

      per FEE64 Rate spectra - attachments 11

      per 1.8.W spectra - 20us FSR - attachments 12-13

12.25 DSSSD bias & leakage current OK - attachment 16

      FEE64 temps OK - attachment 15
      *except* aida02 ASIC temp which is known to be u/s

      ADC data item stats - attachments 14

      system stable
  405   Tue Mar 8 10:26:56 2022 TD NHTuesday 8 March 2022
11.25 System wide checks OK except aida06 clock fail status 6 - attachments 1-13

      ADC data item stats - attachment 14

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage currents - attachment 16
  
      DSSSD#1 OK. DSSSD#2 leakage current has increased from c.5uA to c. 11uA overnight.


11.43 per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - attachments 17-24

      per FEE64 1.8.L spectra - attachments 25-26
          aida12 1.8.L pulser peak width 143 ch FWHM
          aida02 1.8.L pulser peak width 439 ch FWHM

      Per FEE64 rate spectra - attachment 27

12.00 Cycled DSSSD#2 bias from -100V to 0 to -100V leakage current -7.7uA but ramping slowly (c. 5nA/10s ... ?)

13.26 DSSSD#2 bias -100V leakage current -8.54uA

15.46 - Noticed that DSSSD#1 braid no longer has filter, DSSSD#1 core still does. DSSSD#2 is unfiltered on both.

      - Swap FEE64 power connectors #3 with #4, and #6 with #7, this means both n+n FEE64s are on the same paired channel

      No significant change to stats

      - Notice braid was only connected to FEEs 4 and 8, which meant beam-left bias guard ring 0v was floating. Use T to bring bias braid over to FEES 2 and 6 as well.
        Middle bias guard ring 0v is still floating

      No significant change to stats, amplitude of 100 kHz waveform in n+n waves seems attenuated though

     aida12 1.8.L pulser peak width 76 ch FWHM
     aida02 1.8.L pulser peak width 438 ch FWHM
 
     Attachment 28 - HV leakage current
     Attachment 29 - FEE ADC stats
     Attachment 30 - p+n side waveforms
     Attachment 31 - n+n side waveforms
     Attachment 32 - p+n pulser peak widths
     Attachment 33 - n+n pulser peak widths

15.55 - Reattach filter to DSSSD#1 braid

     No major change, possible improvement in waveforms 
     Attachment 34 - Stats
     Attachment 35,36 - waveforms

16.46 - Reattach LK1 jumper to FEES 2, 4, 6, 8

     Waveforms look OK, p+n rate slightly worse?
     Attachment 37 - Stats
     Attachment 38, 39 - waveforms
     Attachment 40 - Per FEE Rates
  360   Tue Jun 8 10:51:55 2021 TDTuesday 8 June
11.52 DAQ continues file S496/R32_340
      High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning

11.53 ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      3     12      8      3      3      2      2      1      4      7   : 40508
aida02 :      5      5     10      6      1      3      3      2      1      4      7   : 40540
aida03 :     25      7     12      5      1      2      2      3      2      3      7   : 39740
aida04 :     19      6     11      5      1      3      2      3      2      3      7   : 39820
aida05 :      9      6      7      5      0      2      2      3      1      4      7   : 40548
aida06 :     10      8      7      4      3      3      1      3      1      4      7   : 40600
aida07 :      9      3      4      8      1      3      3      2      1      4      7   : 40508
aida08 :     12      5      8      5      3      3      3      2      1      4      7   : 40632
aida09 :     15      4      8      5      1      3      3      2      1      4      7   : 40508
aida10 :     23      5     12      5      1      2      2      2      1      4      7   : 40228
aida11 :     15      9     11      6      2      4      3      3      2      3      7   : 40308
aida12 :     14      9      4      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      9      6      3      4      4      3      3      2      3      7   : 40260
aida14 :     11      5      7      6      4      4      3      3      2      3      7   : 40324
aida15 :     16      8      7      4      1      2      2      4      1      4      7   : 41136
aida16 :     15      8      7      4      3      4      3      3      2      3      7   : 40236

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_339 - attachment 9
       max deadtime 0.24% aida04
  630   Tue May 7 13:25:00 2024 JB, CC, NH, HMATuesday 6 May - dismount snout

Status before dismount

14:26 DSSSD#1 119.99 V @ 17.968 uA

          DSSSD#2 119.95 V @ 11.212 uA

          Leakage currents slightly elevated from Monday.

S4 temperature at 26 °C.

14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.

Plan:

- Dismount snout.

- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.

 

 

  296   Wed May 5 16:57:48 2021 TDTuesday 5 May
17.55 Grafana DSSSD bias & leakage current - attachment 1
       Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37 
      DSSSD bias & leakage currents - attachment 2
ELOG V3.1.4-unknown