There is a new operation available in the System Functions menu in the System Wide Checks page. ( The page needs to be reset once when the FEEs are powered before this function will appear )

Check the Options files are all the same size

Using this function will provide a list of the file sizes of the Options/< fee name >/CONTENTS files along with the last time they were accessed as a list.

It's up to the user, at present, to interpret the results but more is possible if required ....

12:55 We are going to try the follow changes to the AIDA scheme of 241012025.

- Fitting LK1 (aida01, aida14), removing LK1 (aida03, aida07, aida11, aida15) - only 1x LK1 needs to be fitted per Si wafer
- Removing LK4 to see what (if anything) changes.

Check that all of the screws securing the copper foil to ground are tight - snout install/removal tends to loosen screws.

- Remove the bias line between aidas 12-3-15 and aidas 16-7-11
- Remove the test circuit
 (These two should eliminate any extra ground path on the front, except via the DSSD)

- Use a multimeter to confirm 0 Ohm between all the copper foils  (short the two probes to correct for resistance of the probe leads)

- Try combinations of LK2/LK4 on the n+n fees

13:01 We check first the grounding with a multimeter and the different copper grounding components. They should all be grounded to the large iron frame, which is in turn grounded to the platform. We also checked that the CAEN HV PSU was also grounded to the platform (it is).

13:22 We removed the LK on aida03, aida07, aida11, aida15 and added LK1 to aida01 and aida14. We try starting the DAQ and seeing the response from these changes.

13:25 We tried biasing with this configuration and the first DSSD on HV01 read a leakage current of 40 uA (too high). So we biased it down and reassessed. 

13:30 It turns out we had LK1 fitted to aida04. We tried biasing again but this time DSSSD on HV01 tripped. They are fitted with LK4 so we biased down and removed these link on aida06 and aida08.

13:35 Trying to bias now with LK4 removed from aida06 and aida08. We tried fitting and removing again. Now the setup appears to bias. This was probably due to a stray piece of copper shielding touching the PCB.

14:22 We are checking the noise conditions now. TEMPS OK. FADC Align and whiterabbit timestamps OK.

14:30 With the changes above we have noticed an intense 5/2 ms pulsed signal in the wave forms. See attachment 2

15:20 We decabled everything and the current scheme from 24102025, see prev. elog entry is in attachment 1.

15:34 We have unbiased AIDA1 and the wave forms improve for AIDA2 (?), 5/2 ms pulsed signal is still observed in AIDA1

19.06 Found HV OFF, DAQ GOing

      DAQ STOP
      FEE64 power OFF


> 12:55 We are going to try the follow changes to the AIDA scheme of 241012025.
> 
> - Fitting LK1 (aida01, aida14), removing LK1 (aida03, aida07, aida11, aida15) - only 1x LK1 needs to be fitted per Si wafer
> - Removing LK4 to see what (if anything) changes.
> 
> Check that all of the screws securing the copper foil to ground are tight - snout install/removal tends to loosen screws.
> 
> - Remove the bias line between aidas 12-3-15 and aidas 16-7-11
> - Remove the test circuit
>  (These two should eliminate any extra ground path on the front, except via the DSSD)
> 
> - Use a multimeter to confirm 0 Ohm between all the copper foils  (short the two probes to correct for resistance of the probe leads)
> 
> - Try combinations of LK2/LK4 on the n+n fees
> 
> 13:01 We check first the grounding with a multimeter and the different copper grounding components. They should all be grounded to the large iron frame, which is in 
turn grounded to the platform. We also checked that the CAEN HV PSU was also grounded to the platform (it is).
> 
> 13:22 We removed the LK on aida03, aida07, aida11, aida15 and added LK1 to aida01 and aida14. We try starting the DAQ and seeing the response from these changes.
> 
> 13:25 We tried biasing with this configuration and the first DSSD on HV01 read a leakage current of 40 uA (too high). So we biased it down and reassessed. 
> 
> 13:30 It turns out we had LK1 fitted to aida04. We tried biasing again but this time DSSSD on HV01 tripped. They are fitted with LK4 so we biased down and removed 
these link on aida06 and aida08.
> 
> 13:35 Trying to bias now with LK4 removed from aida06 and aida08. We tried fitting and removing again. Now the setup appears to bias. This was probably due to a stray 
piece of copper shielding touching the PCB.
> 
> 14:22 We are checking the noise conditions now. TEMPS OK. FADC Align and whiterabbit timestamps OK.
> 
> 14:30 With the changes above we have noticed an intense 5/2 ms pulsed signal in the wave forms. See attachment 2
> 
> 15:20 We decabled everything and the current scheme from 24102025, see prev. elog entry is in attachment 1.
> 
> 15:34 We have unbiased AIDA1 and the wave forms improve for AIDA2 (?), 5/2 ms pulsed signal is still observed in AIDA1

16:30
    All system wide checks passed
    Temps shown in attachment #1 
    Histos shown in attachment #2
      NB: Zoomed in, linear scale
    Biases shown in attachment #3
    Stats shown in attachment #4
    Merger rates fine 3 Mega items / sec
    TapeServer looks happy, 25 MB/s

Paddy was asking us to give a rough estimate for the number of implants per spill.
I zoomed in on the rates histogram (attachment #2) and it looks like roughly 5 implants per channel in SSD 1 and SSD2.
So this is 5*128*2=1280.  Note that I don't know the rate window size (e.g., 1 second, 1 spill?)  Probably the window does not catch a full spill.
This might be an over-estimate because an implant in SSD2 may leave a signal in SSD1.
Paddy was getting around 1k or 2k fragments per spill, but to first order it seems consistent.

18:36
   Near R9_372 they slightly opened some FRS slits, as there is a belief we may be blocking some of 94Pd.
   Implant rate seems similar to before, see attachment #5
   Biases did not change more than normal, see attachment #6

20:15
   All system wide checks passed
   Temps shown in attachment #7
   Stats shown in attachment #8
   Biases shown in attchment #9

22:12 
   All system wide checks okay
   Histo #10
   Bias #11
   Stat #12
   Temps #13

13:11 System wide checks all ok except *aida02 fails ADC calibration*
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue May 04 10:57:38 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

ASIC clocks have been synchronised and ADC re-calibrated

Rate spectra - attachment 1
Layout7 - attachment 2
Layout8 - attachment 3

Detectors at 130V

Statistics - attachment 4

V-I Curve - attachment 5
Voltage	Ch0	Ch1
10	1.345	1.49
20	2.01	2.285
30	2.405	2.8
40	2.635	3.13
50	2.82	3.37
60	2.95	3.57
70	3.08	3.725
80	3.185	3.865
90	3.26	3.98
100	3.33	4.105
110	3.42	4.22
120	3.525	4.35
130	3.64	4.46
140	3.79	4.37
150	3.98	4.37
150+10min	3.9	4.485

- Bias and basic checks: the pulser FWHM appears to be ~170, which is the same we have obtained at the end of last week.

- Initialised the Merger and Tape server

- loading of settings: EXPERIMENTS/AIDA/2025Nov03-16.21.44

- slow comparator thresholds set to 700 keV for all X FEES

- y FEEs disabled in the discriminator

- Starting data taking with following details:

MIDAS data in: Bi207Centre_700

MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq')

We take data overnight in this configuration and continue the grounding checks tomorrow

Some FEES were not acquiring data for some reason we could not understand. We lowered the thresholds to 200 keV, then rose them again to 700 keV. It seems they are now acquiring. We start data taking again.

Anydesk has crashed. I am not able to set it up again. I switch off bias and fees

A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21

Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64

S4 access will be limited. 

I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal.

Rates in all FEEs much higher than before - waveform shows very noisy in all systems. - attachment 1

Good event statistics - attachment 2

DISC in 7 and 3 above 300k

ASIC check load performed.

Good event statistics - attachment 3

Thesholds at 0xa for all FEE

Disc rate now 0 across all FEE


Pulser settings: attachment 4

pulser peak widths -   FEE         width(ch)
                              1             128
                              2             145
                              3             84
                              4             74
                              5             207
                              6             125
                              7             172
                              8             80
                              9             193
                              10            183
                              11            238
                              12            138

Attachments 5 & 6: pulser peaks for all FEE64

16.24: DAQ stop

       slow comparator threshold changed to 0x64

       ASIC check performed

       Alpha run/DAQ start (w/ data transfer enabled) - writing to file 31Oct19/R13

       Merger and TapeService ok - attachments 7 & 8

08:09 DAQ crashes at start of shift (!)

      again happens at same time as beam being lost

      All FEE64 lost connection

      Beam downtime for next 3-4 hours

      Performed powercycle and reset

08:40 AIDA back up and running, writing to R36

08:42 *all* system wide checks ok

      temperatures ok - attachment 1

      statistics ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:57 options file size check;


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue Apr 20 08:22:14 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

08:59 no error messages in database check terminal
      stats ok

09:16 waveforms - attachments 4 & 5

09:30 no error messages in database check terminal
      stats ok

10:07 no error messages in database check terminal
      stats ok

10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes

      powercycle attempted, but FEE modules remain unconnected

      TD performs further power cycle

11.12 reboot aida05

      DAQ reset/setup

      system wide checks

      sync asic clocks OK
      clock status & fpga timestamp errors OK
      
      WR decoder status

		 Base 		Current 	Difference
aida09 fault 	 0xb307 : 	 0xb307 : 	 0  
aida09 : WR status 0x40
 aida10 fault 	 0x3d6c : 	 0x3d6c : 	 0  
aida10 : WR status 0x40
 aida11 fault 	 0xa610 : 	 0xa610 : 	 0  
aida11 : WR status 0x40
 aida12 fault 	 0x7c7 : 	 0x7c7 : 	 0  
aida12 : WR status 0x40
 White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)

earlier on, Akash secured HDMI cable to aida07

11:55 AIDA powercycle and reset, DAQ ok now

      writing to file NULL/R45

12:00 CB takes over

11:55 CA got AIDA back online

12:00 Stats OK (attach 1)
No faults found
UCESB OK - no beam atm (attach 2)
Temps OK (running hotter than yesterday, but not worrying yet) - attach 3
Bias OK (attach 4)

System wide checks
Clock - *all* pass
ADC - 06 fails
WR - *all* pass
FPGA *all* pass

Memory info
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      4      1      3      2      1      2      4      2      3      7   : 39876
aida02 :     23      6      1      2      2      2      1      4      2      3      7   : 39772
aida03 :     24      4      4      1      2      3      1      4      2      3      7   : 39904
aida04 :     10      3      1      1      1      3      1      4      2      3      7   : 39728
aida05 :     12      2      3      2      1      4      2      3      1      4      7   : 40688
aida06 :     10      5      4      2      2      3      3      4      2      3      7   : 40400
aida07 :     22      6      1      3      1      3      1      4      2      3      7   : 39864
aida08 :     20      4      4      1      2      3      2      4      2      3      7   : 40144
aida09 :      1      2      2      3      1      2      1      3      3      3      7   : 40148
aida10 :      3      3      1      2      2      2      2      3      2      3      7   : 39412
aida11 :     25      7      0      1      0      3      2      3      2      3      7   : 39484
aida12 :     21      7      3      1      3      2      2      3      2      3      7   : 39580


Waiting for beam.

12:42 Stats OK, UCESB ok, No faults. No beam.
13:21 Stats OK, UCESB ok, No faults. No beam.
13:24 Implants in DSSD#3. ASIC check. Disappeared. No beam.

14:00 No beam.
Stats OK (attach 5)
Temps OK (attach 6)
UCESB OK
Bias  OK (attach 7)

System wide checks
Clock OK
ADC calibration 06 fails
WR OK
FPGA OK

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      1      3      2      3      3      2      3      7   : 39848
aida02 :     15      8      2      2      2      3      2      3      2      3      7   : 39644
aida03 :      8      1      2      3      4      3      2      3      2      3      7   : 39720
aida04 :     25     12      3      1      1      3      1      4      2      3      7   : 39892
aida05 :     13      4      4      4      0      3      2      3      1      4      7   : 40596
aida06 :     21      8      3      3      0      3      3      4      2      3      7   : 40356
aida07 :     17     11      0      1      1      3      1      4      2      3      7   : 39804
aida08 :     16      8      2      3      2      3      1      4      2      3      7   : 39936
aida09 :     17      6      3      2      1      2      1      4      2      3      7   : 39716
aida10 :      3      3      2      2      4      1      2      2      3      3      7   : 39940
aida11 :      4      4      4      1      1      3      3      2      2      3      7   : 39248
aida12 :      2      4      1      3      1      2      2      3      2      3      7   : 39384

14:30 Stats OK, UCESB ok, No faults. No beam.
15:00 Stats OK, UCESB ok, No faults. No beam.


15:24 Beam is back!
Stats OK (attach 8), UCESB ok (attach 9), No faults.

15:50 Shift handed over to Liam

Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Anydesk address now restored to 832827869

Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Attachment 1 - analyser.f 

Modified version of TDR format analyser program to handle 64 bit WR timestamps

Attachment 2 - analyser program output

12092 blocks = 773888kb

8 timewarps

0 information code 5 data items

data file does not commence with either information codes 5 or 4

times/rates not correctly calculated - program assumes 10ns/tick 

Attachment 3 - od -x output 

S496/R5_18 - attachment 1

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 0xf
                             n+n FEE64s 0x20
Sampling ADCs OFF


S496/R5_32 - attachment 2

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 & aida15 0xf
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


S496/R5_45 - attachment 3

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xc
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


* N.B. Ignore 'elapsed idle time(s)' metric which does not work with WR timestamps
 

Analysis of data file R2_0

Attachment 1 - summary output of TDR analyser

Attachment 2 - verbose output of TDR analyser