DSSSD MSL type BB18 24cm x 8cm 3208-3/3208-21/3208-22

FEE64 configuration see https://elog.ph.ed.ac.uk/AIDA/872 attachment 2

Bias -150V -6.590uA ambient temperature +24.7 deg C d.p. +13.7 deg C RH 50.3%

BNC PB-5
amplitude 10.0V
attenuation x10
decay time 1ms
tail pulse
frequency 25Hz

PB-5 output direct to p+n junction side FEE64 aida01 or aida12, or n+n Ohmic side FEE64 aida02


aida01 1.8.L pulser peak width  61 ch FWHM ~  46keV FWHM => 5s threshold 98keV 
aida12 1.8.L pulser peak width  56 ch FWHM ~  42keV FWHM => 5s threshold 89keV
aida02 1.8.L pulser peak width 102 ch FWHM ~  77keV FWHM => 5s threshold 163keV


slow comparator  0xa 100keV ( all p+n junction FEE64s ) 
                 0xf 150keV ( all n+n Ohmic FEE64s )

per FEE64 Rate spectra - attachment 1
 p+n FEE64s ( aida010, aida01, aida09, aida12, aida03, aida11 ) rates dominated by hot channels, other channels typically <1Hz ( 25Hz pulser to aida12 )
 n+n FEE64s ( aida02, aida04 ) rates ~ 10-20Hz/channel
 Note aida06 and aida08 are not connected to anything and should be ignored


ADC data item stats - attachment 2


For further information see https://elog.ph.ed.ac.uk/AIDA/906 and https://elog.ph.ed.ac.uk/AIDA/907


To Do

- repair/replace Honeywell HSS-DPS dew point sensor
   USB-controlled ac mains relay interlock currently overrriden
   do NOT operate AIDA unattended

- aida04 asic #1 u/s - replace ASIC mezzanine

- electrically isolated test signal distribution box req'd

- aida10 asic #4
   v. high rates observed and large signal transients 
   cause unclear ... ASIC/adaptor PCB/cabling/Si wafer ?

- extended background alpha run to check all DSSSD bond wires
   pulser OFF
   slow comparator 0x64

- bPlas + 2x triple DSSSD + bPlas stack assembly and test

- all up, in beam test

The photographs show the cooling water controls and temperature/pressure gauges outside S4 and the connections used by AIDA within S4.

The AIDA MBS FDR will be x86l-119 from now on, not x86l-94

the MBS relay and startup scripts will be changed for this

There was a power failure in the morning of 24.08.2023 in the Rhein-Main area affecting GSI

The Aida workstation (aida-3) has been restarted, it is unknown if the Pis in S4 rebooted as well (there is a UPS)

29.8.23 TD Both RPi systems rebooted four days ago.

Firefox browser proxy setting changed to 'Auto-detect proxy settings for this network' ( Firefox -> Edit -> Settings -> Network Settings -> Settings )

When using the VETAR connected to the HDMI port of the root MACB the settings for all the MACB in the system 0x3.

This setting takes the Clock and Data line from the input HDMI and outputs it via teh HDMI output ports.

When using the Emulator connected via the SMA connectors on the back of the root MACB then the setting for the root MACB should be 0xD and all others should be 0x3.

Attached is the .jed file for programming the MACB and the .vhd source file to help with understanding of the settings.

The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible

Investigating AIDA noise with a TA041 differential probe and oscilloscope 

AC Mains (DESPEC platform AC, L-N)
Probe attentuation = 1:100

Fig 1: Main AC waveform [X: 5ms/div, Y: 100 V/div]
Fig 2: Zoomed in at peak (20 V FSR, any less and the waveform clipped) [X: 10us/div, Y: 20V/div]
Fig 3: Longer time base and FFT of 0-5 MHz. No significant frequency harmonics noticed [X: 5ms/div, Y:20 V, FFT X: 500 kHz/div, Y: 10 dBm/div]

No significant noise or distortion present, fully within any AC specification. 
Note that at the moment there is almost no load on AC
Equipment on on DESPEC rack: AIDA NIM, AIDA Raspberry Pis, bPlas PC (+ WR) + 2x DESPEC NIM crates
 No autofill, VME crate or detectors
All big machines at GSI (SIS, FRS) off (suspect pumps are on)

Ion catcher not on (I think under repair)

-

FEE PSU studies
Probe connected to 5V exposed power pin on FEE64 (+v) and to grounding crimp on FEE64 (-v)
No adapter board connected
Attentuation = 1:10


Fig 4: FFT when FEEs are *off* - essentially probe+scope noise [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 5: FFT when FEEs are *on* - notice 1.4 MHz peak in FFT, also seen on ADC waveform readout before (fig 6) [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]

Fig 7: 500 ns/div 5V output on FEE, single FEE on the PSU [X: 500 ns/div, Y: 100 mV/div]

Note average max voltage is 5.31 V (power on) and ~ 70 mV "peak to peak" -might be from probe/scope?

Also see voltage changes with FEE power draw:

Power on  : 5.45 V (different scale to above)
SETUP ran : 5.51 V 
FADCs off : 5.86 V
ACQ Go:   : Unchanged; ASIC threshold 0xa: Unchanged

Also check situation on a fully loaded PSU (8 fees connected and powered on)

Power on:   5.29 V (fig 8)
SETUP ran:  5.36 V (fig 9)
FADCs off:  5.64 V (fig 10)

All X: 500 ns/div, Y: 100 mV/div

Both cases observe voltage rises as current draw drops (as expected for voltage drop along a cable)

Noise on 'scope seems to get slightly worse with reduced current (and higher voltage)

No sign of strong 100 kHz noise as seen in ADC traces beforehand

Todo:

- Check -6V and 7V rails
- Check 5V and noise when front-end card is added and pulser/HV connected
- Check between two FEE64 grounds 
- Check direct out of PSU vs ground to see if 1.4 MHz appears on PSU side or FEE64 side

-
11.10.22 Updates

Attachement 11 - 5V PSU on upper PSU with no FEEs attached whatsoever. No 1.4 MHz (on FFT) but clear low frequency beats from switching - presumably low/no load behaviour

Attachement 12 - 5V PSU on aida12 with 8 FEEs on PSU. Longer time base to allow lower frequencies in FFT. 1.4 MHz switching spikes visible but nothing around 100 kHz region

Attachments 13-16: 5V PSU on aida12 at 20 mV/div vertical and 1, 0.5, 2, 5 us/div horizontal respectively

12.10.22 Updates

Attachment 17: -6V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: -6.21 V

Attachment 19: 7V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: 7.46V

Measurement between AIDA12 ground and Reference ground/copper bar
+ve (red probe) attached to copper bar at ground point (not strong connection at present)
-ve (black probe) attached to ground crimp on aida12 (connected to cooling plate)
aida12 no adapter board connected: connections are PSU, Ethernet, HDMI and TTY only

Attachment 21: 5 us/div 100 mv/div waveform, big oscillations present. Not seen before FEEs turned on (8 FEES, 1-7+12)
Attachment 22: 10 ms/div for FFT, sharp peak at exactly 100 kHz observed...


Attachment 23: Between 5V PSU (+ve) and 19" rack (-ve) with  no FEEs connected to PSU
 See strong 100 kHz oscillations too, note that voltage isn't 5V as PSU is floating w.r.t. ground
 Looks to be common mode noise (on both 5V and Return of PSU)

Attachment 24: Same as 21 but using thick crocodile clips on probe to ground and aida12. Noise is attenuated but still present

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use

Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 

The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system

In no particular order

1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)

   observe +/- 6V, 12V, 24V lines with/without load 

   try new CAEN NIM bin and/or NIM bin of different type

2) Measure actual voltages at FEE64 power connector input

   OH suggests fab of power adaptor for safe observation - contact EW

3) rev B adaptor PCB

   invert 125 way ERNI - check for mech conflicts
   paired HV input (avoid Lemo-00 T pieces)
   consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
   straight jumpers
   shrouded Samtec headers - consider mech issues/consequences of using eject clips too
   re-visit HV filtering & separate trace ground

4) isolation transformer

   as practical matter may be necessary to operate all platform from isolation transformer
   consider hire of appropriate unit

   need method to measure isolation - will require permit to work or equiv

5) investigate S4 area ac mains

   NH discussing with GSI electricians

6) Systematic measurement of AIDA PSU noise

   Spec linear AIDA FEE64 PSU

7) Redesign of snout
   Return to 1mm welded box Al for lower stage of snout  for added rigidity

8) Revisit calculation of cable lengths. Particularly for the triple

9) 



   

All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 

11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?

Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after

00:07 Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_896 - attachment 1
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6 & 7

00:15 Check ASIC control all FEE64s, all ASICs


02:03

analysis of file S505/R5_914 - attachment 8
 zero timewarps
 deadtime all FEE64s << 1% 

per FEE64 1.8.H spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 94 ch FWHM

per FEE64 stat & rate spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today


03:41

analysis of file S505/R5_926 - attachment 19
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24


07:00

analysis of file S505/R5_954 - attachment 25
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

07:45 Spoke to David and the beam has been gone since about 05:30
      Reason for the loss of beam is a vacuum issue before the FRS
      They are waiting for the experts

08:31 Statistics ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage currrents ok - attachment 3
      ASIC clock check ok
	
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2
	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  

      Currently on file R5_771
      Analysis of file R5_770 (No beam) - attachment 4
      Around 0.25% deadtime on AIDA02 rest even less

09:02 Current free HDD space 965 GB
      Current tape server rate 4979 kB/s
      Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours

11:41 Statistics ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7
      ASIC clock check ok
	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
      Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently?

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change