09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39

Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Anydesk address now restored to 832827869

Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference

> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.

The following messages are in the system log very often:

Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk)

Turn on AIDA for Dry Run demonstrations and so on

All system wide checks, temp, bias OK

Noise situation is dreadful (but has not been optimised). Deterioriation since first mounted, suspect cabling issues with bPlast and BB7.

 Note thresholds at 0x32 (!!!) to not brutalise the DAQs during testing

 aida08 seems OK

Server running to MBS totally fine

18:00

Carole grounded some of the Bplast and this reduced the rates in AIDA, although they are a bit fluctuatey. Due to position constraints she couldn't ground it all
Also AIDA ribbon cables are not grounded yet
The indication is these fixes should make a lot of difference to the situation

AIDA is now powered off for the end of day

AIDA has 8 FEEs and 1 DSSSD 

Aida08 (HDMI#12) had no WR again, I moved it to a different MACB and now it gets WR
The MACB (currently with jsut HDMI#10) seems issues, check/replace the upstream HDMI and thent eh MACB (after dry run!)
 
DSSSD#1 biased to -120 V, leakage current 5.6 uA (fig 1)
 (also on Grafana)

Temps OK fig2

System Checks (fig3-5)
 Clocks OK
 aida07 fails calibration, others OK
 WR OK (aida04 0x10, seems OK)
 FPGA OK

Did "Synchronise ASIC clocks" to align ASIC clocks

(Notes for scalers: SC41L HDMI 5, SCI41R HDMI 9)

18:12 FRS is taking beam, AIDA is powered off and unbiased

In preparation for the dry run the following *temporary* changes to the FEE numbering have been prepared
These should be reverted after the dry run to ensure cable->fee agreement again

AIDA09 => AIDA06
AIDA11 => AIDA07
AIDA12 => AIDA08

This will allow the merger to run with 8 FEEs for 1 DSSD

dhcpd.conf updated

The 2023Oct19-13.46.30 should work with this numbering (check tomorrow)

As should layout GSI_triple_test_renumber 

AFTER dry run:

Revert DHCP and prepare for full 16 FEEs 

Make new ASIC settings key for 16 FEEs and prepare the aidaXX folders

Prepare a new Layout.mlf set

The behaviour of the DSSSD leakage current at low voltages and during biases is unusual and varies depending on how the adapter boards are connected
To summarise the behaviour I have observed

Minimum bias configuration:
 4 adapter boards, one n+n (LK1), three p+n (-ve bias), ground from n+n to one p+n
 Voltage (and leakage current) unstable at low voltages, seems to settle at around -60 V
 Drops can include 0 leakage current

Full adapter configuration:
 8 adapter boards, ground ring complete
 Same as minimum, but the drops seem to be much smaller (and not to 0 leakage current)
 -60V again seems to be the turnover to a stable leakage current 

In both cases the leakage current during ramping appears basically the same as when settled

Full into FEEs
 8 adapter boards, fully connected to 8 FEEs
 The leakage current is *much* higher during ramp,up to 17 uA near the end. No fluctuations
 Once ramping has finished the current quickly drops back down and settles at the nominal leakage current
 This has been observed in October/December too, it is not new (https://elog.ph.ed.ac.uk/AIDA/910)
 During power up of the FEEs the current sometimes drops briefly (when the ASICs get programmed, I believe)


I think it is related to the ground (more or less current flowing through the HV supply instead of alternate paths?)
It should be kept in mind when testing new detectors to not worry about the detector at low voltages

new Downstream DSSD2: 3208-2/3208-5/3208-8

Covered with black cloth.

Wafer 3 (beam left) shows breakdown issues at 90V... curious

We replace with november beam triple: 3208-3/3208-21/3208-22

We must speak to Micron about why so many 3208 wafers seem to have issues at >90V
It seems unlikely to be other issues

Upstream bPlast mounted

new Upstream DSSD: 3208-2/3208-5/3208-8

Covered with black cloth.

V-I behaviour found to be nominal.

Dismounted Snout and biased DSSSD1 channels

leftmost waifer working

middle waifer reaches 90V then current ramps up

rightmost waifer reaches 80V then current ramps up

DSSSD2 had fingerprint near connections, as shown in the attached image some had been squashed.

DSSD#1 undergoes a breakdown at 90V, two of the three wafers show this
- The adapter PCBs themselves have no breakdown at 100V, indicating the   issue is internal to the snout

If we are lucky it may be a loose or misaligned kapton connector inside the snout. If not we will have to remove DSSD#2 and inspect DSSD#1 for damage/lint/etc. If we see nothing it should be replaced

DSSD#2 biases perfectly fine, but the leakage current is unstable with biasing -ve to p+n and gnd (return) to n+n. Leakage current is stable biasing +ve to n+n and gnd to p+n. (https://elog.ph.ed.ac.uk/DESPEC/240311_093933/Downstream_positive_bias_vs._Current_(uA).png very nice curve)

We see the same fluctuations just daisy chaining 3 p+n PCBs togerher with the bias (no DSSSD or gnd links). The fluctuations were reduced by connecting all 8 adapter boards of the DSSD together.
When biasing just *one* PCB in this way, the current is stable
The fluctuations happened changing the HV channel, adapter board PCB and LEMO cables: it doesn't seem to be a defect with a specific thing.

The test today confirmed the HV cables (the only ones used) were correctly isolated from everything (OL = "infinite" resistance to ground)

We saw unstable current indications using the Mesytec MHV-4 to apply voltage to the PCB as well.

The behaviour is odd but doesn't seem to be related to the DSSD, which I suspect is OK. One idea is to try connecting the 3 adapter boards to FEEs and repeating the test, as this introduces another (substantial) path to gnd. Maybe this eliminates the current instability? It's about the only difference left from December.

-

For the next steps I believe the snout must be dismounted to inspect DSSD#1. It would be best to coordinate this with replacing the broken(?) bPlas. If we are lucky we may not have to remove the DSSDs but it is likely we have to remove both DSSDs to swap out #1

After replacing the DSSD(s) we should cover the snout with a clean black bin bag and bias it on the MH table to confirm both detectors work. This saves the effort of carrying it to S4 just to find another problem.
If both detectors make it to 120V we can mount it again more confidently

Once we have two biased detectors we can rearrange some FEEs to get 8 for DSSD#1 to do the noise tests. I suggest we wire them up for the numbering plan in https://elog.ph.ed.ac.uk/DESPEC/532 but in the DHCP renumber 9,10,15,16 to 5,6,7,8 temporarily to allow data to be sent to MBS for the dry run (merger limitation)

When the remaining FEEs are recovered from UK+CRYRING we can instrument DSSD#2 and renumber the FEEs to match the cables

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

 - When the p+n ribbon cables were disconnected for both the upstream and downstream AIDA, the pins and ribbon cables did not appear to have any blemishes.

- Both sets of adaptor PCBs were biased to 100 V and only had an apparent fluctuation in current at c. 0.01 uA. At low voltages (~10V) the current cycled between 0 and 1 uA.

2) Check that all ribbon cables are properly seated in the adaptor PCBs

- When inspected all ribbon cables seemed to be properly seated. A decent press was applied to confirm the seating. The PCBs themselves were inspected and no noticeable damage was observed.

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

- Done for the p+n side, can decide if this is needed for the n+n side.

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

-  Will have to decide further.

5) From email discussion: Check the SHV connector is grounded if one unplugs the cable from the back of the HV module. If yes: We are touching a ground somewhere. If no: it's via the HV (fine).

- When HV#0 was disconnected from the HV module, we used a multimeter on the SHV connecter and frame and read zero resistance 0L on the meter. While when completing the circuit for HV#1 (still) plugged into the HV module we read a resistance of > 1.7 Ohm.

- The HV is touching ground somewhere. This should not be a problem however as the snout is isolated, and this was observed by connecting the SHV to the snout and reading 0L.

6) Properly cover snout with black cloth and bias upstream.

- Snout was covered with black cloth and a black bag. The downstream detector p+n side was biased at 50V and charging/discharging was observed. This is probably due to a short connection somewhere.

Summary: from the p+n side test the voltage-current break down appears to come from inside the snout.

Proposed FEE numbering and wiring plan for upcoming experiment S100 (2x Wide DSSSDs)

Image designed in draw.io, source attached 

FEE numbering is as S450, minimises cable movement from S505/Narrow AIDA
But means merger is not working with 1 DSSD (until all FEEs installed)

Wiring of adapter boards as from noise tests and what should work for DSSD bias
LK3 on middle bottom adapter to ground DSSD
LK1 on one n+n adapter to ground n+n side bias 
p+n has -ve voltage (w.r.t. ground) bias applied via lower adapter boards
ground loop grounds all adapter boards, except 2 p+n adapter boards which are grounded by the bias lemo shield instead

MACB layout also included, with expected NIM logic signals for the aida scalers:

1: Pulser/Sync clock (send to all subsystems, "trigger 3")0

3/4: Time Machine 
5/6: SC41L/R 

All other FEEs have their scaler available
(Scaler should be in left LEMO on MACB, right is output (AIDA->NIM/unused), bottom 4 are triggers from AIDA (unused)

Test circuit will not be used in experiment due to noise, but can be temporarily set up for pulser walkthrough

Revision 2 correct as of 27 March 2023

Bias tests of AIDA on individual wafers and in parallel. Spreadsheet can be found in attachment 1, and graphic results can be seen in Attachment 2. Summary: Upstream detector cannot be biased in parallel nor individually. Downstream detector can be biased with positively and each wafer can individually be biased negatively. In parallel, it was not possible to bias the downstream detector negatively as indicated by the tests below.

Test downstream DSSSD with positive polarity bias

Configuration as follows:

CAEN N1419ET ch #3 connected to LHS FEE64 adaptor PCB ( looking upstream ) - LK1 *not* fitted LK1 fitted to 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cables from/to GND terminals of 3x ( top )

FEE64 adaptor PCBs *and* LHS FEE64 adaptor PCB 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything, LK3 fitted Total 5x adaptor PCBs installed

No other LKs fitted

Nominal V-I curve, stable leakage current. Attachment 3.

Following this success we attempted to repeat test using negative polarity bias

Configuration as follows:

CAEN N1419ET ch #1 connected to ( top, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cable from/to GND terminals of LHS and ( top, left )

FEE64 adaptor PCBs 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything,

LK3 fitted Total 5x adaptor PCBs installed No other LKs fitted With detector bias

-20V we continue to observe the leakage current cycling between 0 and ~2uA with a frequency ~1Hz ( as before )

Copy configuration used for upstream DSSSD test ( which was successful albeit there was detector breakdown at bias voltages > c. 90V )

CAEN N1419ET ch #1 connected to ( bottom, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( bottom )

FEE64 adaptor PCBs Ground cable jumpered from/to GND terminals LHS, ( left, bottom ), RHS and all 3x top FEE64 adaptor PCBs ( bottom, middle )

FEE64 adaptor PCB LK3 fitted Total 8x adaptor PCBs installed

No other LKs fitted With detector bias -20V we observe leakage current of ~2-3uA.

Current unstable - variations 10-100nA over periods of several seconds Although the leakage current is unstable this is an improvement over previous tests with negative bias. The duplication of upstream and downstream configurations suggests that for some unknown reason it is necessary to connect all 8x FEE64 adaptor PCBs whereas our expectation was that only 4x were necessary.

Summary: Upstream DSSSD Si wafers 1 & 2 breakdown for bias > c. 90V Si wafer 3 OK to 120V Positive bias - not tested Negative bias OK - leakage current stable to c. 90V Downstream DSSSD Si wafers 1, 2 & 3 OK to 120V Positive bias OK Negative bias - leakage current unstable

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

2) Check that all ribbon cables are properly seated in the adaptor PCBs

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

- Voltage-Current test of newly installed AIDA snout. Breakdown observed at around ~92 V. Probably caused my light leakage into the snout, will investigate further.

- Tried to cover the snout with a black cloth. This did not change the breakdown behaviour.

Snout assembly

DSSSDs

Upstream   3208-10/3208-18/3208-20
Downstream 3131-5/3131-10/3131-12

p+n junction side bias continuity checked OK for *all* wafers
 all c. 22 Ohm as expected

n+n Ohmic side bias continuity checked OK for both DSSSDs
 upstream 73 Ohm, downstream 92 Ohm

n+n Ohmic side bias wafer #2 
 positioned at top of snout assembly ( lower stage snout side marked 'T' )

p+n junction side faces downstream


Distances

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.0cm
ref - upstream DSSSD 11.0cm
ref - downstream DSSSD 12.0cm
ref - middle of downstream bPlas 14.0cm

Measurements consistent to +/-1mm between LHS and RHS of snout assembly as viewed from top side 


13.00 At this position the Kapton PCBs bulged in/out and there was some concern that when the upper stage
snout was installed it would the Kapton PCBs into the 'active' area. It was decided to move the 2x bPlas
and 2x DSSSDs c. 5mm downstream.

Jeroen and Phillip cut sections from 4x PEEK 6mm spacers so that they would clip onto 3mm dia support rods.

We raised the assembly fro the inter stage and inserted the modified spacers between the inter stage and 
the other spacers above. The Kapton PCBs now run fairly straight to the DSSSD connectors.

Distances re-measured as

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.6cm
ref - upstream DSSSD 11.5cm
ref - downstream DSSSD 12.4cm
ref - middle of downstream bPlas 14.5cm

Old ribbon cable assemblies - attachments 1  2

Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors

p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm

Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm



Manufacture of new ribbon cable assemblies - attachment 3

8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables


17.52 from Helena

"These are the detector positions within the snout:

Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame

Upstream AIDA @ 513 mm

Downstream AIDA @ 523 mm

This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is 
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by 
10mm.

The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position."

12.30 FEE64 41:d7:cd ASIC mezzanine u/s, no data, ASIC temperature c. 20 deg C ( ambient ) low
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida08

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps - attachment 1
       aida02 with new ASIC mezzanine continues to ramp to c. 512 deg C - initially reads 0, next refresh c. 70 and third refresh c. 512 deg C
       aida02 Virtex and PSU temps OK, aida02 cooling plate ambient to touch

       aida08 ASIC temp c. 45 deg C as expected, Virtex and PSU temps OK

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 13 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11