13.00 Cooling water temperature and flow OK - attachment 1

      Test of AIDA 2x MSL type BB18(DS)-1000 24cm x 8cm DSSSDs 'as is' ( to be defined later in this Elog )

      DSSSD bias & leakage current OK - attachment 2
      Ambient temp 24.7 deg C, d.p. 7.1 deg C, RH 32.4%
      Leakage current c. 6.6uA => 4nA/cm2/100um ( very good )

      FEE64 temps OK - attachment 3
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder status - attachment 4

      WR timestamps OK - attachment 5

      WR (info code 4 & 5), correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 6-12

      per FEE64 Rate spectra - attachments 13-15

      per 1.8.W spectra - 20us FSR - attachments 16-1710129


14.45 FEE64 config check - per https://elog.ph.ed.ac.uk/DESPEC/562


FEE64 # PSU cable #           MAC
1       1                     ?
2       2                     ?
3       3                     ?
4       4                     41 ee 71
5       5                     ?
6       6                     ?
7       7                     f6 5a
8       8                     41 d7 cd
9       ?                     ?
10      10                    41 d0 0e
11      11                    41 ee 0f
12      ?                     ?
13      13                    ?
14      ?                     0d 15
15      15                    ?
16      16                    f6 ed

? = no line of sight

- AIDA FEE64 PSU cabling - see attachment 18

Currently

PSU #1 1-3, 2-4, 9-5, 15-12
PSU #2 14-7, 6-8, 10-13, 11-16

Should be changed to

PSU #1 1-3, 2-4, 9-15, 5-12
PSU #2 14-7, 6-8, 10-11, 13-16


- AIDA snout mount to support frame - attachments 19 & 21
LHS ( looking downstream ) incorrect, RHS OK

- LKs
LK1 fitted aida02, aida04, aida06, aida08
LK3 fitted aida03, aida07, aida01, aida14


Significant gaps in snout ( i.e. possible light leaks ) where bPlas cabling exits snout - attachments 20 & 22

bPlas driver PCBs removed 
bPlas ribbon cables, drain wires and Lemo 00.250 cabling disconnected - some cabling/drain wires touching snout etc


15.15 Adjust bPlas ribbon cables/drain wires/Lemo cables to ensure that none are touching snout or AIDA support frame - significant improvement

      DSSSD bias & leakage current OK - attachment 23

      FEE64 temps OK - attachment 24

      ADC data item stats - attachments 25

      per FEE64 Rate spectra - attachments 26
       6x FEE64s < 20k. all FEE64s < 120k ( cf. 4x FEE64s < 20k, 5x FEE64s > 100k, 1x FEE64 320k earlier - attachment 12 )

      per 1.8.W spectra - 20us FSR - attachments 27-28

17.15

LHS top hat washers
LK1, LK3
PSU power cables
tighten gnd screws
attachments 29-34

Note that some of the newly installed FEE64s have the incorrect rails fitted - FEE64s may ground to AIDA support stand

17.55 CAEN N1419ET LK removed -> floating outputs

18.15 +LK1 restored
attachments 35-40

18.40 +pwwer cycle
attachments 41-46

Changed /MIDAS/Linux/startup/NewMerger

Change parameters -i and -l in master64 to 16 for 16 FEEs
Update NewMerger Options LinksAvailable to 16, LinksInUse to 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Fix NetVar RunOptions 1 (was 0)

Restart Merger HTTPd, Tape, Merger, MBS Spy

Reset/Setup/Go 

16 Links green and status going, all good?

Bias DSSSDs and turn data transfer on

Merger connected, shows rate and updates... no rate in Tape Server?

.. Oops forget to turn on Output to data storage in merger!

Rate in tape server and to MBS: 7 MB/s

Merger, Tape and MBS working with 16 FEEs

The WR error counter for aida02 seems to consantly rise

Tried reseating cable on both ends, no change

However clock status passed, aida02 has a correct WR timestamp and no FIFO/PLL errors seen

Edit to add: aida02 has the faulty ASIC temperature readout as well, related or coincidence?

When booting up AIDA aida06 usually crashes the first time, it fails to get IP from DHCP
After 180 seconds it reboots and seems to connect fine

Log file attached, key part (to me) is this:

27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|

S505 ADC offsets using pulser walkthrough data from data file R1

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly 

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

Mounted on frame:

DSSD 1 (Upstream) :     3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22

Current mapping has been redone for better management.

Going to try optimising noise now.

DHCP updated

new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)

New layouts: /home/npg/LayOut/GSI_Triple_S100

New layout.txt

Firmware of aida11 updated from 0xea40704 to 0x3350706

Temps GOOD fig 5

Rates fig 6, 7

Check adapter alignment aida14 and aida16

bPlas left/right cables are not insulated and shorting to the snout

Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor

From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!

White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)

Turn off bPlas
We see the noise drop a lot

Sensor ( 4 pins )

Red  +24V
Blue   0V
Yellow } contact 
Green  } closure in ( short these two wires together for logic 1 )

N.B. yellow and green wires of unused inputs must be connected together


Output ( 3 pins ) to USB-controlled ac mains relay

Red    NC ( normally closed - with respect to COM with no power applied, single pole double throw relay out )
Blue   NO ( normally open - with respect to COM with no power applied, single pole double throw relay out )
Green COM

Taken 4 FEE64s from CRYRING (the 4 easiest to access)
  41:d8:2b
  41:f6:5a
  41:ee:71
  41:d0:0e

In addition a FEE without rails marked only 20 was found in my old office for 5
- This probably came in a shipment to/for repairs

DSSSD#1 unbiased

DESPEC Platform moved out of beam and gamma platform moved from AIDA
JB works on grounding the original 8 DSSSD#1 FEEs

Connected grounding from ribbon cable drain to adapter PCB
Ground loop connected

AIDA rates unchanged bad still fig 1
Waves fig 2 (p+n 6000-9000)
Waves fig 3 (n+n 8000-10000)
Statistics fig 4

09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39

Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Anydesk address now restored to 832827869

> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.

Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference

The following messages are in the system log very often:

Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk)

Turn on AIDA for Dry Run demonstrations and so on

All system wide checks, temp, bias OK

Noise situation is dreadful (but has not been optimised). Deterioriation since first mounted, suspect cabling issues with bPlast and BB7.

 Note thresholds at 0x32 (!!!) to not brutalise the DAQs during testing

 aida08 seems OK

Server running to MBS totally fine

18:00

Carole grounded some of the Bplast and this reduced the rates in AIDA, although they are a bit fluctuatey. Due to position constraints she couldn't ground it all
Also AIDA ribbon cables are not grounded yet
The indication is these fixes should make a lot of difference to the situation

AIDA is now powered off for the end of day

AIDA has 8 FEEs and 1 DSSSD 

Aida08 (HDMI#12) had no WR again, I moved it to a different MACB and now it gets WR
The MACB (currently with jsut HDMI#10) seems issues, check/replace the upstream HDMI and thent eh MACB (after dry run!)
 
DSSSD#1 biased to -120 V, leakage current 5.6 uA (fig 1)
 (also on Grafana)

Temps OK fig2

System Checks (fig3-5)
 Clocks OK
 aida07 fails calibration, others OK
 WR OK (aida04 0x10, seems OK)
 FPGA OK

Did "Synchronise ASIC clocks" to align ASIC clocks

(Notes for scalers: SC41L HDMI 5, SCI41R HDMI 9)

18:12 FRS is taking beam, AIDA is powered off and unbiased

In preparation for the dry run the following *temporary* changes to the FEE numbering have been prepared
These should be reverted after the dry run to ensure cable->fee agreement again

AIDA09 => AIDA06
AIDA11 => AIDA07
AIDA12 => AIDA08

This will allow the merger to run with 8 FEEs for 1 DSSD

dhcpd.conf updated

The 2023Oct19-13.46.30 should work with this numbering (check tomorrow)

As should layout GSI_triple_test_renumber 

AFTER dry run:

Revert DHCP and prepare for full 16 FEEs 

Make new ASIC settings key for 16 FEEs and prepare the aidaXX folders

Prepare a new Layout.mlf set

The behaviour of the DSSSD leakage current at low voltages and during biases is unusual and varies depending on how the adapter boards are connected
To summarise the behaviour I have observed

Minimum bias configuration:
 4 adapter boards, one n+n (LK1), three p+n (-ve bias), ground from n+n to one p+n
 Voltage (and leakage current) unstable at low voltages, seems to settle at around -60 V
 Drops can include 0 leakage current

Full adapter configuration:
 8 adapter boards, ground ring complete
 Same as minimum, but the drops seem to be much smaller (and not to 0 leakage current)
 -60V again seems to be the turnover to a stable leakage current 

In both cases the leakage current during ramping appears basically the same as when settled

Full into FEEs
 8 adapter boards, fully connected to 8 FEEs
 The leakage current is *much* higher during ramp,up to 17 uA near the end. No fluctuations
 Once ramping has finished the current quickly drops back down and settles at the nominal leakage current
 This has been observed in October/December too, it is not new (https://elog.ph.ed.ac.uk/AIDA/910)
 During power up of the FEEs the current sometimes drops briefly (when the ASICs get programmed, I believe)


I think it is related to the ground (more or less current flowing through the HV supply instead of alternate paths?)
It should be kept in mind when testing new detectors to not worry about the detector at low voltages