> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.

For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 

Runs are in /TapeData/BGAPR22 located on 8TB disk "SecondDrive" (N.B. 2A experiment is using ThirdDrive)

Old runs and test files were removed from disk. Left is an unknown S481 folder (fibre implant test?) and Patrick
chown to npg:npg as uid changed from old SL6 to CentOS 7

Bias DSSDs, leakage current higher than before (maybe the sheet isn't covering the snout)
Pulser set to OFF
ASIC Thresholds 0x64, rates mostly 0 (good)
Edge strips on aida02 and aida06 show non zero, and lots in aida04 asic 1 are high, will recheck connector tomorrow in hope

Connecting to merger and rate is 60k items/sec with random jumps to 400k items/sec
Start xfer to tape server: Rate approx 100 KB/s (360 MB per hour, 2.88 GB per 8 hours)

19:51 - Will leave overnight running to collect alpha

Fri 29 Apr

14:38 - Run paused for MBS tests with FRS

16:50 - Resume alpha collection over weekend
Pulser OFF, 0x64 thresholds

The snout endcap has been sealed with new copper tape over the red tape

Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive



-

the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s

-

Preliminary alpha background analysis (over weekend, 90 hours statistics)

Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Apr 28 19:50:20 2022
Last event : Mon May  2 14:04:03 2022
Duration   : 324823 seconds
--------------------------------------
MBS Events   (M):     912899 (2 Hz)
AIDA Words   (W): 7472078760 (-3441 Hz)
ADC Words    (A): 2742871865 (-4778 Hz)
-- Decays    (D): 2728717223 (-4821 Hz)
-- Implants  (I):   14154642 (43 Hz)
Info Words   (F): 4729206895 (1 kHz)
-- SYNC48    (S):    2093328 (6 Hz)
-- SYNC63    (s):    2093328 (6 Hz)
-- PAUSE     (P):    3823161 (11 Hz)
-- RESUME    (R):    3823161 (11 Hz)
-- Discrim   (D): 1390520565 (4 kHz)
-- Scaler    (C): 3326853352 (3 kHz)
-- Unknown   (U):          0 (0 Hz)
Timewarps    (T):         20 (0 Hz)
Dead Time    (X): 5024.43 s (1.54682%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
   1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
   2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
   3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
   4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
   5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
   6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
   7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
   8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
   9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
  10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
  11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
  12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
  13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
  14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
  15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
  16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9

PAUSE/RESUME stats are dominated by 2x FEE64s - aida07 and aida16

These FEE64s also have the highest rates of discriminator data?

> The snout endcap has been sealed with new copper tape over the red tape
> 
> Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
> It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive
> 
> 
> 
> -
> 
> the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
> When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s
> 
> -
> 
> Preliminary alpha background analysis (over weekend, 90 hours statistics)
> 
> Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04
> 
> ======================================
> AIDA Unpacker Analysis
> --------------------------------------
> First event: Thu Apr 28 19:50:20 2022
> Last event : Mon May  2 14:04:03 2022
> Duration   : 324823 seconds
> --------------------------------------
> MBS Events   (M):     912899 (2 Hz)
> AIDA Words   (W): 7472078760 (-3441 Hz)
> ADC Words    (A): 2742871865 (-4778 Hz)
> -- Decays    (D): 2728717223 (-4821 Hz)
> -- Implants  (I):   14154642 (43 Hz)
> Info Words   (F): 4729206895 (1 kHz)
> -- SYNC48    (S):    2093328 (6 Hz)
> -- SYNC63    (s):    2093328 (6 Hz)
> -- PAUSE     (P):    3823161 (11 Hz)
> -- RESUME    (R):    3823161 (11 Hz)
> -- Discrim   (D): 1390520565 (4 kHz)
> -- Scaler    (C): 3326853352 (3 kHz)
> -- Unknown   (U):          0 (0 Hz)
> Timewarps    (T):         20 (0 Hz)
> Dead Time    (X): 5024.43 s (1.54682%)
> ======================================
> ======================================
> Individual FEE Overview
> ======================================
>  FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
>    1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
>    2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
>    3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
>    4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
>    5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
>    6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
>    7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
>    8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
>    9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
>   10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
>   11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
>   12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
>   13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
>   14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
>   15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
>   16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9

bPlas has been mostly cabled in now

Reseat LAN cable (AIDA Switch -> AIDA-3) and reachieve 1 gbit/s speed as needed.

CHeck DAQ with MBS, working well (16 MB/s). Performance seems better than before on MBS side too

Note implant counts in aida05, observable in rate spectra (fig 1)
Also a few early channels in aida10 seem empty

ASIC check-loads didn't fix
Try powercycle... also didn't fix

Tomorrow will inspect adapter board and ribbon cable, perhaps it is loose.

Absolute rates otherwise seem OK (fig 2) aida01 and aida07 are both very hot.
aida01 is usually quieter but aida07 has always been observed running fast

AIDA FEE64 layout with respect to the beam.

09:00 - Remove bias of DSSDs  and pwoer of FEEs

Remove snout from S4 so DEGAS frame can be mechnically removed

Snout now in FRS Messhuette 

The n+n FEEs were all removed to adjust the platform back to the single configuration

After some confusion we found the orientation of the frame to position the single detectors and moved back

Faulty AIDA cards aida05 and aida14 were removed to be returned backi (ASIC damage)

The cables for aida10 and aida12 were moved back to the left-right side and the n+n cards were installed and reocnnected

p+n cards moved to bring aida11 and aida09 above aida03/aida07 and aida01/aida05. Also move a different card to replace old aida05

While doing this it was noticed the HDMI for aida13 was very loose and actually fell off (I guess the cable was pulled incorrectly)
It has been removed and will eb sent back to be resoldered 

The copper grounding bars and ITEM frame were removed due to conflicting with the single snout layout
The bolts are slightly too short to hold the frame where it won't conflict so we loko for longer bolts to reattach the system
Some new ground cables will be made to reach the extra n+n FEEs

Aida13-Aida16 removed from the PSU 

fig1: layout of snout frame in "single" mode (alu block configuration)


MAC Addresses

aida01 - ba-8a
aida05 - 0d:15
aida09 - ee-01

aida04 - f6-b7
aida08 - ba-2b
aida12 - ba:89 

aida03 - d8-21
aida07 - 41:b4:0c
aida11 - f6-5a

aida02 - ba:22
aida06 - ee:72
aida10 - cf-ac [mistakenly swapped with aida12]

Second switch will stay in S4 and has now been screwed in properly below the switch 1 and the hdmi cables entrance

DHCP *not* updated

Relay is powered *off* so FEEs cannot be turned on, alL HV channels disabled

Tomorrow is a holiday so no work will be happening @ GSI

10.30 : Turn off AIDA FEE64s 

        Prepare a ground cable for snout - M10 terminal on one end for copper bar. Will affix with Cu tape for now (no free screws)

Plan for today:

Remove extraneous ground cables
Check ground of all FEE64s esp. 05 and 07 - include on FEE64 side.
 14:42 - Done all grounds seem snug, extra ground cables removed. Try power on and check

Try powering just DSSD1 FEE64s
 - No change (aida01 125k, aida02,03,04 - 400k)

Try grounding snout
 14:59 Connection to snout very bad as tape does not fix well to thick copper in braid. nO difference either way.

Probe PSU output with Battery DSO if available
 - See figure 2

Try shuffling FEE64 power allocation
 - 15:32 - Now as 1 X 3 X 2 X 4 X
                  5 X 7 X 6 X 8 X
(Also FEEs on own pair)
   No difference to rates

Multimeter voltages for last socket of PSU1 (paired with aida04):
  1 : 6.02 V
  2:  6.02 V
  3:  6.02 V
  4: -6.44 V
  5:  7.74 V

Similar on PSU2

Voltage between right (GND) pin 5 and 1 (pin 5 is COM) is -0.4 V, similar if pin 4 is COM. Pin1-3 are 0 V when paired
  

Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 

Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change

All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 

aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour

Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after

The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system

Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 

The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use