10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11

aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour

Yesterday the sum/inverter was found to be a considerable source of noise in the system.

Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace

Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow

Put inverter back in CAEN crate in S4 and confirm similar behavior.

Move CAEN crate to new mains port, same behaviour

Move CAEN crate to AC voltage stabiliser, same behaviour

Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears  - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears

Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.

MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd

CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?

MACBs and inverter checked on another crate - no signal

Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected

Tested adding bias filters - No change

15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
      We get ok stats even with waveforms on at 0xa- attachment 5

      Waveforms still have the 100kHz oscillation in them though attachments 6 and 7

      Without waveforms we get similar performance to before the switch to the new mains - attachment 8

15:56 Move CAEN NIM crate across to this mains network
      300+kHz on all FEEs with waveforms on. Can recover waveform off performance though

16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.

17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9

01.20 

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

All waveform ADCs disabled

BNC PB-5 pulser
 to p+n FEE64s only

all system wide checks OK *except* ADC calibration, WR decoder & FPGA timestamp errors - see attachments 1 & 2

DSSSD bias & leakage current OK - see attachments 3 & 4

FEE64 temps OK - see attachment 5

statistics OK - ADC data, pause, resume & correlation scaler data items
                data push, data flush, aida01, aida02 - see attachments 6-13

per FEE64 1.8.L spectra - see attachment 14
 aida01 pulser peak width 73 ch FWHM

Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23

All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 

Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change

Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 

10.30 : Turn off AIDA FEE64s 

        Prepare a ground cable for snout - M10 terminal on one end for copper bar. Will affix with Cu tape for now (no free screws)

Plan for today:

Remove extraneous ground cables
Check ground of all FEE64s esp. 05 and 07 - include on FEE64 side.
 14:42 - Done all grounds seem snug, extra ground cables removed. Try power on and check

Try powering just DSSD1 FEE64s
 - No change (aida01 125k, aida02,03,04 - 400k)

Try grounding snout
 14:59 Connection to snout very bad as tape does not fix well to thick copper in braid. nO difference either way.

Probe PSU output with Battery DSO if available
 - See figure 2

Try shuffling FEE64 power allocation
 - 15:32 - Now as 1 X 3 X 2 X 4 X
                  5 X 7 X 6 X 8 X
(Also FEEs on own pair)
   No difference to rates

Multimeter voltages for last socket of PSU1 (paired with aida04):
  1 : 6.02 V
  2:  6.02 V
  3:  6.02 V
  4: -6.44 V
  5:  7.74 V

Similar on PSU2

Voltage between right (GND) pin 5 and 1 (pin 5 is COM) is -0.4 V, similar if pin 4 is COM. Pin1-3 are 0 V when paired
  

Spectra layouts for 2x DSSSDs can be found in directory ~/LayOut/2xDSSSDs

Attachment 1 - DSSSD bias & leakage current - OK

Attachment 2 - ADC data item stats

Attachment 3 - FEE64 temps - OK

Attachments 4-7 - aida01, 02, 03 & 04 2.*.W spectra

Noise observed for asics #1 & #2 of aida02 consistent with no cable attached - confirmed by visual inspection - ribbon cable disconnected from Samtec header of adaptor PCB

To Do list

1) re-connect ribbon cable to aida02 adaptor PCB
2) good quality ground to snout - use screws at inter-stage
3) install heavy duty grounds
4) ... 


Attachment 8 - adc data item stats - slow comparator 0x64

Attachment 9 - per FEE64 rate spectra - slow comparator 0x64

Attachment 10 - per FEE64 rate spectra - slow comparator 0xa

-

13:30 - Attach heavy duty ground to AIDA 

Attachment 11 - per FEE64 rate spectra - slow comparator 0xa

Try power cycle as system ground changed significantly now

Try to reattach aida04 TTY connector - still wrong

Archive old tty logs (ttyUSB*) to zip and delete them, makes easier to track startup 

Attachment 12 - per FEE64 rate spectra - slow comparator 0xa

No major difference, aida01 quieter than before

aida05 has some missing channels - loose ERNI?

Attachements 13 & 14 - Waveforms

Temporarily raise to 0x64 and test merger for DTAS synchro test later... all works good to MBS 
Change NETVAR MERGE.LinksAvailable to 8

Notice one copper screw touching aluminium, add some kapton tape behind it to isolate it. No difference

Voltage diff between aida02,aida06 < 1 mV
Voltage diff between copper bars < 0.5 mV (but not 0)

A comment from DTAS people that they observed a large increase in noise recently - perhaps from accelerator
May indicate current power/gronud oddities?

Attachment 15 - Rates at 0x1f (= 310 keV)

DSSSD#1 reasonable (aida04 a bit bad), but mostly just pulser (25 Hz)

DSSSD#2 much worse esp. p+n sides (aida05, aida07)

ERNI connectors pushed in and feel fully in

2x MSL type BB18(DS)-1000 installed


Adaptor PCB LK config

all FEE64s LK2, LK4 fitted
n+n FEE64s LK1 fitted
bottom p+n FEE64s LK3 fitted


AIDA PSU config

top 1-3 2-4
bottom 5-7 6-8


DSSSD bias -> p+n FEE64 core, n+n FEE64 braid


Heavy duty grounds not connected to FEE64 adaptor PCBs


ASIC settings 2021Apr29-13-16-00
 slow comparator 0xa

BNC PB-5 settings
 amplitude 1.000V
 attenuation x1
 tau_d 1ms
 frequency 25Hz
 polarity +


Attachment 1 - DSSSD bias & Leakage current - OK

Attachment 2 - FEE64 temps OK

Attachment 3 - adc data item stats

Attachments 4-9 - system wide checks OK *except* aida07 fails ADC calibration

Attachments 10 & 11 - 1.8.W spectra 20us FSR

Attachments 12 & 13 - 1.8.L spectra - aida03 pulser peak width 155 ch FWHM

Attachment 14 - per FEE64 rate spectra



19.15 Attachments 15-18 - 2*W spectra for aida01, 02, 03 &04

Attachment 19 - per FEE64 rate spectra

Attachment 20 - adc data item stats

DSSD stack will be as follows

1      3208-7  1019       125              upstream
2      3208-13 1020       120              downstream

First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance

3208-7: 
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

3208-13:
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

Conclusion: All bias bond wires are intact on both DSSDs

Visual inspection is both DSSDs look good with no damage or bond wire issues

Installed in snout: full stack

3 mm bPlas 1
17 mm Air
1 mm AIDA 1  (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2

The n+n FEEs were all removed to adjust the platform back to the single configuration

After some confusion we found the orientation of the frame to position the single detectors and moved back

Faulty AIDA cards aida05 and aida14 were removed to be returned backi (ASIC damage)

The cables for aida10 and aida12 were moved back to the left-right side and the n+n cards were installed and reocnnected

p+n cards moved to bring aida11 and aida09 above aida03/aida07 and aida01/aida05. Also move a different card to replace old aida05

While doing this it was noticed the HDMI for aida13 was very loose and actually fell off (I guess the cable was pulled incorrectly)
It has been removed and will eb sent back to be resoldered 

The copper grounding bars and ITEM frame were removed due to conflicting with the single snout layout
The bolts are slightly too short to hold the frame where it won't conflict so we loko for longer bolts to reattach the system
Some new ground cables will be made to reach the extra n+n FEEs

Aida13-Aida16 removed from the PSU 

fig1: layout of snout frame in "single" mode (alu block configuration)


MAC Addresses

aida01 - ba-8a
aida05 - 0d:15
aida09 - ee-01

aida04 - f6-b7
aida08 - ba-2b
aida12 - ba:89 

aida03 - d8-21
aida07 - 41:b4:0c
aida11 - f6-5a

aida02 - ba:22
aida06 - ee:72
aida10 - cf-ac [mistakenly swapped with aida12]

Second switch will stay in S4 and has now been screwed in properly below the switch 1 and the hdmi cables entrance

DHCP *not* updated

Relay is powered *off* so FEEs cannot be turned on, alL HV channels disabled

Tomorrow is a holiday so no work will be happening @ GSI

09:00 - Remove bias of DSSDs  and pwoer of FEEs

Remove snout from S4 so DEGAS frame can be mechnically removed

Snout now in FRS Messhuette 

17.40 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK

15.11 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK

16.31 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK

17.20 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps - OK

18.01 DAQ stopped

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachment 2 - DSSSD bias & leakage current

Attachment 3 - FEE64 temps - OK

Attachment 4 - ADC data item stats