AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 11 of 35  ELOG logo
IDdown Date Author Subject
  498   Fri Jun 24 22:36:45 2022 TDSaturday 25 June 00:00-08:00
23:36 Check ASIC contorol
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_154 - attachment 1
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

stats - attachments 2-6

per FEE64 1.8.H spectra - attachments 7 & 8

per p+n FEE64 1.8.L spectra - attachment 9
 aida01 pulser peak width 101 ch FWHM

per FEE64 stat & rate spectra - attachments 10 & 11

adc data item stats - attachment 12

FEE64 temps OK - attachment 13

DSSSD bias & leakage currents OK - attachments 14 & 15

00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17


03:03

analysis of file S505/R5_189 - attachment 18
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20

adc data item stats - attachment 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage currents OK - attachment 23

03:30

per FEE64 1.8.H spectra - attcahments 24 & 25


06:09 no beam 

analysis of file S505/R5_221 - attachment 26
 zero timewarps
 all dead times << 1%

all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28

adc data item stats - attachment 29

FEE64 temps OK - attachment 30

DSSSD bias & leakage currents OK - attachment 31
  497   Fri Jun 24 15:04:06 2022 MarcFriday 24th June - evening shift

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

 

 

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

 

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

 

 

  496   Fri Jun 24 06:56:06 2022 OHFriday 24 June 008:00-16:00
07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16
  495   Fri Jun 24 00:34:55 2022 AMFriday 24 June 00:00-08:00

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good

  494   Thu Jun 23 15:10:17 2022 Philippos PapadakisThursday 23 June 16:00-24:00
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc525 : 	 39  
aida08 fault 	 0xf0e9 : 	 0xf166 : 	 125  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
                 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x23 : 	 9  

Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)


18:05
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc527 : 	 41  
aida08 fault 	 0xf0e9 : 	 0xf177 : 	 142  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x24 : 	 10  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)


20:22
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf17f : 	 150  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x20 : 	 15  
aida08 fault 	 0x1a : 	 0x25 : 	 11  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)

21:56
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf186 : 	 157  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x26 : 	 12  

Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)

23:50
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc52e : 	 48  
aida08 fault 	 0xf0e9 : 	 0xf18c : 	 163  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
	
			 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x27 : 	 13  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15)
  493   Thu Jun 23 06:56:02 2022 OHThursday 23 June 08:00-16:00
07:56 Taken over from AM
      Current White Rabbit Error status:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51a : 	 28  
      aida08 fault 	 0xf0e9 : 	 0xf132 : 	 73  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA error status:
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link

      Statistics ok - attachment 1
      FEE temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      Analysis of file R3_93 - attachment 4
      Still deadtime in aida02 of ~15%
      Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
      
      Currently 2.2 TB of space on /media/SecondDrive
      Current data rate of 6.23MB/s
      At current data rate this should last 4 days

08:27 Currently no beam
      Problem with Alvarez 3 now
      Tomorrow will have a 1 hour break due to changing of the source

09:36 Beam back

09:59 System wide checks
		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51b : 	 29  
      aida08 fault 	 0xf0e9 : 	 0xf138 : 	 79  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  

      Stats ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7

11:14 FRS group is changing from Hg setting back to Pt

11:36 Back on Pt R3_129

14:14 Stats ok - attachment 8
      Temperature ok - attachment 9
      Bias and leakage currents ok - attachment 10
      System wide checks:
      WR:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc523 : 	 37  
      aida08 fault 	 0xf0e9 : 	 0xf156 : 	 109  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x1b : 	 10  
      aida08 fault 	 0x1a : 	 0x20 : 	 6

      Analysis of R3_156 - attachment 11  
  492   Wed Jun 22 23:18:45 2022 AM, OH, TD, MAThursday 23 June
00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
      Attempted to recover restarting merger but this caused 01 to drop out.

00:10 DAQ recovered and AIDA01 recovered with a reboot


01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good

03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good

05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good

07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good
  491   Wed Jun 22 15:09:37 2022 MAWednesday 22 June 16:00-00:00

16:00 Take over from Magda mrning shift

Beam was off of sometime for tunning to Mercurey setting.

17:00 attached 1-3

19:00 attached 4-6

21:00 attached 7-9

23:30 10-12

 

  490   Wed Jun 22 10:09:12 2022 PJCSINFO: FEE64 supply Voltages

Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

  489   Wed Jun 22 07:12:46 2022 MSWednesday 22 June 8:00-16:00

Took over the night shift from Tom.

8:10 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.

12:19 OH Disabled all discriminators to reduce the data rate.

13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10

14:00 : attachments 11-13

16:00 : attachments 14-16

 

  488   Tue Jun 21 23:29:16 2022 TDWednesday 22 June 00:00-08:00
00.28 all histograms & stats zero'd
      baseline counters

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

BNC PB-5 pulser
 to p+n FEE64s only
  amplitude 1.0V
  attenuator x1
  decay time 1ms
  polarity +
  frequency 2Hz

all AD9252 waveform ADCs disabled

Tape Server -> no storage mode

00.36 check ASIC control all FEE64s all ASICs

      all system wide checks OK

00.41 BNC PB-5 pulser rate increased from 2Hz to 22Hz per request by HA/NH - to ensure DTAS correlation checker has sufficient rate of identified pulser events 

per p+n FEE64 1.8.L spectra - attachment 1
 aida01 pulser peak width 88 ch FWHM

per FEE64 rate spectra - attachment 2

adc, pause and correlation scaler data items - push, aida01 and aida02 stats - attachments 3-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage current OK - attachments 10 & 11

02.28 aida05 zero adc data items, unable to halt aida05 - all other FEE64s OK

      restart NewMerger -> now able to halt aida05

      DAQ go OK, merge rate c. 1.4M data items/s

per FEE64 1.8.H spectra - attachments 12 & 13
 approx flat distribution to c. 10GeV


adc data items stats OK - attachments 14

FEE64 temps OK - attachment 15

DSSSD bias & leakage current OK - attachments 16 & 17

02.55 Some problems with GO4 data analysis for PID (FRS PID looks OK). 204Pt setting. Will commence writing data to disk file S505/R2_0

04.12 per FEE64 1.8.H spectra - attachments 18 & 19
      c. 9GeV peak in upstream DSSSDs (aida01, 2, 3 & 4) indicating (high energy) heavy fragments are stopping
      Low energy events near c. 200MeV lower mass, high energy, low energy loss - fission fragments?

04.20 per FEE64 stats spectra - attachment 20
      indicates implant x-y distribution - off centre (more significantly in x than in y) but DSSSDs appear to see most of distribution
      remember raw distro will have significant fraction of fission fragments - may be misleading - need to gate on high energy events to see x-y distro of ions of 
interest

adc data items stats OK - attachments 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage current OK - attachments 23



06.10 per FEE64 rate spectra - attachment 24

per FEE64 1.8.H spectra - attachments 25 & 26

per FEE64 stats spectra - attachment 27
     
adc data items stats OK - attachments 28

FEE64 temps OK - attachment 29

system wide checks OK *except* WR & FPGA errors - see attachments 30 & 31

DSSSD bias & leakage current OK - attachments 32

07.52 analysis of file S505/R2_47 - attachment 33
      all FEE64 dead times < 1% *except* aida02 c. 7%
  487   Tue Jun 21 10:19:05 2022 OH MLTuesday 21st June 08:00-00:00
11:19 Stats ok 0xa on all FEE - attachment 1
      Current pulser settings - attachment 2
      FEE Temperatures ok - attachment 3
      Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
      Layout 1 after recovering channels - attachment 4
      New stats (Note increased rate on 3 and 7) attachment 5
      Bias and leakage currents ok - attachment 6

      S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.

14:05 Have had it confirmed that the flange beam dump is in position in S4

15:43 Stats ok -attachment 7
      Temp ok - attachment 8
      Bias and Leakage current ok - attachment 9


17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
      As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
      During the experiment this will be changed to SC1 trigger to solve this.

19:49: Stats ok - attachment 10
       temperatures ok - attachment 11
       Bias - Leakage current ok - attachment 12
  486   Mon Jun 20 09:26:23 2022 OH, TDMonday 20 June
10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11
  485   Sun Jun 19 19:55:20 2022 NHaida03 and aida04
aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour
  484   Sun Jun 19 12:36:10 2022 OHSunday 19th June
Yesterday the sum/inverter was found to be a considerable source of noise in the system.

Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace

Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow

Put inverter back in CAEN crate in S4 and confirm similar behavior.

Move CAEN crate to new mains port, same behaviour

Move CAEN crate to AC voltage stabiliser, same behaviour

Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears  - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears

Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.

MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd

CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?

MACBs and inverter checked on another crate - no signal

Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected

Tested adding bias filters - No change

15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
      We get ok stats even with waveforms on at 0xa- attachment 5

      Waveforms still have the 100kHz oscillation in them though attachments 6 and 7

      Without waveforms we get similar performance to before the switch to the new mains - attachment 8

15:56 Move CAEN NIM crate across to this mains network
      300+kHz on all FEEs with waveforms on. Can recover waveform off performance though

16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.

17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9
  483   Sun Jun 19 00:22:24 2022 TDSunday 19 June
01.20 

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

All waveform ADCs disabled

BNC PB-5 pulser
 to p+n FEE64s only

all system wide checks OK *except* ADC calibration, WR decoder & FPGA timestamp errors - see attachments 1 & 2

DSSSD bias & leakage current OK - see attachments 3 & 4

FEE64 temps OK - see attachment 5

statistics OK - ADC data, pause, resume & correlation scaler data items
                data push, data flush, aida01, aida02 - see attachments 6-13

per FEE64 1.8.L spectra - see attachment 14
 aida01 pulser peak width 73 ch FWHM
  482   Sat Jun 18 09:46:57 2022 OH, NHFriday 17th June
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23
  481   Tue Jun 14 12:15:33 2022 NHS450 Archived
All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 
  480   Mon Jun 13 16:38:37 2022 NHMon 13 Jun
Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change
  479   Fri Jun 10 10:32:46 2022 NHFri 10 Jun
Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 
ELOG V3.1.4-unknown