12:37 DAQ found crashed. Had crashed at some time following 7:50am following previous statistics update
      Unable to recover the FEEs so forced to power cycle
      Following power cycle statistics much improved from yesterday.
      In the waveforms the lower frequency component that was prevalent yesterday is no longer visible.
      The waveforms are not perfect by any means though
      Also note that following an ASIC synchronisation all of the FEEs lose ADC calibration and must be manually recalibrated.

16.42 UTC+1

      per FEE64 1.8.W spectra

      20us FSR attachments 7 & 8
      200us FSR attachments 9 & 10
      2ms FSR attachments 11 & 12
      20ms FSR attachments 15 & 16


      rate spectra attachment 13
      good events statistics attachment 14

NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290

14.40 Rate spectra - attachment 1
      adc data items - attachment 2
       slow comparator 0xa
       stats for p+n much improved, n+n remain poor
      1.8.W spectra - attachments 3 & 4
      1.8.L spectra - attachments 5 & 6
       pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM

17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7

08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8

On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen.
We were also unable to connected to MIDAS via ssh port forwarding.

Since the restart we have been unable to write to file.

The tapeserver will allocate/mount/open a file but nothing will be written to it.
The merger will be receiving data but never shows as having any "Current links with data".

/MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23  2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119

The current disk structure is:
[npg@aidas-gsi npg]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
sdb      8:16   0   7.3T  0 disk 
├─sdb1   8:17   0   200M  0 part 
├─sdb2   8:18   0   500M  0 part 
└─sdb3   8:19   0   7.3T  0 part 
  ├─vg_aidas2-lv_root (dm-2)
       253:2    0    50G  0 lvm  
  ├─vg_aidas2-lv_home (dm-3)
       253:3    0   7.2T  0 lvm  /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
  └─vg_aidas2-lv_swap (dm-4)
       253:4    0   7.8G  0 lvm  
sdc      8:32   0   7.3T  0 disk 
└─sdc1   8:33   0   7.3T  0 part /media/ThirdDrive
sdd      8:48   0   7.3T  0 disk 
└─sdd1   8:49   0   7.3T  0 part /media/SecondDrive
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0   200M  0 part /boot/efi
├─sda2   8:2    0   500M  0 part /boot
└─sda3   8:3    0 465.1G  0 part 
  ├─vg_aidasgsi-lv_root (dm-0)
       253:0    0    50G  0 lvm  /
  ├─vg_aidasgsi-lv_swap (dm-1)
       253:1    0   7.8G  0 lvm  [SWAP]
  └─vg_aidasgsi-lv_home (dm-5)
       253:5    0 407.3G  0 lvm  /home


Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3

TapeServer config file - attachment 4
Merger options - attachment 5

Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png

08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)

10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11

08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.

      They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
      They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID

08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2

09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
      Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
      Data rate 16.3MB/s
      ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.

09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage

09:47 Beam back turned off no storage. On file R1_10

11:04 Now that we have implants on the detector we get a better look at the strips
      From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
      This adaptor board corresponds to FEE11

12:17 Leakage currents screenshot 512.png

13:30 Beam off. Expected back at 15:00...

14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
      They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
      They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
      Rates are as before. No better, no worse.

15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...

08:00 OH Takes over

      Statistics still look good from yesterday. Rate is still down.
      Dead time in FEEs 2 and 4 is 2.04%

      System wide checks clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x500 : 	 38  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      *Baseline reset*
      FPGA check ok
      Stats - attachment 1
      Temp - attachment 2
      Bias - attachment 3
      

08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips.
      I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4
      The results of the test appear to indicate that it is the network link that we are saturating causing dead time
***	Before		                  After	
FEE	DT (s)  Dead time %	           DT (s)	Dead time %
0	0.013	0.004184451	           0.026	0.010025681
1	6.352	2.04458693	           4.097	1.57981599
2	0.295	0.09495484	           0.292	0.112596112
3	6.349	2.043621288	           8.255	3.183153771
4	0.021	0.006759497	           0.03	        0.011568094
5	0.285	0.091736032	           3.388	1.306423377
6	0.02	0.006437616	           0.149	0.057454865
7	1.219	0.392372712	           2.133	0.822491459
8	1.137	0.365978485	           0.457	0.176220627
9	0	0	                       0	0
10	1.633	0.525631369	           1.545	0.595756823
11	0.061	0.01963473	           0.166	0.064010118
12	0.059	0.018990968	           0.085	0.032776265
13	0	0	                       0	0
14	0.016	0.005150093	           0.023	0.008868872
15	0.282	0.09077039	           0.168	0.064781324


      Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased

09:00 They are stopping the run to decrease the bias on the bPlast detector
09:25 No noticable change in AIDA
      They will now begin to change the spill structure.

09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change

10:21 Beam is back on AIDA

10:39 System wide checks. 
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x506 : 	 6  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

11:00 
      Stats - attachment 4
      Temp - attachment 5
      Bias - attachment 6

12:46 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50b : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

14:36 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50f : 	 15  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

DSSSD bias OFF. DAQ STOPped

Attachment 6 FEE64 temps OK

Attachment 1-5 system wide checks OK 

Re-assembled triple snout re-installed on the AIDA stand - DSSSD ribbon cables and grounds re-connected.
FEE64 adaptor PCB LKs per https://elog.ph.ed.ac.uk/DESPEC/409

Attachments 1 & 2 1.8.L spectra
  p+n FEE64 aida10 pulser peak width 97 ch FWHM
  n+n FEE64 aida04 pulser peak width 550 ch FWHM

Attachment 3 rate spectra
  missing channels aida01, aida07 & aida11 - misaligned ERNI and/or Samtec connectors?

Attachments 4 & 5 1.8.W spectra 

Attachment 6 ADC data item rates
 aida10 OK, all others c. 300-400k

Attachment 7 FEE64 temperatures OK

Attachment 8 DSSSD bias & leakage currents OK

08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
      It is currently sending data pause requests and data push requests but nothing else - attachment 1
      telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
      FEE appears to have hung. - attachment 2
      Still passes all system wide checks.
      When performing a check load can see the tclsh process take up CPU usage

09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
      AIDA14 recovered but ASIC2 still missing
    
      Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
      Given access would change the adapter board on AIDA05.
      Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
      Would retain the option of a speedybacktrack

      Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
      A threshold of 0x4 reduced the rate in the HEC channels considerably
      A threshold of 0x6 removes entirely

      Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned

09:59 Had access so replaced the adapter card on AIDA05
      Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
      This could also have been the cause


10:09 Restarted DAQ and there is no change to the behavior of AIDA05
      Still see events in HEC
      Again raising threshold to 0x5 in ASIC1 largely removes them. 

13.10 Attachments 3-5
      DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
      ASIC check load
      ASIC slow comparator all FEE64s all ASICS 0xa
      ADC data item stats

13.17 Attachments 6-9
      DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
      ADC data item stats
      NewMerger/Tape Server

18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
      It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2

      To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
      After powering on it was found there was no improvement and possible worse noise than before

      We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14

ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2

 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6

Required packages not included as default

• csh (For running MIDAS scripts)
• xinetd (Runs the rdate server for the FEEs)
• Telnet (Communication with the FEEs)
• dhcp

Configuration changes

• FEEs uses NFS2 which is no longer enabled by default
  • Must edit config in /etc/nfs.conf to uncomment the following lines:
    • [nfsd]
    • vers2=y
• /etc/xinetd.d/time-stream must be edited by uncommenting

  • disable        = no

• Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

  • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

  • Likewise can check status with `systemctl status $SERVICE_NAME`

• Also need to check the firewall status of the ethernet port connected to the FEEs.

  • Easiest solution is to set the port to be trusted

Optional extras

• For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
  • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
  • Can also add screenshot to this - Attachment 2
• Gnome tweaks also allows the number of workspaces to be changed
• Also recommend setting workspaces to span displays

An interesting find with the new sum inverter is that with the pulser at default settings the pulser peak on the n+n side is above the range of the ADC.
We have checked signals from the sum inverter and pulser and both are equal in magnitude.
We therefor determine it is over range as the charge is spread over 4 FEEs vs 12 on the p+n side.
This possibly wasn't an issue with the previous inverter if it produced a lower amplitude.

For tests going forward p+n voltage will be 2V. n+n voltage will be 0.5V - Attachment 1
Going to look into a 20dB attenuator

Seems there were issues with the lemo cables in the n+n pulser chain

Pulser spectra

• Changes to the nn pulser circuit yesterday have improved the peak widths on the p+n side
• p+n peak widths averact around 100 channels
• n+n around 1000 at 0xa threshold
• n+n 1.8.L spectra - attachment 1
• p+n 1.8.L spectra attachment 2

p+n peak widths

n+n peak widths

Daily checks

• Fee temperatures ok - attachment 3
• Bias and leakage currents ok - attachment 4
• Stats spectra with current grounding - attachment 5
  • Current grounding is just a cable going between each adaptor board and its respective FEE
  • Interfee grounding done by power supply and also pulser circuit
• Rates for each of the asics - attachment 6
• p+n waveforms - attachment 7
• n+n waveforms - attachment 8
  • Lots of noise evident on them

Plans for the day

• Connect FEEs up to the copper earth bars installed
• Powercycle and check noise levels
• If n+n remains significantly higher investigate the n+n pulser circuit further
• Investigate the effects of different jumper connetions At present n+n have all LK jumpers on and p+n have lk2-4

Grounding implemented

• Each FEE given direct connection to heavy duty earth bar via adaptor board lemo ground
• Earth bars are then connected to platform ground on AIDA frame
• Photo of grounding - attachment 1
• Improvements not as large as expected from previous experience though
  • Particularly in the n+n side where widths still appear awful
• While implementing the grounding it was noticed the ground lemo socket on FEE7 was damaged and no longer gripped onto a lemo cable
  • It was noted that it would need changing at some point but continued with it for the time being
• Jumper configuration was n+n LK1-4 all on, p+n LK2-4 all on

p+n widths

• Compared to the width with no grounding the FEEs for the first DSSD (First 6) range from no change to slightly worse
• 1.8.L spectra - attachment 3

n+n widths

• Widths still very large
• Width for 8 is around 500
• 1.8.L attachment 2

Additional info

• Statistics - attachment 4

Jumper optimisation

• Went to our previously found optimal jumper configuration for the trips
• n+n LK1 only
• p+n LK 2 and 4 on all FEE
• p+n LK3 on non isolated cables (PCB ground on lower middle cable and n+n side bias on top middle cable)
• While changing the jumpers we also swapped out the damaged adaptor board on aida07 for a new one
• Saw an improvement for most FEEs

• Unfortunately saw no improvement for the n+n
• We event tested having the pulser only going into a single FEE (aida08)
• Still have a width of 622 for that FEE
• To compare FEEs we have layout 1 - attachment 5
• Layouot 1 on a common timescale - attachment 6
  • Note that the rate is typically dominated by a single strip or a couple of strips within a FEE
  • Many channels are just showing the pulser rate 25Hz
• p+n  1.8.L attachment 7
• p+n waveforms attachment 8
• n+n waveforms attachment 9
• Testing the noise on the n+n strips we observe noise with slow comparator set to 200 (2MeV)
• Meanwhile the p+n strips are silent at 0x64 for the most part.
• DAQ left running overnight with 0x64 on ASIC settings to see if we observe any alphas

Plan for the day

• Test daisy chaining the n+n across to the other n+n strips
  • Currently the n+n braid is only plugged into either aida06 or aida08 (depending on DSSD)
  • All of the jumpers are however grounded
  • Will still test
  • May investigate also trying this on LK3 on adaptors for aida01 and aida03
• Observe the output of the new sum inverter on the scope to check for variation in amplitude

Tests performed

• After leaving overnight noticed the pulser was not fully inserted into aida09
  • Little change in rates observed in 9 but otherwise nothing else
• Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
  • No change observed in rates
• Also tested covering the snout in light tight material
  • No Changes
• Unplugged the system monitor from 6 and 8, no change
• Connected the snout to ground with copper tape
  • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
  • Removed the ground and powercycled and rates restored to where previously
• Changed the bias supply jumper inside to have the bias line connected to the rack ground
  • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
    • Recrimped and inserted again
  • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
  • Pulser widths however larger than yesterday

p+n width

n+n width

• Connected up entire pulser circuit to n+n FEEs

Switching bias supply back

• because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
• DAQ was then left overnight as had to leave to catch the bus
• Rates are observed to be back at their higher values - attachment 2
• n+n 1.8.L spectra - attachment 3
• p+n 1.8.L spectra - attachment 4

p+n widths

n+n widths

• The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

n+n

• Don't see a pulser peak in 4 or 2
• Will replace cable between 4 and 2 and 2 and 6
• Statistics at 0x64 with pulser on - atachment 5
• Now see peaks - attachment 6
• Statistics at 0xa - attachment 7

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

• Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
• 0x64 stats - attachment 7
• 0xa statistics - attachment 8
• p+n 1.8.L - attachment 9
• p+n waveform - attachment 10
• n+n waveform - attachment 11
  • Same high frequency component in both
  • 35 channel spacing corresponds to a frequency of ~1.4MHz

Next tests

• Returned braid to FEE 4 and 8
  • Recovered earlier rates and performance
• Re-added the ground cable to the snout
  • Much worse noise performance across all FEEs
  • p+n rates in 200k
• Removed the ground from the snout
  • Got really good rates originally - attachment 12
  • Noticed 1 and 5 failed ADC calibration
  • Calibrated them and there rates increased massively - attachment 13
• Notice that the pulser peak in FEE2 and 6 keeps dropping out
  • As the cables have been changed multiple times thought it could be that an adaptor board was broken
  • 4 seemed the most likely suspect with its high rates and triple peaks
  • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
  • Pulser peaks were still muissing
  • Issue could be then with adaptor board 2 possibly

Conclusions

• Rates on the FEEs are in a better place now than they were at the start of the week
  • p+n in particular have shown a large amount of improvement
• There are still however issues with the n+n FEEs
• I think there is likely an issue with one of the adaptor boards
  • Possible aida02 at the pulser connection point
  • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
  • Would recomend swapping out aida02 as a test next week
• Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
  • Nic would like to use something other than clingfilm in future
  • One suggestion would be mylar which could be held in place with strips of double sided tape
  • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
• Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
• I am at a slight loss of what else to try

• I have left the daq running at 0x64 overnight to check for alphas
• Histograms and statistics zeroed

09:59 Noticed that the right hand ribbon cable for aida07 was offset vertically by 1. i.e. Only 1 of the 2 rows of pins in. Unbiassed detectors and reconnected.
      Rebiassed detectors - leakage current on DSSD2 increased to 9.96uA - Attachment 1

      Power cycling FEEs

      Following restart the FEEs show some improvement in rates - attachment 2
      Rate spectra - attachment 3
      Still issues with aida11 in the rate spectra (Likely a misaligned ERNI connector)



      Pulser on rate spectra - attachment 4
      Pulser off rate spectra - attachment 5
      We can see that with the pulser on there are events in the channels in aida01 and aida11
      With the pulser off these channels disappear.
      ERNI connector is then likely aligned the issue is more likely the DSSD ribbon cable is misaligned.

      Taking bias off to investigate.


10:39 Could not observe anything out of alignement about either the connectors or the adaptor boards
      Note that after pushing on the connector AIDA01 had largely recovered

13:30 Top of snout taken off while mounted in S4. It was observed that one of the ribbon cables for aida11 was off by 1 row.
      This was corrected in situ and the snout was put back together and made light tight again.
      Oddly leakage current still the same - DSSD2 has come down though - attachment 6
      FEEs powercycled for new tests

      Half the channels in aida11 had appeared again
      Took apart once more and found the other header also off by one row which has been corrected

      FEEs powercycled for new test

      All of aida11 now showing - attachment 7
      Statistics at 0xa shown in attachment 8
      All p+n for DSSD1 are good (aida01, 03, 09, 10, 11 and 12)
      p+n for DSSD2 are so so (aida05, 07, 13, 14, 15 and 16)
      All n+n 2, 4, 6 and 8 are poor still


14:50 p+n waveforms - attachment 9
      n+n waveforms - attachment 10

      Current jumper configuration
      LK1 (Bias ground) on FEEs 6 and 8
      LK2&4 on all p+n
      LK3 on aida03 and aida07 (Bottom middle)

15:00 Investigated whether there was a light leak. Placed black material over snout.
      Leakage on DSSD2 dropped to 5.6 - attachment 11
      There was a light leak
      Stats have decreased slightly - attachment 12

16:00 Move raspberry pis atop 5mm alminimum plate to keep away from FEE64 PSUs
      No change

16:30 Remove pulser network from all FEEs 
      Rates get noticably worse in many FEEs... why?
      Stats - attachment 13

      Possible ideas include braid touching things it shouldn't (Adapter PCBs?) 

Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23