Tests performed

• After leaving overnight noticed the pulser was not fully inserted into aida09
  • Little change in rates observed in 9 but otherwise nothing else
• Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
  • No change observed in rates
• Also tested covering the snout in light tight material
  • No Changes
• Unplugged the system monitor from 6 and 8, no change
• Connected the snout to ground with copper tape
  • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
  • Removed the ground and powercycled and rates restored to where previously
• Changed the bias supply jumper inside to have the bias line connected to the rack ground
  • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
    • Recrimped and inserted again
  • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
  • Pulser widths however larger than yesterday

p+n width

n+n width

• Connected up entire pulser circuit to n+n FEEs

Switching bias supply back

• because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
• DAQ was then left overnight as had to leave to catch the bus
• Rates are observed to be back at their higher values - attachment 2
• n+n 1.8.L spectra - attachment 3
• p+n 1.8.L spectra - attachment 4

p+n widths

n+n widths

• The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

n+n

• Don't see a pulser peak in 4 or 2
• Will replace cable between 4 and 2 and 2 and 6
• Statistics at 0x64 with pulser on - atachment 5
• Now see peaks - attachment 6
• Statistics at 0xa - attachment 7

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

• Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
• 0x64 stats - attachment 7
• 0xa statistics - attachment 8
• p+n 1.8.L - attachment 9
• p+n waveform - attachment 10
• n+n waveform - attachment 11
  • Same high frequency component in both
  • 35 channel spacing corresponds to a frequency of ~1.4MHz

Next tests

• Returned braid to FEE 4 and 8
  • Recovered earlier rates and performance
• Re-added the ground cable to the snout
  • Much worse noise performance across all FEEs
  • p+n rates in 200k
• Removed the ground from the snout
  • Got really good rates originally - attachment 12
  • Noticed 1 and 5 failed ADC calibration
  • Calibrated them and there rates increased massively - attachment 13
• Notice that the pulser peak in FEE2 and 6 keeps dropping out
  • As the cables have been changed multiple times thought it could be that an adaptor board was broken
  • 4 seemed the most likely suspect with its high rates and triple peaks
  • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
  • Pulser peaks were still muissing
  • Issue could be then with adaptor board 2 possibly

Conclusions

• Rates on the FEEs are in a better place now than they were at the start of the week
  • p+n in particular have shown a large amount of improvement
• There are still however issues with the n+n FEEs
• I think there is likely an issue with one of the adaptor boards
  • Possible aida02 at the pulser connection point
  • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
  • Would recomend swapping out aida02 as a test next week
• Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
  • Nic would like to use something other than clingfilm in future
  • One suggestion would be mylar which could be held in place with strips of double sided tape
  • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
• Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
• I am at a slight loss of what else to try

• I have left the daq running at 0x64 overnight to check for alphas
• Histograms and statistics zeroed

Grounding implemented

• Each FEE given direct connection to heavy duty earth bar via adaptor board lemo ground
• Earth bars are then connected to platform ground on AIDA frame
• Photo of grounding - attachment 1
• Improvements not as large as expected from previous experience though
  • Particularly in the n+n side where widths still appear awful
• While implementing the grounding it was noticed the ground lemo socket on FEE7 was damaged and no longer gripped onto a lemo cable
  • It was noted that it would need changing at some point but continued with it for the time being
• Jumper configuration was n+n LK1-4 all on, p+n LK2-4 all on

p+n widths

• Compared to the width with no grounding the FEEs for the first DSSD (First 6) range from no change to slightly worse
• 1.8.L spectra - attachment 3

n+n widths

• Widths still very large
• Width for 8 is around 500
• 1.8.L attachment 2

Additional info

• Statistics - attachment 4

Jumper optimisation

• Went to our previously found optimal jumper configuration for the trips
• n+n LK1 only
• p+n LK 2 and 4 on all FEE
• p+n LK3 on non isolated cables (PCB ground on lower middle cable and n+n side bias on top middle cable)
• While changing the jumpers we also swapped out the damaged adaptor board on aida07 for a new one
• Saw an improvement for most FEEs

• Unfortunately saw no improvement for the n+n
• We event tested having the pulser only going into a single FEE (aida08)
• Still have a width of 622 for that FEE
• To compare FEEs we have layout 1 - attachment 5
• Layouot 1 on a common timescale - attachment 6
  • Note that the rate is typically dominated by a single strip or a couple of strips within a FEE
  • Many channels are just showing the pulser rate 25Hz
• p+n  1.8.L attachment 7
• p+n waveforms attachment 8
• n+n waveforms attachment 9
• Testing the noise on the n+n strips we observe noise with slow comparator set to 200 (2MeV)
• Meanwhile the p+n strips are silent at 0x64 for the most part.
• DAQ left running overnight with 0x64 on ASIC settings to see if we observe any alphas

Plan for the day

• Test daisy chaining the n+n across to the other n+n strips
  • Currently the n+n braid is only plugged into either aida06 or aida08 (depending on DSSD)
  • All of the jumpers are however grounded
  • Will still test
  • May investigate also trying this on LK3 on adaptors for aida01 and aida03
• Observe the output of the new sum inverter on the scope to check for variation in amplitude

Pulser spectra

• Changes to the nn pulser circuit yesterday have improved the peak widths on the p+n side
• p+n peak widths averact around 100 channels
• n+n around 1000 at 0xa threshold
• n+n 1.8.L spectra - attachment 1
• p+n 1.8.L spectra attachment 2

p+n peak widths

n+n peak widths

Daily checks

• Fee temperatures ok - attachment 3
• Bias and leakage currents ok - attachment 4
• Stats spectra with current grounding - attachment 5
  • Current grounding is just a cable going between each adaptor board and its respective FEE
  • Interfee grounding done by power supply and also pulser circuit
• Rates for each of the asics - attachment 6
• p+n waveforms - attachment 7
• n+n waveforms - attachment 8
  • Lots of noise evident on them

Plans for the day

• Connect FEEs up to the copper earth bars installed
• Powercycle and check noise levels
• If n+n remains significantly higher investigate the n+n pulser circuit further
• Investigate the effects of different jumper connetions At present n+n have all LK jumpers on and p+n have lk2-4

An interesting find with the new sum inverter is that with the pulser at default settings the pulser peak on the n+n side is above the range of the ADC.
We have checked signals from the sum inverter and pulser and both are equal in magnitude.
We therefor determine it is over range as the charge is spread over 4 FEEs vs 12 on the p+n side.
This possibly wasn't an issue with the previous inverter if it produced a lower amplitude.

For tests going forward p+n voltage will be 2V. n+n voltage will be 0.5V - Attachment 1
Going to look into a 20dB attenuator

Seems there were issues with the lemo cables in the n+n pulser chain

DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
Temperature menu is still responsive and all FEEs report a temperature
Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
Merger shows no data arriving as expected - Attachment 3
Merger data buffers for each FEE - attachment 4
Merger time errors - attachment 5

DAQ has not yet been restarted in case Patrick wanted to look.

> DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
> Temperature menu is still responsive and all FEEs report a temperature
> Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
> Merger shows no data arriving as expected - Attachment 3
> Merger data buffers for each FEE - attachment 4
> Merger time errors - attachment 5
> 
> DAQ has not yet been restarted in case Patrick wanted to look.

Looking at the first attachment..... aida09 has rebooted. So it is not setup at all.

This has caused the NewMerger to hang .... which surprises me .... as I thought it was designed to spot a bad FEE and continue with the rest.

Yes I will have a look today as I'm in Daresbury.

More later ......

Required packages not included as default

• csh (For running MIDAS scripts)
• xinetd (Runs the rdate server for the FEEs)
• Telnet (Communication with the FEEs)
• dhcp

Configuration changes

• FEEs uses NFS2 which is no longer enabled by default
  • Must edit config in /etc/nfs.conf to uncomment the following lines:
    • [nfsd]
    • vers2=y
• /etc/xinetd.d/time-stream must be edited by uncommenting

  • disable        = no

• Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

  • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

  • Likewise can check status with `systemctl status $SERVICE_NAME`

• Also need to check the firewall status of the ethernet port connected to the FEEs.

  • Easiest solution is to set the port to be trusted

Optional extras

• For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
  • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
  • Can also add screenshot to this - Attachment 2
• Gnome tweaks also allows the number of workspaces to be changed
• Also recommend setting workspaces to span displays

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 

Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24

 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6

During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.

- The adaptor boards have no connection to the FEE ground over the ERNI connector
   - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
    - Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
    - With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
    - The cost of this is the rate of FEE8 goes up to 300k
    - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
   

> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.

Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.

Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.

launcher1
TclHttpd @8015 for AIDA
/MIDAS/Linux/startup/HTTPD

launcher2
TclHttpd @8115 for AIDA
/MIDAS/Linux/startup/HTTPD@8115

launcher3
TapeServer
/MIDAS/Linux/startup/TapeServer

launcher4
New Merger for AIDA
/MIDAS/Linux/startup/NewMerger

Launcher5
MBS Relay
/MIDAS/Linux/startup/datarelaymbs