DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
Temperature menu is still responsive and all FEEs report a temperature
Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
Merger shows no data arriving as expected - Attachment 3
Merger data buffers for each FEE - attachment 4
Merger time errors - attachment 5

DAQ has not yet been restarted in case Patrick wanted to look.

Required packages not included as default

• csh (For running MIDAS scripts)
• xinetd (Runs the rdate server for the FEEs)
• Telnet (Communication with the FEEs)
• dhcp

Configuration changes

• FEEs uses NFS2 which is no longer enabled by default
  • Must edit config in /etc/nfs.conf to uncomment the following lines:
    • [nfsd]
    • vers2=y
• /etc/xinetd.d/time-stream must be edited by uncommenting

  • disable        = no

• Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

  • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

  • Likewise can check status with `systemctl status $SERVICE_NAME`

• Also need to check the firewall status of the ethernet port connected to the FEEs.

  • Easiest solution is to set the port to be trusted

Optional extras

• For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
  • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
  • Can also add screenshot to this - Attachment 2
• Gnome tweaks also allows the number of workspaces to be changed
• Also recommend setting workspaces to span displays

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 

Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24

 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6

> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.

During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.

- The adaptor boards have no connection to the FEE ground over the ERNI connector
   - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
    - Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
    - With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
    - The cost of this is the rate of FEE8 goes up to 300k
    - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
   

Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.

Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.

launcher1
TclHttpd @8015 for AIDA
/MIDAS/Linux/startup/HTTPD

launcher2
TclHttpd @8115 for AIDA
/MIDAS/Linux/startup/HTTPD@8115

launcher3
TapeServer
/MIDAS/Linux/startup/TapeServer

launcher4
New Merger for AIDA
/MIDAS/Linux/startup/NewMerger

Launcher5
MBS Relay
/MIDAS/Linux/startup/datarelaymbs

Proxy for firefox and anydesk

Contents of an email from NH to OH and TD about tests performed on an additional bias supply:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

so I looked at the other supply in the lab (in a NIM crate there, cable direct to DSO)

Still see the same noise wave - also it looked a lot worse at 50V but that may be a worse NIM crate or something else - the current also wasn't 0 (calibration?)

More unusually the signals remain even if the NIM crate is off (DSO_Lab_NoP) and when the SHV is removed from the HV completely (DSO_Lab_NoCable)

> > 
> > 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
> >       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
> >       Switch Cooknell SA1 Sum Amp output from negative to positive
> > 
> >       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> > 
> >       This probably indicates
> >       - noisy Cooknell SA1
> >       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> > 
> > > Investigate noise situation of AIDA (no DSSSDs connected)
> > > 
> > > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > > - back-side aidas (02,04,06,08) worse resolution than front-side
> > > - high-frequency noise still noticeable in waveforms
> > > 
> > > Figs 7-8: Remove pulser in (from NIM rack) only
> > > - no noticable change
> > > 
> > > Figs 9-10: Remove all pulser cables from adapter boards
> > > - looks improved
> > > 
> > > Figs 11-12: HV Braid->GND Jumpers removed
> > > 
> > > Fig 13-14: All HV cables removed
> > > - aida06, aida14 isn't updating
> > > - very noticable improvement
> > > 
> > > Fig 15-16: Intermediate HV cables re-added
> > > - signal worse in FEE64s with HV cables
> > > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > > 
> > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > > 
> > > Pulser widths now:
> > > - aida14 (-ve amplitude) = 13.5 channels
> > > - aida08 (+ve amplitude) = 40 channels (?)
> 
> Removed the T-piece between pulser and summing amplifier
> pulser FWHM still ~30 channels - not the T-piece, probably summing amp
> 
> Checked the noise of HV and LV on battery DSO
> - LV - no noticeable noise on any channel (< 5 mV rms)
> - HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
>        independent of bias voltage 
>        See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
>                       AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2
> 
> Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
> 100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
> Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 

1nF/3k => 333kHz low pass rolloff

10nF/1M => 100Hz low pass rolloff

>  Check on one DSSSD, if see improvement can migrate to adapter boards in new revision
> 
> Things to do:
> TD:
>  Inline filters
> NH:
>  Check same behaviour other N1419ET HV supplies
>  Check line voltages from CAEN NIM bin

> 
> 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
>       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
>       Switch Cooknell SA1 Sum Amp output from negative to positive
> 
>       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> 
>       This probably indicates
>       - noisy Cooknell SA1
>       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> 
> > Investigate noise situation of AIDA (no DSSSDs connected)
> > 
> > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > - back-side aidas (02,04,06,08) worse resolution than front-side
> > - high-frequency noise still noticeable in waveforms
> > 
> > Figs 7-8: Remove pulser in (from NIM rack) only
> > - no noticable change
> > 
> > Figs 9-10: Remove all pulser cables from adapter boards
> > - looks improved
> > 
> > Figs 11-12: HV Braid->GND Jumpers removed
> > 
> > Fig 13-14: All HV cables removed
> > - aida06, aida14 isn't updating
> > - very noticable improvement
> > 
> > Fig 15-16: Intermediate HV cables re-added
> > - signal worse in FEE64s with HV cables
> > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > 
> > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > 
> > Pulser widths now:
> > - aida14 (-ve amplitude) = 13.5 channels
> > - aida08 (+ve amplitude) = 40 channels (?)

Removed the T-piece between pulser and summing amplifier
pulser FWHM still ~30 channels - not the T-piece, probably summing amp

Checked the noise of HV and LV on battery DSO
- LV - no noticeable noise on any channel (< 5 mV rms)
- HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
       independent of bias voltage 
       See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
                      AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2

Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 
 Check on one DSSSD, if see improvement can migrate to adapter boards in new revision

Things to do:
TD:
 Inline filters
NH:
 Check same behaviour other N1419ET HV supplies
 Check line voltages from CAEN NIM bin