AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 17 of 36  ELOG logo
ID Date Author Subject
  393   Tue Feb 1 15:00:28 2022 NHAIDA CentOS 7 Startup
Starting up AIDA on CentOS 7 to confirm things work (based on some CARME experience)

NFS settings:
/etc/exports copied over from old disk
line to old /MIDAS_130718 export removed (commented) to see if necessary to copy over
nfsv2 enabled in nfs.conf
NFS service enabled and started (systemctl enable nfs-server ; systemctl start nfs-server)

DHCP setting:
DHCP was disabled, enabled and started (systemctl enable dhcpd ; systemctl start dhcpd)
Raspberry Pis pick up IP and become responsive again

Firwall setting:
Interface p4p1 (AIDA switch #1) is changed to zone 'trusted' so that FEEs can connect to NFS
(panicked before unable to get RPC port)

See FEEs starting to boot!
Getting both / and /MIDAS successfully (note confirming that old /MIDAS_130718 is not needed)

FEEs 4 and 9 take longer to boot others (observed before, seems to be PSU slots 4/5 in first take a while to provide voltage)
FEE9 seems to panic after mounting root, waiting to see if it reboots itself (no serial)
... Nope but power cycle fixes it

All FEEs online and Reset/Setup completed... Temps OK

Note 1: CentOS 7 uses systemd, monitoring log is best done as
`journalctl -f` as root.
npg not in sudoers (should be done)

Note 2: System used as graphical terminal for Edinburgh double-alpha experiment until Feb. 11
AIDA operated in background and using port forwarding to access MIDAS/Web mostly
UI has weird workspace configuration by default needs changing to better cope with AIDA windows

Note 3:
Only 4 FEEs are producing serial data to the Pi_Monitor
Probably 2 of the USB hubs were knocked lose (from the USB extender cables) when the Fatima/HPGe cables were removed last year
S4 access is limited but will try to reconnect ASAP

Note 4:
AIDA setup unfortunately quite limited in functionality due to other running experiments:
WR is unavailable as VETAR VME crate was powered off
Not all FEEs are connected to pulser or grounded. Some cables were removed during testing last year.
Some ground cables removed and pulser inverter removed as needed at CARME and S4 access is now limited for WASA tests
  392   Fri Oct 29 23:00:04 2021 NHGround cable information
For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 
  391   Fri Oct 29 10:49:01 2021 OHNoise tests - 28th October -29th October
Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24
  390   Wed Oct 27 15:12:50 2021 OH, NHNoise tests - 27th October
 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6
  389   Tue Oct 26 16:08:46 2021 NHFEE and Adaptor board groundings
> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.
  388   Thu Oct 21 16:18:51 2021 OHFEE and Adaptor board groundings
During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.

- The adaptor boards have no connection to the FEE ground over the ERNI connector
   - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
    - Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
    - With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
    - The cost of this is the rate of FEE8 goes up to 300k
    - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
   
  387   Thu Oct 21 09:33:08 2021 OHNoise tests - 21st Oct 2021
Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.
  386   Thu Oct 21 09:08:41 2021 OHMIDAS start up messages
  385   Wed Oct 20 13:26:08 2021 NH, OHAIDA Noise + Filtering
Turn on AIDA (SL6) to investigate noise situation and impact of new RC filters on bias input

Current setup:
- No Detectors
- Pulser in fee14, fee8 *only*, no bias connected. FEEs isolated from each other

Pulser widths::
p+n grounds connected via bias link (no connection to HV bias itself)

ASICFEE14 WidthFEE8 Width
1.8.L14.4121.86
2.8.L13.0221.87
3.8.L12.5721.52
4.8.L13.7523.59


Note: FEE8 is connected via SA1 inverter -> cause of worse resolution

Plug bias into outer FEEs (14 + 8 + chaint o 6)

ASICFEE14 WidthFEE8 Width
1.8.L14.5924.43
2.8.L12.7322.25
3.8.L12.7623.06
4.8.L14.3623.31


Long run over lunch:

ASICFEE14 WidthFEE8 Width
1.8.L14.4725.52
2.8.L12.8324.26
3.8.L12.7524.15
4.8.L14.5826.93

Figures 1 & 2

Add HV filters

ASICFEE14 WidthFEE8 Width
1.8.L14.3423.53
2.8.L12.6723.82
3.8.L12.7823.59
4.8.L13.8825.76

Figures 3 & 4

Small improvement maybe?

Figs 5&6 - Oscilloscope before & after HV filter added

--

Plans for tomorrow (21/10/21)

- Reintroduce pulser cables through all FEEs to see impact on noise (maybe improved now)
- Measure pulser noise in battery DSO (+ other sourceS?)
- Add a DSSD and see impact on noise with HV filter
- Resistance from FEE64 adapter to mechanical ground
- Check pulser on the battery powered scope (Is the signal common to the power supply)
  384   Tue Oct 19 16:17:49 2021 OHAIDA-gsi server to centos 7
Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.
  383   Mon Oct 18 14:20:32 2021 OHTop bar launchers
launcher1
TclHttpd @8015 for AIDA
/MIDAS/Linux/startup/HTTPD

launcher2
TclHttpd @8115 for AIDA
/MIDAS/Linux/startup/HTTPD@8115

launcher3
TapeServer
/MIDAS/Linux/startup/TapeServer

launcher4
New Merger for AIDA
/MIDAS/Linux/startup/NewMerger

Launcher5
MBS Relay
/MIDAS/Linux/startup/datarelaymbs
  382   Mon Oct 18 13:03:23 2021 OHDHCPd.conf backup
  381   Mon Oct 18 13:01:04 2021 OHProxy config
Proxy for firefox and anydesk
  380   Fri Sep 3 13:39:42 2021 OHAdditional noise tests performed at GSI
Contents of an email from NH to OH and TD about tests performed on an additional bias supply:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

so I looked at the other supply in the lab (in a NIM crate there, cable direct to DSO)

Still see the same noise wave - also it looked a lot worse at 50V but that may be a worse NIM crate or something else - the current also wasn't 0 (calibration?)

More unusually the signals remain even if the NIM crate is off (DSO_Lab_NoP) and when the SHV is removed from the HV completely (DSO_Lab_NoCable)
  379   Sun Aug 22 18:40:08 2021 TDAIDA Noise
> > 
> > 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
> >       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
> >       Switch Cooknell SA1 Sum Amp output from negative to positive
> > 
> >       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> > 
> >       This probably indicates
> >       - noisy Cooknell SA1
> >       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> > 
> > > Investigate noise situation of AIDA (no DSSSDs connected)
> > > 
> > > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > > - back-side aidas (02,04,06,08) worse resolution than front-side
> > > - high-frequency noise still noticeable in waveforms
> > > 
> > > Figs 7-8: Remove pulser in (from NIM rack) only
> > > - no noticable change
> > > 
> > > Figs 9-10: Remove all pulser cables from adapter boards
> > > - looks improved
> > > 
> > > Figs 11-12: HV Braid->GND Jumpers removed
> > > 
> > > Fig 13-14: All HV cables removed
> > > - aida06, aida14 isn't updating
> > > - very noticable improvement
> > > 
> > > Fig 15-16: Intermediate HV cables re-added
> > > - signal worse in FEE64s with HV cables
> > > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > > 
> > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > > 
> > > Pulser widths now:
> > > - aida14 (-ve amplitude) = 13.5 channels
> > > - aida08 (+ve amplitude) = 40 channels (?)
> 
> Removed the T-piece between pulser and summing amplifier
> pulser FWHM still ~30 channels - not the T-piece, probably summing amp
> 
> Checked the noise of HV and LV on battery DSO
> - LV - no noticeable noise on any channel (< 5 mV rms)
> - HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
>        independent of bias voltage 
>        See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
>                       AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2
> 
> Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
> 100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
> Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 

1nF/3k => 333kHz low pass rolloff

10nF/1M => 100Hz low pass rolloff

>  Check on one DSSSD, if see improvement can migrate to adapter boards in new revision
> 
> Things to do:
> TD:
>  Inline filters
> NH:
>  Check same behaviour other N1419ET HV supplies
>  Check line voltages from CAEN NIM bin
  378   Fri Aug 20 14:33:24 2021 TDAIDA Noise
> 
> 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
>       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
>       Switch Cooknell SA1 Sum Amp output from negative to positive
> 
>       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> 
>       This probably indicates
>       - noisy Cooknell SA1
>       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> 
> > Investigate noise situation of AIDA (no DSSSDs connected)
> > 
> > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > - back-side aidas (02,04,06,08) worse resolution than front-side
> > - high-frequency noise still noticeable in waveforms
> > 
> > Figs 7-8: Remove pulser in (from NIM rack) only
> > - no noticable change
> > 
> > Figs 9-10: Remove all pulser cables from adapter boards
> > - looks improved
> > 
> > Figs 11-12: HV Braid->GND Jumpers removed
> > 
> > Fig 13-14: All HV cables removed
> > - aida06, aida14 isn't updating
> > - very noticable improvement
> > 
> > Fig 15-16: Intermediate HV cables re-added
> > - signal worse in FEE64s with HV cables
> > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > 
> > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > 
> > Pulser widths now:
> > - aida14 (-ve amplitude) = 13.5 channels
> > - aida08 (+ve amplitude) = 40 channels (?)

Removed the T-piece between pulser and summing amplifier
pulser FWHM still ~30 channels - not the T-piece, probably summing amp

Checked the noise of HV and LV on battery DSO
- LV - no noticeable noise on any channel (< 5 mV rms)
- HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
       independent of bias voltage 
       See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
                      AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2

Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 
 Check on one DSSSD, if see improvement can migrate to adapter boards in new revision

Things to do:
TD:
 Inline filters
NH:
 Check same behaviour other N1419ET HV supplies
 Check line voltages from CAEN NIM bin
  377   Thu Aug 19 10:47:13 2021 TDMIDAS spectra
  376   Thu Aug 19 10:37:50 2021 TDAIDA Noise
11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
      Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
      Switch Cooknell SA1 Sum Amp output from negative to positive

      Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14

      This probably indicates
      - noisy Cooknell SA1
      - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1

> Investigate noise situation of AIDA (no DSSSDs connected)
> 
> Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> - back-side aidas (02,04,06,08) worse resolution than front-side
> - high-frequency noise still noticeable in waveforms
> 
> Figs 7-8: Remove pulser in (from NIM rack) only
> - no noticable change
> 
> Figs 9-10: Remove all pulser cables from adapter boards
> - looks improved
> 
> Figs 11-12: HV Braid->GND Jumpers removed
> 
> Fig 13-14: All HV cables removed
> - aida06, aida14 isn't updating
> - very noticable improvement
> 
> Fig 15-16: Intermediate HV cables re-added
> - signal worse in FEE64s with HV cables
> - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> 
> Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> 
> Pulser widths now:
> - aida14 (-ve amplitude) = 13.5 channels
> - aida08 (+ve amplitude) = 40 channels (?)
  375   Tue Aug 17 12:39:25 2021 NH, TDAIDA Noise
Investigate noise situation of AIDA (no DSSSDs connected)

Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
- aida16 very noisy ASICS 1&2, normal ASICS 3&4
- back-side aidas (02,04,06,08) worse resolution than front-side
- high-frequency noise still noticeable in waveforms

Figs 7-8: Remove pulser in (from NIM rack) only
- no noticable change

Figs 9-10: Remove all pulser cables from adapter boards
- looks improved

Figs 11-12: HV Braid->GND Jumpers removed

Fig 13-14: All HV cables removed
- aida06, aida14 isn't updating
- very noticable improvement

Fig 15-16: Intermediate HV cables re-added
- signal worse in FEE64s with HV cables
- aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good

Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.

Pulser widths now:
- aida14 (-ve amplitude) = 13.5 channels
- aida08 (+ve amplitude) = 40 channels (?)
  374   Mon Jul 5 14:36:15 2021 NHMonday 5 July
AIDA Unbiased and powered down
All FEEs were running before stoppage. None had rebooted.
Aida05 did not respond until merger was restarted, all other DAQs responded fine.

Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup
AIDA will remain in Snout in lab for time being.

FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now
ELOG V3.1.4-unknown