This is a suggested sequence of actions to help with the current problem with the White Rabbit timestamp distribution. Test the system between each intervention. If a test provides an answer to the problem then proceed accordingly rather than continue with the suggestions.

• Please sketch the current MACB wiring.
• Unplug and replug the connectors of the VETAR HDMI connecting cable. If possible clean the pcb and cable HDMI connectors with contact cleaner.
• Disconnect the VETAR and connect the Emulator. If the clocks are not correct then at the Local and Waveform control page change the entry for LMK03200 control register at offset 5 to 0xB. If clocks are now ok then repeat for all the FEEs. Be aware that this value will be set back to 0x1B during SETUP. If the Emulator fails then continue with VETAR and setting 0x1B.
• Connect one or more FEE64 to the base MACB.
• Change the base MACB and repeat the test above.
• Connect a FEE to the VETAR direct. 

I am at present unable to advise further. However should there be more then I will update this entry.

11.00 No WLAN/WiFi
      came-gsi private LAN appears to be OK

      AIDA FEE64 power ON

11.09 FEE64 temps - attachment 1
      aida06 low, otherwise OK

11:10 CARME IE514 4.0e-11 mbar, temp unknown
      IE514 OFF

11.22 Detector bias ON - attachment 2
      found ch 2 up ramp set to 63V/s! changed to 1V/s

11.30 system wide checks OK except 
      aida02 & aida06 fail WR decoder status

      WR ts control - attachments 3-4


12.02 ASIC settings updated for aida09-aida16
      ASIC settings DB key 2024Jan20-12:02:08

      Unable to save DB key to Options - use ASIC settings restore as workaround

12.30 Layouts for 16x FEE64s created - directory ~/LayOut/16xFEE64
      Note layouts include spectra from aida07, aida08 and aida16 so an error message will be generated but it can be ignored


12.40 per FEE64 Rate spectra - attachments 5-6
      per FEE64 p+n & nn 1.8.W spectra - attachments 7-8

12.57 per FEE64 p+n & n+n 1.8.W spectra 20us FSR - attachments 9-10

Test signals 

EG&G Ortec 448 Pulser
Amplitude 90,000
attenuation x10
Normalise 5.00
t_r 100ns
tau_d 50us
Frequency 50Hz ( relay line freq )
Polarity +

test + direct from 448, test - from Cooknell SA1 Sum & Invert 

Test + daisy chain -> aida01 - aida02 - aida05 - aida06 - 50 Ohm
Test - daisy chain -> aida03 - aida04 - aida07 - aida08 - 50 Ohm

aida08 ( DSSSD #2 n+n Ohmic bias ) LK2 removed, LK1 added
aida04 ( DSSSD #1 n+n Ohmic bias ) no LK2 fitted, LK1 added

DSSSD#1 SHV - 2x Lemo 00.250 cable replaced by standard SHV - BNC/BNC Lemo-00.250 coaxial cable

Adaptor PCB ground daisy chain aida01 - aida02 - aida03 - aida04

DSSSD#2 - DSSSD#4 HV cabling unchanged

15.30 Power ON
      System wide checks OK except
          aida02 fail WR decoder status - attachment 11

      FEE64 temperatures - attachment 12
       aida06 ASIC temp low

      DSSSD bias - attachment 13
     
      ADC data item stats - attachment 14
       slow comparator 0x64

      per FEE64 Rate and 1.8.W 20us FSR spectra - attachments 15-17

16.00 per FEE64 rate spectra - attachment 18

16.15 per FEE64 Rate and 1.8.W 20us FSR spectra - attachments 19-21

17.36 Switch from VETAR2 ( all MACB modules mode 0x3 ) to NIM WR emulator ( root MACB module 0xd, all others 0x3 )
      All FEE64s fail Global clock and WR decoder - attachments 22-24
      WR timetsamp control - attachment 25


18.10 Noted MACB #4 mode actually set to 0x2 *not* 0x3 - changed to 0x3

      Re-test with VETAR2 WR - same behaviour as observed this morning - see attachments 26-30




Summary

n+n sides appear to be OK but noisy

no data observed from aida06 & aida11
 aida11 known to be disconnected from adaptor PCB

aida15 asic #2 may be faulty

aida01, aida05, aida12 - data suggest FEE64/adaptor PCB alignment issue ?

aida02, aida03, aida09, aida14 - data suggest adaptor PCB damaged pins ?


aida07 - network config issue ?

aida08 - not completing boot

need to attach sys console to aida07 & aida08 to diagnose problem.

12.20 CARME IE514 4.1e-11mbar, ambient temperature 16.7 deg C

      FEE64 cabling complete

      WR source - VETAR2 borrowed from Nik Kurz and opwerating standalone ( no BuTIS reference )
      All MACBs mode 0x3


15.20 Power ON FEE64s #1 - #15

      aida02 did not boot - /etc/dhcp/dhcpd.conf typo - fixed

      aida04 did not boot - AIDA FEE64 PSU output #4 fuses open circuit - fuses replaced from AIDA FEE64 PSU spare channel ( aida16 not connected yet ) - fixed

      aida07 did not boot - power OK - FEE64 comfort LEDs OK - network switch link LED OK - network config?

      aida08 boots - power OK - comfort LEDs OK - network switch link LED OK - requests RFS / but does not request RFS MIDAS - can ping aida08 OK, cannot telnet aida08 'connection refused'

      aida09 did not boot - FEE64 comfort LEDS OFF - swap FEE64 #9 & #10 power cables - aida09 boots OK, aida10 does not boot
      Restore #9 & #10 power cables - swap AIDA FEE64 outputs #1 & #2 ( aida09 & aida10 ) - aida09 boots OK
      Conclude AIDA FEE64 PSU ( no label ) output #1 is faulty. Replace AIDA FEE64 PSU ( label ISOL3 ). aida09 & aida10 boot OK

      aida15/aida16 MAC addresses were swapped in https://elog.ph.ed.ac.uk/CARME/467 - fixed

      aida01-aida06, aida09-aida15 boot OK - aida07 does not boot, aida08 does not complete boot

16.25 Configure MIDAS DEE64 servers aida01 - aida06, aida10 - aida15 

      FEE64 temperatures - attachment 1
      aida06 looks low ( c. 25 deg C cf. c. 40 deg C )

      System wide checks - attachments 2-8
       aida02 fails WR decoder

      WR timestamps ( attachments 7 & 8 ) look odd/low - values always c. 0x2???


17.30 CARME IE514 OFF

      WR NIM emulator SMA cables to root MACB ( rear panel ) disconnected at front panel

      Power cycle FEE64s

      System wide checks & WR timestamp control - attachments 9-14
       no change

17.45 Detector bias -20, -100 & 150V - attachments 15-17
      CAEN N1419ET non-floating outputs ( to be checked ), HV cable SHV - 2x Lemo 00.250, LK1 not fitted 

18.40 Detector bias -150V attachment 18




To Do
Check ASIC settings
Configure HV
Check LKs
Ground cabling
      

Network ( dhcpd.conf ) and MIDAS startup.tcl/Options/ASIC settings/rfs filesystems configured


17.20 CARME IE514 4.1e-11mbar, ambient temperature 16.7 deg C
      Pressure appears to be drifting upwards - see attachment 1

Spare AIDA hardware 

1x AIDA FEE64 PSU
1x USB-controlled ac mains relay 

Transferred from EX 2.039 store ( ESR roof ) to CARME storage bins adjacent to CRYRING counting room ( CRYRING roof ).

CRYRING Hall

12 USB-Jtag cables ( sys console ) + 1x to Edinburgh Jan 2024 to fabricate more with spare Jtag PCBs
7x HDMI FEE64 - MACB cables
0x 0.3m HDMI-HDMI cables - require 8x 0.3m HDMI - HDMI cables + 2x 3m HDMI-HDMI cables
2x MACB
1x NIM WR emulator
1x Dlink 7 port USB extender
1x Netgear 24-port Gbit network switch
1x CAEN N1419ET ( ch #0 u/s ? )
0x 10m RJ45 Cat 6 network cables yellow/green/purple/brown - require 2x of each

CAEN N1419ET remote control via carme-gsi and nnrpi1 - see https://elog.ph.ed.ac.uk/CARME/472

15x ( of 16x ) AIDA power cables available, labelled #1 - #15, installed

16x ( of 16x ) RJ45 Cat 6 network cables available, 15x labelled #1 - #15

16x ( of 16x ) HDMI cables available, labelled #1 - #16

Login to nnrpi1 from carme-gsi using ssh

Execute the caen_screen.sh script

pi@nnrpi1:~ $ cat ./caen_screen.sh
if ! screen -list | grep -q "caenhv"; then
	screen -f -S caenhv /dev/ttyACM0 9600
else
	screen -x caenhv
fi
pi@nnrpi1:~ $ ./caen_screen.sh

Enter the text string 'caen'

If the CAEN display program is already running you will only see page updates/changes.
- enter q ( quit ) and then d ( display ) to re-display the entire screen.

See attachment 1

Change RPi from CARMEraspberrypi to nnrpi1 to provide carme-gsi access to Rly16 to control USB-controlled ac mains relay.
nnrpi1 is already defined in carme-gsi:/etc/dhcp/dhcpd.conf

Transfer RPi network connection from CRYRING local network to AIDA 24-port network Gbit switch

Change /etc/hostname from

CARMEraspberrypi

to

nnrpi1

Change /etc/hosts from

127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

127.0.1.1	raspberrypi	

to

127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

127.0.1.1	nnrpi1

Reboot RPi

Login to RPi and start MIDAS 8015 service

/MIDAS/TclHttpd/linux-arm/TclHttpd-server &

Open new browser tab on carme-gsi

http://nnrpi1:8015/AIDA/Rly16/Rly16.tml

FEE64

#1 - #16 -> aida01 - aida16

Network configuration - /etc/dhcpd/dhcpd.conf - attachment 1

MIDAS Options

Added Options sub-directories for aida09-aida16 - attachment 2

MIDAS ASIC settings

Added sub-directories for aida09-aida16 - attachment 3

MIDAS configuration - /MIDAS/config/TclHttpd/carme-gsi"8015 - attachment 4
Currently only aida01-aida08 defined

MIDAS embedded rfs filesystems for aida01-aida24 alreday exist - attachment 5

All FEE64s connected to rev B CARME adaptor PCBs
LK1 fitted to n+n bias FEE64 ( aidaxx, aidaxx, aidaxx, aidaxx )
No other LKs fitted


MACB

WR source NIM WR emulator or VETAR2

See https://elog.ph.ed.ac.uk/DESPEC/517 for MACB mode configuration


CAEN N1419ET


EG&G Ortec 448 Pulser


ASIC settings


Options

10.00 Outflow manifold returned from workshop with leaking attachment point welding re-worked.

11.30 Outflow hoses re-attached to outflow manifold. Faulty male Swagelok replaced.
      Outflow manifold tested on floor - no leaks observed.

      Mounted outflow manifold. All inflow/outflow hoses attached to FEE64s - see attachments 1-4.
      FEE64 #11 detached adaptor PCB - temporarily secured by cable tie.

      Inflow/outflow valves opened. Pin hole leak observed at 3/8" connector gasket - indicated by arrow attachment 
3.
      Leak fixed by tightening fitting.

      1x spare inflow/outflow connected together - attachment 4.

11.40 CARME IE514 3.8e-11mbar, ambient temperature 16.5 deg C

15.05 8x AIDA FEE64 power cables ( #1 - #8 ) installed FEE64s #1 - #8
      Note - Samtec lock clips broken on 3 of 8 cables due to careless handling/poor storage
      AIDA FEE64 power cables tied to one of the FEE64 copper coolant pipes.

      See attachments 5-7

15.30 USB-controlled ac mains relay - USB connected to RPi CARMEbakingpi currently used for CAEN N1419ET 
      Rear panel key switched to override - no flow/d.p. sensors currently installed - front panel water ( yellow ) and override ( red ) LEDs active.

16.15 Transfer CARMEbakingpi from CRYRING network to AIDA 24-port Gbit network switch connected to carme-gsi
      CARMEbakingpi becomes nnrpi1 p- can connect OK via ssh from carme-gsi

      Start server ( port 8015 ) manually

      ssh pi@nnrpi1
      
      /MIDAS/Tcl/Httpd/linux-arm/Tcl-Httpd-server &

      See attachment 8

Cooling water manifolds


The manifolds were returned this afternoon.

Inflow manifold - blue hose, female Swagelok connectors

Attached to inflow hose and bled using bleed/drain valve.
Small leak observed at the 3/8" connection between the manifold and the bleed/drain valve - attachment tightened and leak fixed.
Inflow manifold installed.

Outflow manifold - red hose, male Swagelok connectors

Attached to outflow hose and bled using bleed/drain valve.
Small leak observed at the weld immediately *below* the weld that had been re-worked - this weld had not been re-worked.
Significant leak observed at one of the male Swagelok connectors in the lower half of the manifold - it does not seal correctly.


DR will return this manifold to the workshop for welding re-work Tuesday am.
The faulty male Swagelok will be replaced.


AIDA cabling


16x AIDA PSU, HDMI and RJ45 Cat 6 cables and 4x SHV-BNC cables required.

Available

8x AIDA PSU cables 
1x AIDA PSU cable ( in one of two white plastic storage boxs with 7x FEE64s returned from STFC DL ) now appears to be missing

16x AIDA HDMI cables 

8x RJ45 Cat 6 network cables
1x RJ45 Cat 6 network cable ( network switch to CRYRING network patch panel - to be replaced )
3x RJ45 Cat 6 network cables found in AIDA plastic box ( S4 Messehutte )

3x SHV-BNC 

Will check DESPEC store above ESR with NH tomorrow afternoon.

*Updated following FEE changes. Correct as of 31/01/24

FEE modules are labelled according to the diagram in attachment. Note attachment is looking downstream. Each FEE module is also labelled with its assigned number and whether it is a bias connection on the FEE holders. MAC addresses and FEE descriptions are as follows;

1 -  41:d8:20  - top left detector -  pn bias

2 -  41:d8:2a  - top left detector - 

3 -  41:d0:05  - top left detector - 

4 -  41:b4:0b  - top left detector -  nn bias

5 -  42:0d:0c  - top right detector -  pn bias

6 -  41:0d:16  - top rightdetector - 

7 -  41:b4:16  - top right detector - 

8 -  41:ba:2b  - top right detector -  nn bias

9 -  41:d0:0e  - bottom left detector -  nn bias

10 -  41:d8:2b  - bottom left detector - 

11 -  41:d7:cc  - bottom left detector - 

12 -  41:d7:c2  - bottom left detector -  pn bias

13 -  41:ba:22  - bottom right detector -  nn bias

14 -  41:ee:72  - bottom right detector - 

15 -  41:a0:71  - bottom right detector -

16 -  41:ee:71  - bottom right detector -  pn bias

Found two leaks in the welding of the manifolds, marked them in tape. Manifold to be repaired on Monday. Remember to re-install missing hoses, which are  in a bucket in the cave.

See attached how it is supposed to look like.

Left hand hose (inlet) was too short and was cut to dismount the manifold. Davide will obtain a new tube of the same type and mount it for us. NOTE THAT THE INLET connector (BLUE manifold) was leaking. Unclear if this was the hose or poor plumbing.

Installed cable management system that was never used during the last beamtime. Started to thread cables through it. There should be sufficient space to re-install the manifolds with the cable tray in place. If this is difficult, the tray is installed with zip ties only so altering position should be relativly straightforward. The mounting points of the outlet manifold on the CARME chamber can also be rotated around to give a bit more space.

Cleaned up some boxes and moved heaters and some empty boxes to the CRYRING roof.

Connected all FEEs.

The two top fees corresponding to the bottom right detector are connected to damaged flexi rigid PCBs. The top one is missing an pin that should not be connected to anything, while the bottom one has a slightly cracked connector and is missing the top left pin. Spares should be in Edinburgh.

Found that the flexi rigid PCB connected to the bottom most FEE of the bottom-left detector has a damaged ground lemo connector. Possibly pulled during mounting? The bias cables were off most of the time. Seems to bias fine using the other bias lemo, but not ideal.

Attached, a V-I plot of the four detectors being biased. Note bottom-left was only biased to 120V vs. 150 since it shows relatively high leakage current. Time is wrong - bias was given over ~2 minutes. Unclear why the x-axis is incorrect.

Leakage currents for all detectors, and especially bottom-left and bottom-right drop significantly waiting a sufficently long amount of time, e.g. down to ~5.5 uA after 15 minutes.

To create this plots in case it is needed.

ssh to CARMEbakingpi

cd Programs/caenlogger

python3 caenlogger.py

This will capture the data and pass it to the server (carme-gsi) in ~/caen_current.dat where they will be APPENDED. Delete / save / move the file from the CARME server if required. Backups are in Programs/caenlogger/bias_logs

THEN Bias on

THEN

ssh carme-gsi

cd Programs/caenlogger

gnuplot plot_current.gp

After discussions with Davide, changed the holes in the FEE holders for the bottom-right detector to slots in order to slightly change the angle that the FEE inserts into the flexi rigid PCB. This was sufficient, with some difficulties, to insert both FEEs into the flexi rigid.

Attempted to finally mount the last two fees, the top two for the bottom-right detector, which should have been very easy. Somehow managed to break one pin on each PCB in similar spots. One is in the third column (not in use), and one is in the fourth column (in use). This latter PCB connector was unfortunately dropped while trying to dismount it, and the ERNI connector cracked. That connector can be used anymore? Cannot find spares in GSI, must be in Edinburgh?

Dismounted all the hoses from the manifold, and starting to remove teflon from the plugs. Will use instead gaskets + anti-sieze compound. Quite a number of plug thread are damaged, likely in welding. Unclear if this was the reason behind the leaks. Should be done with the manifolds tomorrow.

Mounted water manifold on CARME with help from GSI plumber. Connected all the hoses with Swagelok self-locking connectors. Tested for water leaks - it leaks from nearly every port. The issue appear to be the connections between the manifold and the plugs, as opposed to the plugs and the hoses which do not leak.

Will try to find a solution tomorrow.

FEEs connected from bottom-left, top-left and top-right detector.

Top right detector showed same behaviour described in previous ELOG entry. Removed all jumpers from PCBs. Beahaviour is gone.

Removed all jumpers from flexi-rigid PCBs for bottom-left detector. Behaviour is improved, but not gone - still 13 uA at 150V. Around 3.5 uA at 120V with no conditioning - that may be the working point.

Missing

- Power cables and ethernet cables for half the fees

- Fix leaks on water manifold

- Fix / change / re-arrange fee holder for bottom-right detector to avoid it fouling on CARME frame.

Started mounting some FEEs. We ran into some mechanical issues due to the new ERNI connectors on the new flexi-rigid adaptor PCBs. Davide skimmed 6.5 mm off the edge with the groove (the groove is gone). This is suffiicent to support the flexi-rigid and in spite of our earlier concerns, the flexi-rigid can be mated/un-mated with the FEEs without major alignment issues.

We still have issues for the bottom-right detector, where the FEE holder will just not fit - will probably figure something out with Davide.

We observe a peculiar issue with the bottom-left detector. At 150V, the leakage current for this detector as tested with just the adaptor cards on the feedthroughs that carry the bias (no FEEs) is of the order of 3 uA.

We mounted all four flexi-rigid PCBs - still 3 uA.

We mounted the FEE that slots into the bottom-most feedthrough (which corresponds to the n-n bias). Still 3 uA.

We mounted the FEE that slots in to the second bottom feedthrough (not corresponding to bias). 15 uA! Upon lowerint the bias, the leakage current goes to zero around 30V as opposed to 0V like all the other detectors, and like this detector when this particular FEE is not mounted.

Tests were performed exchanging all PCBs, and all FEEs. The same behaviour is observed. It is not an issue with the FEE or the PCB. We suspect some cross-contact between detector bias and some strip that feeds into the second FEE? In any case, it is most likely a vacuum side issue - nothing we can do now.

The detector can be biased to 80V or so without major issue, and it appears this behaviour improves over time, i.e. leaving the bias on reduces the leakage current dramatically so it may even be possible to reach 150V.

See plots attached. Note in 2nd plot the plotting script has some issues - detector names are correct until the bias in Ch#2 is brough to 150V, then the colours are incorrect. Only Ch#2 was being changed. Note the drop in leakage current by a factor 2 just increasing the bias to 150V for a few seconds and returning down.

Davide extracted the stuck blind plugs and re-threaded all missing ports. Completed manifold assembly, except for main inlet/outlet - left for plumber.

Cut 8x2 2m hoses and assembled with Swagelok quikfit and teflon ferrules.

We're missing back ferrules in brass for the repaired fees. It should be possible to buy them from GSI.

Attempted assembly of fee, Flexi rigid and black delrin support on bench. Davide shimmed the black delrin supports by 6.5 mm from ridge. They now fit length wise on new adaptor cards. Unfortunately, the decision to flip the connector means the Flexi rigid is no longer at the correct height for the erni connector in the fee. See attached.

Inserting the Flexi rigid without the black support is too dangerous, and defeats the point of having rails. Will probably mount Flexi rigids tomorrow on black support, not screw into the fee holder, and align by eye. Tried once and it works - see last two attachments - but it will have unclear impact on the ease to unmount the fee without damaging the Flexi rigid and will make mounting fee holder on flanges harder. No alternative solutions in current timeframe.

Left bias in overnight. Leakage current decreased and stabilised. See attached. Bias off as we prepare to mount the fees.

Assembled most of the connectors on the new (green) manifolds. Two blanking pulgs stuck when mounting - cannot remove them or tighten them. Swarf? Will ask Davide's help tomorrow. Cannot use the manifold until this is fixed.

Checked FEE assembly in cassette. Need to shorten black Delrin supports by ~6 mm. Holes should end up in the correct position thanks to rails being effectively longer (same length, different hole position on FEE cassette). Will ask Davide to skim one if possible. If not, will have to mount without supports.

Finally managed to get the CAEN program working to control N1419 remotely from Baking Pi. Plotting bias and leakage current overnight to test stability. See attached. IE514 sensor off for these tests.