DAQ was left running last night from 19:30. Detector bias 80V with a leakage current ~3.4 uA. Pulser OFF.

Observed alphas in spectra overnightin the 20,000 to 28,000 channel range for the pn side and 36,000 to 44,000 range on the nn side of the detector. Zero energy noise peak is at channel 32000.

Alphas checked for AIDA5 and AIDA6 (pn) with ~150 alphas counted using a range 20,000-26,000/27,000 depending how large the noise peak tail is. Attached spectra show the spectra for the 4 ASICs of AIDA5 and 6.

9:30 Bias remained stable overnight - 60V 2.95 uA.

10:00 No grounding attached initailly. Pulser ON, lots of noise across all ~250-300 kHz across biased detector. Waveforms and histograms saved to /tmp. Pulser width for aida 6 ~300 channels. Aida 7,8 similar, aida 5 producing no ADC data. 

11:00 Grounding connected as in elog 181. Rates decreased to ~180-250 kHz. Waveforms and histograms saved to /tmp. 

12:00 Bias slowly increased too 100V giving a leakage current of ~3.5 uA which was stable for several hours. Bias varied between 60V and 100V for noise reduction perposes, 80V deemed least noisy at the time, noise still ~180-200 kHz

13:30 New grounding trialled. Bottom detector removed from pulser and grounding loops so that only the biased detector is grounded. Bias set to 80V

14:00 Rates significantly reduced for aida 6,7,8 - aida5 still producing no ADC data. Rates are in range 15-50 kHZ with noisy channels accounting for most of the rates. Waveforms and histograms saved to /tmp. Pulser peak on aida 6 ~300 channels. 

16:00 Aida 5 fee investigated to reason it is not producing data. FEE was not properly connected to ERNI connector due to incorrect installation of black adaptor card mount. Adaptor card mount reinstalled and fee remounted. Aida 5 now produces ADC with similar rates to aida6,7,8. 

During investigation of aida5, aida4 and 5 were swapped. Config files relating to MAC addresses, polarity etc of FEES has been updated. Elog 175 will be appended to include this swap.  

19:00 Bias set to 80V, DAQ running with pulser off to allow for alpha detection overnight. 

FEE mounts for 5,6,7,8 disassembled to access adaptor boards. No sign of debris which could cause short on adaptor boards. 

Adaptor boards removed and checked against spares with multimeter. Both show same characteristics. Pins on feedthrough checked using multimeter. No longer see -18.5 KOhm between bias pins, no see 140 kOhm which is similar to bias pin to non bias pin from prevous tests.

Camera on interaction chamber removed and replaced with foil, no change. 

Ion gauge (Edinburgh group) turned ON

Decided to move detectors using motors. Movement caused change in bias characteristic of the detector. Initial movement was 50 repetitions in and out over which the bias was increased from 10V to 140V. Over this range the detector exhibited diode like behaviour. The difference between the in and position and out position caused a 0.5 uA difference in leakage current. Once stationary, the current was stabel at 140V but after 20s current increased rapidly and tripped.  

Increasing bias again to 30V while stationary (out position on motors) gave 7.3 uA , going to 60V tripped again. 

Moving the detectors IN but keeping stationary gave a stable current at 30V of 4.2 uA increasing to 70V tripped again. Turning off the gauge reduced the current at 30V to 2.9 uA and 70V to be be ~15.5 uA. Opening an optical window at 70V saw no change in leakage current. 

Possibility thermocouple attached to or around the detector may have come loose within the chamber? Thermocouples outisde of chamber removed from thermocouple flange so they don't interact with outside of chamber. 

Detector biased to 60V - 10 uA. Detector position moved in and out 20 repetions, during movement current dropped to 2.85 uA. After motion stopped leakage current remained stable at 2.85 uA. Optical window opened no change. Torch light shone into optical window - increase in leakage current to 3.5 uA. Torch off - return to 2.85 uA. 

Motion started again for 50 repetitions to test stability of leakage current. Current varies between 2.84-2.88 uA during movement, settles to 2.85 uA once motion stopped. 

FEE holder reassembley. Involved turning off bias, unplugging bias cables and rebuilding assembley. FEE holders reaasembled with FEEs reconnected. Water cables, HDMI, ethernet, power connected. Bias turned off and on during reassembley multiple times to check for changes. Each time bias at 60V remained 2.85uA. Each time lower voltages appear to show diode like response in leakage current output. 

Pulser connected and FEEs turned back on. Aida05 appears not producing ADC data rates 8 kHz on statistics page -investigate further. Aida06 rate ~70 kHZ can see pulser. Aida07, 08 nn bias show rates ~300 kHZ possible not enough bias for nn side. Rate for all FEEs yesterday (unbiased) was 200-300 kHZ except for AIDA03 disconnected and AIDA02 which wasn't producing ADC data. Photo of Aida06 spectra attached (log scale for pulser). 

FEEs turned OFF. Bias left on 60V - 2.85 uA see if stable overnight, stable for approx 2.5 hours before leaving cryring. Detector moved to OUT position, pin back in. 

To do - Grounding, reduce noise, waveforms,  investigate AIDA05, how high can we bias?  

Detector is not biasing as it has in previous tests. Last bias 15/1/22 see previous elogs, max leakage current ~10 uA at 150V

Detector bias increases well above 10uA for voltages as low as 30-40V. Optical windows covered, Y09 valves closed, large ion pump OFF and both ion gauges OFF produces similar results but with a slightly lower leakage current.

Grounding and pulser cables removed from FEEs and jumper attached to nn bias adaptor card - no change. Bias module swapped out - no effect. NIM rack shows expected voltages (6,12,24) on multimeter.

AIDA03 - disconnected. Behaves as expected. Get waveforms and pulser peak. Pulser peak FWHM ~ 70 channels, noise transients observed.

AIDA02 - Not producing ADC data

AIDA01,04,05,06,07,08 - Very high rates - noisy

Steps taken:
- MIDAS_Releases, Embedded and Exports files brought across from the previously working SL6 aida-gsi install. Brought as tar balls to preserve ownership
- dhcpd.conf set up with same IP addresses as used on the SL6 install from aida-gsi
- nfs-utils package installed on centos7
- Paths to directories listed in /etc/exports checked that they point to the correct locations on the centos system
- Firewall disabled to confirm that it is not interfering
- setenforce 0 to confirm selinux is not interfering.


- Attachment 1 Screenshot of FEE console for aida04 as it stops booting.
- Attachment 2 text dump of /var/log/messages during the FEE boot sequence
- Attachment 3 text dump from tcpdump filtered to aida04
- Attachment 4 /var/lib/dhcp/dhcp.leases file as mentioned in the /var/log/messages dump
- Attachment 5 A copy of dhcpd.conf
- Attachment 6 A copy of /etc/hosts
- Attachment 7 A copy of /etc/exports    
- Attachment 8 Text dump of ifconfig -a (Note the FEEs are conencted to p4p1)

From the messages dump it seems there is an issue being encountered with dhcp leases for the static IP addresses of the FEEs. They are then getting assigned new IP addresses which are getting put in the leases file.

Grounding

Ground cables have been daisy chained together from FEE to FEE. Ground cables are screwed into a feee bolt hole on the FEE body using an M3 bolt. 

LEMO cables have been connected in a similar way between the grounds on the adaptor boards for each FEE. Short LEMO cables go between J6 on pairs of adaptor boards in the same FEE holder and long LEMO cables go between J7 on adaptor boards in different FEE holders. 

The LEMO grounds from the adaptor boards and the ground cables attached to the FEE bodies are then connected together and then connected to the CRYRING grounding cable. 

Pulser

Jan has kindly lent us a pulser. We send the pulser to an inverter giving us a +ve and -ve signal. 

The +ve signal goes into the pn FEEs (1,2,5,6)

The -ve signal goes into the nn FEEs (3,4,78)

LEMO cable connecting FEEs in the same holder using J3 on adaptor board. J4 is used for recieving the pulse signal, connecting FEEs in different holders or terminating the signal. 

Power, Ethernet and HDMI cables connected to FEEs. To prevent cables coming out of connectors, power and HDMI cables are zip tied to FEE cooling pipe (see attached image).

Correct FEE MAC addresses have been updated in /etc/dhcp/dhcpd.conf

Remote access to carme sever has been setup. Remote access instructions in elogs 176-179. 

New ASIC settings which reflect polarity of FEES produced 2022Jan27-16-43-00 

New Options file updated to point to new ASIC settings. 

/MIDAS/config/TclHttpd/carme-gsi@8015/startup.tcl updated for 8 FEEs 

/MIDAS/Linux/startup/NewMerger Updated line from "./master64 -i 16 -l 16 -p 11001  &" to "./master64 -i 8 -l 16 -p 11001  &"

Quick motors test performed. Motors accesible remotely (elog 179). Motors moving output (BNC 1 on motors box) connected to motors module (BNC 1). When not moving +12V is observed at BNC 2 on motors module. When moving 0V is observed on BNC 2.  

1. ssh <user>@lxpool.gsi.de
2. ssh <user or carme>@atppc022.gsi.de
3. ssh pi@CARMEmotorspi (Standard npg password)

To use both anydesk and firefox on the carme server require an internet connection.

As the server is on the cryring local network an ssh tunnel needs to be opened to the GSI proxy server.

To do this: ssh -L 8080:proxy.gsi.de:8080 carme@atppc025` and leave terminal open and running.
Both anydesk and firefox have been configured to use the proxy localhost:8080

For the carme password ask J. Marsh or C. Bruno

1. ssh into lxpool network: ssh <user>@lxpool.gsi.de
2. ssh into atppc022.gsi.de: ssh carme@atppc022.gsi.de (This is an rpi access server. Don't run code)
3. ssh into carme server: ssh npg@carme-gsi

For the password to the carme user account on the cryring network please contact O. Hall or J. Marsh.

Water cooling has been connected to the FEE's. Flow meter installed to outlet manifold by way of two right angled pieces. No leaks observed rom any points in the coolant loop, loop will be bled using taps tomorrow. Water turned OFF overnight. 

FEE's powered up with the MAC addresses of each read. 1 is the very bottom FEE and 8 the very top. FEE's have also labelled with the MAC addresses.  

1. 0xd8 0x80 0x39 0x42 0x0d 0x0c  

2. d8:80:39:41:d8:2b

3. d8:80:39:41:d0:05

4. 0xd8 0x80 0x39 0x41 0xd7 0xc2  (Now AIDA 5 see elog 185)

5. 0xd8 0x80 0x39 0x41 0xb4 0x0b (Now AIDA 4 see elog 185)

6. d8:80:39:41:d8:20

7. 0xd8 0x80 0x39 0x41 0xb4 0x16

8 .0xd8 0x80 0x39 0x41 0xd7 0xcd  

MACB firmware has been updated by Nic

One ion pump showed the warning 'arcing' (see attached photo) and was subsequently turned OFF. Most likely cause is the cable was moved during installlation of the flow meter as the pump showed no error this morning. The pump was turned back ON, no error observed -> cable not damaged but sensitive to movement.   

All 8 FEEs have been installed in their holders.

The damaged feedthrough does not have the D-connector on the adaptor board inserted into the falnge but the FEE is connected to the adaptor board. 

FEE's were carefully installed, with one person inserting FEE and another making sure FEE was straight and on the correct pin positions. The spring lock on the bottom FEE's have been tightend and zip ties have been placed to make sure they are secured. 

Rack has been populated with cables from the rack to chamber laid out. Cables mostly go above the rack and hang off the large frame. This frame or the CARME frame takes most of the weight of the cables before they are plugged into the FEEs.   

Cables are numbered 1-8. Brown ethernet cables are numbered 1-4, green 5,6 and yellow 7,8. Numbers correspnd to Bottom FEE (1,2), Side bottom FEE (3,4), Side top FEE (5,6), Top FEE (7,8)

Vaccum
DN250CF 1x gaskets     
DN200CF 5x gaskets  
DN160CF 1x gaskets, 2x blank flanges   
DN100CF 69x gaskets, 19x blank flanges, 14 double feedthroughs (1 damaged) 
DN63CF 3x gaskets, 2x blank flanges    
DN300CF 5x gaskets
DN350CF 0x gaskets
1X WIRE SEAL
DN35CF 3x gaskets, 13 blank flanges   
DN16CF 22x silver gaskets, 27x copper gaskets   

Swagelock
SS-QTM2-D-8M0 Male quick disconnect X18
SS-QTM2A-B-8M0 Female quick disconnect X21
SS-8M0-1-6RS Manifold adaptor X39
NY-8M3-1 Front ferrule X118
NY-8M4-1 Back ferrule X120
Stainless ferrules XMANY

FEE/FEE HOLDER PARTS
FEES x1
Fully assembled FEE holder (MINUS ADAPTOR BOARDS) x1
Cylindrical supports x32
Feedthrough mounts x12
Black adaptor board mounts x11
Spring loaded FEE lock x26
FEE holder side panels x5 pairs
Adaptor boards x2 (1 slightly bent pins)
FEE rail bolts x4
M3X16 281-013 x91
M3 locknuts 260-5815 x34
Springs 821-273 x40 
M4x25 (2mm too long) 281-057 x32
M6x20 281-120 x69
M3x6 280-981 x34
Assembled Hose 1.5M (Red) x8
Assembled Hose 1.5M (Blue) x8
Blue Hose ~16M
Red Hose ~10M

We require additional gaskets for those we are low on (<5) DN35,63,160,250,350 and more wire seals 

We should have enough swagelock and FEE holder parts to construct another 4 FEE holders for the remaining required for mounting all moving detector FEEs.

We also need fuses for the USB mains relay and power supply -> 2x 250v 10A, 16x 250v 1.6A. Dimensions for both are the same, length=20mm, Diameter=5mm (images attached)

Additional fittings provided by GSI for the manifold are in the swagelock box.

11.00 Per https://elog.ph.ed.ac.uk/CARME/170 and https://elog.ph.ed.ac.uk/CARME/171 issues have been identified with the FEE64 carriers and adaptor PCBs

Some additional comments

1) Per https://elog.ph.ed.ac.uk/CARME/170 attachments 6-9 we can see that there are no cable strain reliefs to which we can attach the FEE64 power, RJ45 network and 
timestamp HDMI cabling. It would probably be straightforward to design something which could be attached to the FEE64 carrier side panels. For the time being we will use 
the copper piping of each FEE64 as the nearest, mechanically stable point to which we secure the cabling.

2) Before the skimming of 1x Delron rail yesterday there was a small amount (<1mm say?) of transverse movement of the FEE64 during insertion into the carrier and it is 
not clear why. Small sample of measurements of FEE64 carriers, spacers etc.

Delron adaptor PCB support widths 92.85, 92.87, 92.90
Al DN100CF flange attachment width 92.97, 92.99, 93.01
Al FEE64 carrier spacing rod 93.03, 93.04, 93.03
FEE64 width (w/ 1x c. 2mm skimmed Delron rail) 100.2, 100.2, 100.3 (3 measurements along the length of the 1x FEE64)
FEE64 carrier width 102.8, 103.0 (2 measurements of 1x FEE64 carrier)


probably indicates that the spacing between the FEE64 plates is consistent. There is some variability (~0.1mm) in the Delron adaptor PCB which could affect the relative 
position of the ERNI pins and sockets. Is there some variability in the width of the FEE64s (w/ rails)?

The ERNI connector is not central on the adaptor boards causing misallignment with the FEE. 

The adaptor board is 80mm long. The ERNI connector is 50mm long. The connector is 13mm from the 2x Lemo side and 17mm from the 3x Lemo side. This results in misallignment between the FEE and adaptor board. 

To correct for misallignment, 2mm will be skimmed from one of the rails on the FEE. 

Skimming the rail allows the FEE to now be inserted straight into the adaptor board connector. Attached image shows test FEE holder on the bench. FEE needs to be carefully inserted to make sure it is straight as it can now move around within the FEE holder with the 2mm skimmed off the rails. For CARME, where sight lines and space are limited, one person is required to hold and push in the FEE whilst another checks the positioning of the connectors is straight. 

4 FEE holders installed on CARME. Mounting procedure devised due to limited working space to install.

Mounting procedure:
>Install FEE holder mounts (image 1) onto feedthrough flange

>Partially make up FEE holders on workbench. Only 3 cylindrical support rods, spring latches and one of the black adaptor card mounts should be attached to aid install. Bolts should be loose to aid install on CARME. 

>Mount FEE holder to flange

>Connect first adaptor card into D connector on feedthrough, then screw adaptor card to black mount. 

>Screw in second black adaptor card mount but do not tighten bolts. Connect second adaptor card to D connector and screw to black mount. 

>Attach remaining cylindrical supports and tighten all bolts.   

Bolts to install FEE holder mount onto flange (M4x25) are 2mm too long for the hole on the flange. Washer were initially used however this prevents adaptor cards being able to be secured onto thier black mounts. Bolts were shortend 2mm by davide to install. 5 shortend bolts remain in addition to thebag of unshortend M4x25 bolts. 

On the nn flange of the bottom detector (non bias pin D-connector) the feedthrough is damaged. Damage may be repairable, to prevent further damage an adaptor card was not plugged into this D-connector. (image 1)

A bias test was performed for the top detector once adaptor cards are installed. First test is the bias test performed prior to adaptor cards being installed, second test was performed after installation of adaptor cards.   

Installing FEEs on the outer side of the ring in the future may have some mechanical conflict with the gas target frame, see attached images. 

2x cooling water manifolds with 2x 8 ports have been installed - see attachments 1 & 2

'drain' ports at top of manifold to permit release of trapped air volumes. Local technicians will install a standard blank in one manifold and a small nipple valve on the 
other so that we can bleed air in a controlled way.

#Multimeter detector pin tests. 
Pins on feedthrough flanges investigated using multimeter for the top and bottom detector due to no response from bottom detector when bias was applied. Bias pins on 
correct D connectors are number 78. 

#Top detector
>Observe -18.5 kohm; between nn bias pin and pn bias pin  (multimeter 200kohm setting)
>observe open line between nn bias pin and pins 'near' pn bias pin (multimeter 200kohm setting)

>Between random pins on same D connector (nn bias pin connector) ~0.02Mohm (multimeter 2Mohm setting)
>Between random pins and unconnected on same D connector (nn bias pin connector) - open line (multimeter 2Mohm setting)
>Between random pins and bias pin on same D connector (nn bias pin connector) ~0.01-0.02 Mohm (multimeter 2Mohm setting)

>Between nn bias pin and random pin on another D connector ~-0.4Mohm (multimeter 2Mohm setting)

#Bottom detector 
>Observe open line between nn bias pin and pn bias pin (multimeter 200kohm+2Mohm; setting)
>Observe open line between nn bias pin and pins 'near' bias bias pin (multimeter 200kohm+2Mohm; setting)

>Between random pins on same D connector (nn bias pin connector) ~0.02Mohm (multimeter 2Mohm setting)
>Between random pins and unconnected on same D connector (nn bias pin connector) - open line (multimeter 2Mohm setting)
>Between random pins and bias pin on same D connector (nn bias pin connector) ~0.01-0.02 Mohm (multimeter 2Mohm setting)

>Between random pins on same D connector (pn bias pin connector) ~0.02Mohm (multimeter 2Mohm setting)
>Between random pins and unconnected on same D connector (pn bias pin connector) - open line (multimeter 2Mohm setting)
>Between random pins and bias pin on same D connector (pn bias pin connector) ~open line (multimeter 2Mohm setting)

>Between nn bias pin and random pin on another D connector ~-0.4Mohm (multimeter 2Mohm setting)

We see a similar response for both detectors except that the pn bias pin gives no response. 
We have looked to find the bias pin on nearby pins to pin 78 and on the other side of the D connector (in case of cabling error-mirror or swapped cables) but cannot find. 

4 FEE holders constructed and ready to install

Baking tent, frame and trays fully removed from CARME chamber. Removal of baking frame required removing air lines from upper gate valve and repositioning of vacuum hose connecting turbo pump to backing line and turbo power cable. Some difficulty encountered removing upper baking frame now the target support frame is assembled. Small modification seperating section in two would aid future installation and disammebly. 

Baking trays removed with difficulty. Main baking tray bent to extract from the frame -> updated design required for future bakeout cycle. 

VI tests performed on the upper and lower detectors. Upper detector shows characteristic curve (attached plot), lower detector shows no response to increased voltage. Multimeter used on adapter boards with resistance of ~18.5 kΩ observed between bias pins for the upper detector. Open line seen between bias pins on the lower detector, some connections observed for other pins on the board -> requires further investigation.  

NOTE - Channel 0 on bias supply is unstable, should not be used. 

 Most equipment for installation of FEE's and coolant manifolds found, assembley of FEE holders started. Full list of current swagelock inventory as follows:

Dry run complete.

YR09 section valves closed. Section isolated. Manual valve between electron target and CARME remain open.

GSI vacuum division IE514 gauge connected.

Gleb reports gate downstream opened. Vacuum goes up and recovers as expected.