08:52 Overnight the DAQ was left running to check if there was any individual radioactivity in the Si.
      This is stored in R7_75

      It was observed that even with no beam the detectors continued to increase in leakage current overnight but slowly - Attachment 1
      We then stopped the DAQ and gave the detector a single shake.
      This caused the leakage currents to decrease but not to the same level that they did last night - Attachment 2

13:45 Bias tripped as people removed the cover on the windows on the interaction chamber
      Iset was still at 11uA for safety
      Will leave off until they are finished

      

09:00 Detector still biased to 60V and leakage current has been mostly stable overnight.
      Only minor fluctuations observed - attachment 1

      FEE temps all ok - attachment 2
      Stats at 60V - attachment 3
      System wide checks all ok

      Increasing bias to 80V
      60V 9.45uA
      70V 9.7uA
      80V 80 9.82uA Still have diode response

10:15 In prep for taking data with beam tuning have tested the merger and tapeserver.
      To enable writing to file had to implement the method in https://elog.ph.ed.ac.uk/DESPEC/36
      Probably a corrupution caused in the difference between the carme system and the aida system.
      Now can write data to file. To limit the data of FEEs 01-04 have set threshold to 0xFF

10:33 Ion source was started this morning but there was no noticable effect on the leakage currents.

10:56 Leakage current is slowly rising continuing the trend at 60V. - Attachment 4
      This could just be the temperature shift during the day or linked to the ion source being powered on.

11:18 Pulser peaks for the FEEs using layout 3 and 4.
      Layout 3 - attachment 5
      Layout 4 - attachment 6
      The pulser peaks appear to be asymetric and in FEE 7 we are observing a double peak.
      Will check the circuit when we next have access.

11:23 Have started writing data to disk in advance of them putting beam into the ring.
      Writing to R4_8 at around 1.8MB/s. N.B. TapeServer shows the same behavior as AIDA with writing nothing to files R4_0->R4_7
      FEE 01-04 slow comp 0xFF
      FEE 05-08 slow comp 0x64
      Pulser running at 50Hz or so

14:50 Having been writing to DAQ since lunch.
      Leakage currents were mostly stable but we are now seeing an increase in leakage currents at a much greater rate - attachment 7

14:58 Attachment8 Updated leakage currents.

15:00 Due to the rapidly increasing bias we asked them to stop the beam.
      Motors were then cycled but we did not notice the leakage coming down.
      Bias was turned off and then motors cycled again.
      Bias was then set to 60V - Run R5
      Bias raised to 80V - R6
      Carlo requested we stop writing data to disk

16:25 Stopped writing to disk. Will retry bias at 80V and monitor.

18:13 Motors cycled before leaving for night no beam in ring - attachment 9
      to test if the increase is beam related we will leave at 80V overnight with the beam blocked all night.
      To avoid the rapidly increasing leakage current the iSet on the CAEN module is at 11uA with a trip time of 2s.
      

09:09 Arrived at GSI for comissioning beamtime.
      CARME currently in a stopped state, with all FEEs powered off.
      Upper detector is biased with 80V and has a leakage current of 9.92uA.
      This is up on the previous recording by JM of 5.4uA recorded in elog 187.
      It is possible light is reaching the detector via the interaction chamber which shall be checked.

09:19 Cave checked:
      Water is running to the FEEs (OK for power on when required)
      There are 4 uncovered windows on the interaction chamber. Will see if we can get these covered.

14:29 Tested different bias voltages.
      Between 80V and 120V got a normal diode response with small increases in leakage current
      At 120V leakage current was stable at 9.92 for one minute.
      Checked waveforms on DAQ and looked at current again and it was up to 16uA.
      Turned off bias and let ramp down.
      Now when trying to ramp up detector in similar position to last week with a diode response between 0-30V and a rapid increase above that.
      Carlo spoke to Gleb who was beam tuning. Which around the time of our trip he opened the valves and put beam into the ring for a rotation.
      Gleb put in several faraday cups prior to the CARME chamber yet we still see the non-diode bias characteristics.
      He will enter in 30 minutes and we will check if anything looks out of place. Will also remove motor pin to give option of movement.

15:00 Gleb and Carlo enters the cave.
      Dipole and beam off.
      Pin removed.
      Diode response restored 9.4uA at 40V. Which is only slightly above where it was during the morning


      Before they got anywhere with starting again the current has gone up again..

16:49 Since around 15:30 the current has come down to its levels earlier today at 40V. We are preparing a script to monitor the current levels going forward.

17:00 Gleb stopped tuning the beam for the day.
      There is no beam in the ring.
      Magnets and everything else continue as if there was beam in the ring.

19:40 Attempted to raise bias to 50. Immediately started to trip
      Carlo cycled motors multiple times.
      Bias in out position dropped.
      To lower
      Have made it to 60V. Will leave there overnight with leakage current tracking running.

      No beam so no daq running.

JM plans to install new crimped, 3mm dia copper drain wires (a la AIDA@DESPEC) this afternoon. 

Initial Grounding - Lemo cables connecting adaptor cards ground in daisy chain. Fee bodies connected by 3mm copper drain wires in daisy chain. Two chains brought together 
at final adaptor card which is connected to cryring ground. 

Attachments 1-5

ADC data item rates   all <10k
ASIC settings         all LEC slow comparators 0x64
1.8.L spectra aida05-08 widths 556, 511, ?, 570 ch FWHM respectively
1.8.W spectra 20us FSR
Adaptor card grounding image

1.8L widths appear broader than previous day of ~300 width. Reason unknown, no change made to setup as yet. 

Second Grounding - 3mm copper drain wires connect each adaptor card to Fee body. Fee bodies daisy chained together using 3mm copper drain wire and connected to cryring 
ground. 

ADC data item rates   all <10k
ASIC settings         all LEC slow comparators 0x64

Attachments 6-8
1.8.L spectra aida05-06 widths 620, 550 ch FWHM respectively, 1.1.L aida08 width 350 ch FWHM
1.8.W spectra 20us FSR. Some extra high frequency noise
Adaptor card grounding 
 

Third Grounding - 3mm copper drain wires connect adaptor cards in daisy chains. Fee bodies daisy chained together using 3mm copper drain wire. Two chains brought 
together and connected to cryring ground. 

ADC data item rates   all <10k
ASIC settings         all LEC slow comparators 0x64
1.8.L spectra aida05-08 similar to initial grounding ~500 ch FWHM

Attachments 9-11
1.8.W spectra 20us FSR.
AIDA5-8 channel rates. Rates dominated by noisy strips. Missing strips come and go as ASIC settings checked. 
Adaptor card grounding

Some work still required to reduce noise in the system and reduce pulser widths. Pulser should also be checked on scope 

Gauge turned back ON. Bias 80V, 3.5 uA before. Bias 80V, 5.4 uA after. Bias turned off and on again. NO change to diode characteristic, current ramps high before coming 
down once bias on. Bias left ON 80V to see if stable when second gauge turned on and valves reopened. Pulser ON. Fee power OFF. Water cooling ON. 

No significant changes to waveforms, pulser width or rates with gauge ON. Other noise contributions probably outweigh effect of the gauge at this point. 

3mm copper grounding cable and attachments, crimper and Yorkshire tea left in control room. 

DAQ was left running last night from 19:30. Detector bias 80V with a leakage current ~3.4 uA. Pulser OFF.

Observed alphas in spectra overnightin the 20,000 to 28,000 channel range for the pn side and 36,000 to 44,000 range on the nn side of the detector. Zero energy noise peak is at channel 32000.

Alphas checked for AIDA5 and AIDA6 (pn) with ~150 alphas counted using a range 20,000-26,000/27,000 depending how large the noise peak tail is. Attached spectra show the spectra for the 4 ASICs of AIDA5 and 6.

9:30 Bias remained stable overnight - 60V 2.95 uA.

10:00 No grounding attached initailly. Pulser ON, lots of noise across all ~250-300 kHz across biased detector. Waveforms and histograms saved to /tmp. Pulser width for aida 6 ~300 channels. Aida 7,8 similar, aida 5 producing no ADC data. 

11:00 Grounding connected as in elog 181. Rates decreased to ~180-250 kHz. Waveforms and histograms saved to /tmp. 

12:00 Bias slowly increased too 100V giving a leakage current of ~3.5 uA which was stable for several hours. Bias varied between 60V and 100V for noise reduction perposes, 80V deemed least noisy at the time, noise still ~180-200 kHz

13:30 New grounding trialled. Bottom detector removed from pulser and grounding loops so that only the biased detector is grounded. Bias set to 80V

14:00 Rates significantly reduced for aida 6,7,8 - aida5 still producing no ADC data. Rates are in range 15-50 kHZ with noisy channels accounting for most of the rates. Waveforms and histograms saved to /tmp. Pulser peak on aida 6 ~300 channels. 

16:00 Aida 5 fee investigated to reason it is not producing data. FEE was not properly connected to ERNI connector due to incorrect installation of black adaptor card mount. Adaptor card mount reinstalled and fee remounted. Aida 5 now produces ADC with similar rates to aida6,7,8. 

During investigation of aida5, aida4 and 5 were swapped. Config files relating to MAC addresses, polarity etc of FEES has been updated. Elog 175 will be appended to include this swap.  

19:00 Bias set to 80V, DAQ running with pulser off to allow for alpha detection overnight. 

FEE mounts for 5,6,7,8 disassembled to access adaptor boards. No sign of debris which could cause short on adaptor boards. 

Adaptor boards removed and checked against spares with multimeter. Both show same characteristics. Pins on feedthrough checked using multimeter. No longer see -18.5 KOhm between bias pins, no see 140 kOhm which is similar to bias pin to non bias pin from prevous tests.

Camera on interaction chamber removed and replaced with foil, no change. 

Ion gauge (Edinburgh group) turned ON

Decided to move detectors using motors. Movement caused change in bias characteristic of the detector. Initial movement was 50 repetitions in and out over which the bias was increased from 10V to 140V. Over this range the detector exhibited diode like behaviour. The difference between the in and position and out position caused a 0.5 uA difference in leakage current. Once stationary, the current was stabel at 140V but after 20s current increased rapidly and tripped.  

Increasing bias again to 30V while stationary (out position on motors) gave 7.3 uA , going to 60V tripped again. 

Moving the detectors IN but keeping stationary gave a stable current at 30V of 4.2 uA increasing to 70V tripped again. Turning off the gauge reduced the current at 30V to 2.9 uA and 70V to be be ~15.5 uA. Opening an optical window at 70V saw no change in leakage current. 

Possibility thermocouple attached to or around the detector may have come loose within the chamber? Thermocouples outisde of chamber removed from thermocouple flange so they don't interact with outside of chamber. 

Detector biased to 60V - 10 uA. Detector position moved in and out 20 repetions, during movement current dropped to 2.85 uA. After motion stopped leakage current remained stable at 2.85 uA. Optical window opened no change. Torch light shone into optical window - increase in leakage current to 3.5 uA. Torch off - return to 2.85 uA. 

Motion started again for 50 repetitions to test stability of leakage current. Current varies between 2.84-2.88 uA during movement, settles to 2.85 uA once motion stopped. 

FEE holder reassembley. Involved turning off bias, unplugging bias cables and rebuilding assembley. FEE holders reaasembled with FEEs reconnected. Water cables, HDMI, ethernet, power connected. Bias turned off and on during reassembley multiple times to check for changes. Each time bias at 60V remained 2.85uA. Each time lower voltages appear to show diode like response in leakage current output. 

Pulser connected and FEEs turned back on. Aida05 appears not producing ADC data rates 8 kHz on statistics page -investigate further. Aida06 rate ~70 kHZ can see pulser. Aida07, 08 nn bias show rates ~300 kHZ possible not enough bias for nn side. Rate for all FEEs yesterday (unbiased) was 200-300 kHZ except for AIDA03 disconnected and AIDA02 which wasn't producing ADC data. Photo of Aida06 spectra attached (log scale for pulser). 

FEEs turned OFF. Bias left on 60V - 2.85 uA see if stable overnight, stable for approx 2.5 hours before leaving cryring. Detector moved to OUT position, pin back in. 

To do - Grounding, reduce noise, waveforms,  investigate AIDA05, how high can we bias?  

Detector is not biasing as it has in previous tests. Last bias 15/1/22 see previous elogs, max leakage current ~10 uA at 150V

Detector bias increases well above 10uA for voltages as low as 30-40V. Optical windows covered, Y09 valves closed, large ion pump OFF and both ion gauges OFF produces similar results but with a slightly lower leakage current.

Grounding and pulser cables removed from FEEs and jumper attached to nn bias adaptor card - no change. Bias module swapped out - no effect. NIM rack shows expected voltages (6,12,24) on multimeter.

AIDA03 - disconnected. Behaves as expected. Get waveforms and pulser peak. Pulser peak FWHM ~ 70 channels, noise transients observed.

AIDA02 - Not producing ADC data

AIDA01,04,05,06,07,08 - Very high rates - noisy

Steps taken:
- MIDAS_Releases, Embedded and Exports files brought across from the previously working SL6 aida-gsi install. Brought as tar balls to preserve ownership
- dhcpd.conf set up with same IP addresses as used on the SL6 install from aida-gsi
- nfs-utils package installed on centos7
- Paths to directories listed in /etc/exports checked that they point to the correct locations on the centos system
- Firewall disabled to confirm that it is not interfering
- setenforce 0 to confirm selinux is not interfering.


- Attachment 1 Screenshot of FEE console for aida04 as it stops booting.
- Attachment 2 text dump of /var/log/messages during the FEE boot sequence
- Attachment 3 text dump from tcpdump filtered to aida04
- Attachment 4 /var/lib/dhcp/dhcp.leases file as mentioned in the /var/log/messages dump
- Attachment 5 A copy of dhcpd.conf
- Attachment 6 A copy of /etc/hosts
- Attachment 7 A copy of /etc/exports    
- Attachment 8 Text dump of ifconfig -a (Note the FEEs are conencted to p4p1)

From the messages dump it seems there is an issue being encountered with dhcp leases for the static IP addresses of the FEEs. They are then getting assigned new IP addresses which are getting put in the leases file.

Grounding

Ground cables have been daisy chained together from FEE to FEE. Ground cables are screwed into a feee bolt hole on the FEE body using an M3 bolt. 

LEMO cables have been connected in a similar way between the grounds on the adaptor boards for each FEE. Short LEMO cables go between J6 on pairs of adaptor boards in the same FEE holder and long LEMO cables go between J7 on adaptor boards in different FEE holders. 

The LEMO grounds from the adaptor boards and the ground cables attached to the FEE bodies are then connected together and then connected to the CRYRING grounding cable. 

Pulser

Jan has kindly lent us a pulser. We send the pulser to an inverter giving us a +ve and -ve signal. 

The +ve signal goes into the pn FEEs (1,2,5,6)

The -ve signal goes into the nn FEEs (3,4,78)

LEMO cable connecting FEEs in the same holder using J3 on adaptor board. J4 is used for recieving the pulse signal, connecting FEEs in different holders or terminating the signal. 

Power, Ethernet and HDMI cables connected to FEEs. To prevent cables coming out of connectors, power and HDMI cables are zip tied to FEE cooling pipe (see attached image).

Correct FEE MAC addresses have been updated in /etc/dhcp/dhcpd.conf

Remote access to carme sever has been setup. Remote access instructions in elogs 176-179. 

New ASIC settings which reflect polarity of FEES produced 2022Jan27-16-43-00 

New Options file updated to point to new ASIC settings. 

/MIDAS/config/TclHttpd/carme-gsi@8015/startup.tcl updated for 8 FEEs 

/MIDAS/Linux/startup/NewMerger Updated line from "./master64 -i 16 -l 16 -p 11001  &" to "./master64 -i 8 -l 16 -p 11001  &"

Quick motors test performed. Motors accesible remotely (elog 179). Motors moving output (BNC 1 on motors box) connected to motors module (BNC 1). When not moving +12V is observed at BNC 2 on motors module. When moving 0V is observed on BNC 2.  

1. ssh <user>@lxpool.gsi.de
2. ssh <user or carme>@atppc022.gsi.de
3. ssh pi@CARMEmotorspi (Standard npg password)

To use both anydesk and firefox on the carme server require an internet connection.

As the server is on the cryring local network an ssh tunnel needs to be opened to the GSI proxy server.

To do this: ssh -L 8080:proxy.gsi.de:8080 carme@atppc025` and leave terminal open and running.
Both anydesk and firefox have been configured to use the proxy localhost:8080

For the carme password ask J. Marsh or C. Bruno

1. ssh into lxpool network: ssh <user>@lxpool.gsi.de
2. ssh into atppc022.gsi.de: ssh carme@atppc022.gsi.de (This is an rpi access server. Don't run code)
3. ssh into carme server: ssh npg@carme-gsi

For the password to the carme user account on the cryring network please contact O. Hall or J. Marsh.

Water cooling has been connected to the FEE's. Flow meter installed to outlet manifold by way of two right angled pieces. No leaks observed rom any points in the coolant loop, loop will be bled using taps tomorrow. Water turned OFF overnight. 

FEE's powered up with the MAC addresses of each read. 1 is the very bottom FEE and 8 the very top. FEE's have also labelled with the MAC addresses.  

1. 0xd8 0x80 0x39 0x42 0x0d 0x0c  

2. d8:80:39:41:d8:2b

3. d8:80:39:41:d0:05

4. 0xd8 0x80 0x39 0x41 0xd7 0xc2  (Now AIDA 5 see elog 185)

5. 0xd8 0x80 0x39 0x41 0xb4 0x0b (Now AIDA 4 see elog 185)

6. d8:80:39:41:d8:20

7. 0xd8 0x80 0x39 0x41 0xb4 0x16

8 .0xd8 0x80 0x39 0x41 0xd7 0xcd  

MACB firmware has been updated by Nic

One ion pump showed the warning 'arcing' (see attached photo) and was subsequently turned OFF. Most likely cause is the cable was moved during installlation of the flow meter as the pump showed no error this morning. The pump was turned back ON, no error observed -> cable not damaged but sensitive to movement.   

All 8 FEEs have been installed in their holders.

The damaged feedthrough does not have the D-connector on the adaptor board inserted into the falnge but the FEE is connected to the adaptor board. 

FEE's were carefully installed, with one person inserting FEE and another making sure FEE was straight and on the correct pin positions. The spring lock on the bottom FEE's have been tightend and zip ties have been placed to make sure they are secured. 

Rack has been populated with cables from the rack to chamber laid out. Cables mostly go above the rack and hang off the large frame. This frame or the CARME frame takes most of the weight of the cables before they are plugged into the FEEs.   

Cables are numbered 1-8. Brown ethernet cables are numbered 1-4, green 5,6 and yellow 7,8. Numbers correspnd to Bottom FEE (1,2), Side bottom FEE (3,4), Side top FEE (5,6), Top FEE (7,8)

Vaccum
DN250CF 1x gaskets     
DN200CF 5x gaskets  
DN160CF 1x gaskets, 2x blank flanges   
DN100CF 69x gaskets, 19x blank flanges, 14 double feedthroughs (1 damaged) 
DN63CF 3x gaskets, 2x blank flanges    
DN300CF 5x gaskets
DN350CF 0x gaskets
1X WIRE SEAL
DN35CF 3x gaskets, 13 blank flanges   
DN16CF 22x silver gaskets, 27x copper gaskets   

Swagelock
SS-QTM2-D-8M0 Male quick disconnect X18
SS-QTM2A-B-8M0 Female quick disconnect X21
SS-8M0-1-6RS Manifold adaptor X39
NY-8M3-1 Front ferrule X118
NY-8M4-1 Back ferrule X120
Stainless ferrules XMANY

FEE/FEE HOLDER PARTS
FEES x1
Fully assembled FEE holder (MINUS ADAPTOR BOARDS) x1
Cylindrical supports x32
Feedthrough mounts x12
Black adaptor board mounts x11
Spring loaded FEE lock x26
FEE holder side panels x5 pairs
Adaptor boards x2 (1 slightly bent pins)
FEE rail bolts x4
M3X16 281-013 x91
M3 locknuts 260-5815 x34
Springs 821-273 x40 
M4x25 (2mm too long) 281-057 x32
M6x20 281-120 x69
M3x6 280-981 x34
Assembled Hose 1.5M (Red) x8
Assembled Hose 1.5M (Blue) x8
Blue Hose ~16M
Red Hose ~10M

We require additional gaskets for those we are low on (<5) DN35,63,160,250,350 and more wire seals 

We should have enough swagelock and FEE holder parts to construct another 4 FEE holders for the remaining required for mounting all moving detector FEEs.

We also need fuses for the USB mains relay and power supply -> 2x 250v 10A, 16x 250v 1.6A. Dimensions for both are the same, length=20mm, Diameter=5mm (images attached)

Additional fittings provided by GSI for the manifold are in the swagelock box.

11.00 Per https://elog.ph.ed.ac.uk/CARME/170 and https://elog.ph.ed.ac.uk/CARME/171 issues have been identified with the FEE64 carriers and adaptor PCBs

Some additional comments

1) Per https://elog.ph.ed.ac.uk/CARME/170 attachments 6-9 we can see that there are no cable strain reliefs to which we can attach the FEE64 power, RJ45 network and 
timestamp HDMI cabling. It would probably be straightforward to design something which could be attached to the FEE64 carrier side panels. For the time being we will use 
the copper piping of each FEE64 as the nearest, mechanically stable point to which we secure the cabling.

2) Before the skimming of 1x Delron rail yesterday there was a small amount (<1mm say?) of transverse movement of the FEE64 during insertion into the carrier and it is 
not clear why. Small sample of measurements of FEE64 carriers, spacers etc.

Delron adaptor PCB support widths 92.85, 92.87, 92.90
Al DN100CF flange attachment width 92.97, 92.99, 93.01
Al FEE64 carrier spacing rod 93.03, 93.04, 93.03
FEE64 width (w/ 1x c. 2mm skimmed Delron rail) 100.2, 100.2, 100.3 (3 measurements along the length of the 1x FEE64)
FEE64 carrier width 102.8, 103.0 (2 measurements of 1x FEE64 carrier)


probably indicates that the spacing between the FEE64 plates is consistent. There is some variability (~0.1mm) in the Delron adaptor PCB which could affect the relative 
position of the ERNI pins and sockets. Is there some variability in the width of the FEE64s (w/ rails)?

The ERNI connector is not central on the adaptor boards causing misallignment with the FEE. 

The adaptor board is 80mm long. The ERNI connector is 50mm long. The connector is 13mm from the 2x Lemo side and 17mm from the 3x Lemo side. This results in misallignment between the FEE and adaptor board. 

To correct for misallignment, 2mm will be skimmed from one of the rails on the FEE. 

Skimming the rail allows the FEE to now be inserted straight into the adaptor board connector. Attached image shows test FEE holder on the bench. FEE needs to be carefully inserted to make sure it is straight as it can now move around within the FEE holder with the 2mm skimmed off the rails. For CARME, where sight lines and space are limited, one person is required to hold and push in the FEE whilst another checks the positioning of the connectors is straight. 

4 FEE holders installed on CARME. Mounting procedure devised due to limited working space to install.

Mounting procedure:
>Install FEE holder mounts (image 1) onto feedthrough flange

>Partially make up FEE holders on workbench. Only 3 cylindrical support rods, spring latches and one of the black adaptor card mounts should be attached to aid install. Bolts should be loose to aid install on CARME. 

>Mount FEE holder to flange

>Connect first adaptor card into D connector on feedthrough, then screw adaptor card to black mount. 

>Screw in second black adaptor card mount but do not tighten bolts. Connect second adaptor card to D connector and screw to black mount. 

>Attach remaining cylindrical supports and tighten all bolts.   

Bolts to install FEE holder mount onto flange (M4x25) are 2mm too long for the hole on the flange. Washer were initially used however this prevents adaptor cards being able to be secured onto thier black mounts. Bolts were shortend 2mm by davide to install. 5 shortend bolts remain in addition to thebag of unshortend M4x25 bolts. 

On the nn flange of the bottom detector (non bias pin D-connector) the feedthrough is damaged. Damage may be repairable, to prevent further damage an adaptor card was not plugged into this D-connector. (image 1)

A bias test was performed for the top detector once adaptor cards are installed. First test is the bias test performed prior to adaptor cards being installed, second test was performed after installation of adaptor cards.   

Installing FEEs on the outer side of the ring in the future may have some mechanical conflict with the gas target frame, see attached images.