Found CARME at 50C, a bit cooler than expected, probably due to invasive tent changes yesterday to mount missing wires. Increased heaters temperature to 300C

9:35 Started D-2000 #1,3 (behind the DSSD). T=400, tramp 30, thold 90

11:20 Set D-2000 1,3 to 200C. Turned on D-2000 2,4 as above 

13:15 Set D-2000 2,4 to 200C. Turned on UHV1400 set 1 at 250W, tramp 30, thold infinity (will ramp down manually as required)

13:20 No appreciable effects on DSSD temperature. Turned on set 2 as well, which should be mounted close nearby.

13:45 Both sets reached 250W. No effects on detector temperature. Increased to 300W. Vacuum gets slightly worse, but recovers quickly. Not worth further increasing power. Leaving for 90 minutes.

15:20 Strain relief TC at 75C quite hotter than detector TC at 35C. Might be normal. Vacuum division IE514 must have been degassed remotely? Decreased set 1,2 to 75W. Turned on sets 3,4 to 250W as above. 

15:45 Sets 3, 4 now at 250W. Detector TC heating quickly at 13C/h but nowhere near dangerous temperatures yet. Decreased sets 1,2 to 50W. Leaving for 90 minutes.

17:15 Sets 3, 4 to 50W. Slightly worrying temperature difference between detectors and strain relief around 65C. Different materials and different positions could explain this.

17:40 Oddly, detector temperature now going up sharply (temperature diff. down to 40C) since the uhv1400 power was decreased. Cannot think of s logical reason,  it may just be a coincidence and heat transfer is catching up.

Leaving overnight. Will turn on the last two groups of UHV1400 tomorrow morning.

We are moving the detector in run 56

Run number changed 09:44 - Now run 23

Moving motors with the injection

Ring signal very weird, moving in for short time and staying out for 15+ seconds.

R24

Right arm FULLY in (70.49 mm)

Left arm SERVOS ONLY in (36.54)

No appreciable effect on beam (not clipping beam at injection / during circulation)

ML fixed signal cycle. It's now 8 seconds in, 5 seconds out. Motors still not moving. Acquiring more stats in the current conditions.

11:07 - Temps & System wide checks good

Stats as before.

Leakage currents & pressure good.

11:44

Leakage current ok. pressure good. Temperatures ok. Merger ok. System wide checks ok.

In with the left 36.5

In with the right 70.5

Attatchment 1: Rates

Attatchment 2: Temperatures

Attatchment 3: Waveforms layout 7

Attatchment 4: Waveforms layout 8

Attatchment 5: Waveforms layout 1

Attatchment 6: Waveforms layout 2

Attatchment 7: Leakage current

Attatchment 8: lowenergyXYtotal all 4 dets

Attatchment 9: lowenergyExEy all dets

Attatchment 10: loveenergyvExEy

Attachment 11: screenshot from the Shottky

14:00 Run 24 stopped to allow access to the cave.

Helium and liquid nitrogen re-filled in the target.

Pulser walkthrough performed R25.

Turned CARME turbo ON, pressure in the ring with target on is reduced by a factor 2.

Tried to find electron cooler voltages to determine the beam energy. We were unable to do this. Hopefully we find out tomorrow. 

Attempted first motor test in XHV. Wired and connected limit switches and string potentiometer.

Set up raspberry pi to control motors (.124). Set up pi for remote headless access.

Moved jumper on Teknic hub from 1 to 2 motors configuration.

Linux code only sees one motor and gives issues due to improperly set up limit switches.

Switched to Windows Clear path program. Left servo motor refuses to move and sounds stuck. Unable to move by hand - unclear how braking is still active.

Right servo motor initially moves, but got stuck after one move, on limit switch. Complain about sensorless startup which refers to low bias or excessive torque. The second looks more likely from the display but unclear why. Brake engaged for this motor as well? Unable to move by hand even after disconnecting all cables.

Removed safety pin from bellows (compressed air still missing, to be fixed next week). Motors move in. Pressure from 2E-12 to 3.5E-12 mbar and quickly recovers (seconds). Subsequent move cause smaller pressure fluctuation. All good.

Will contact PB too figure out issue with motors. Problem unclear. Might require some dismounting.

Neg and ion cables were taken off and were put in the box.

Top support was taken off.

Motors are rotating again. Have to be careful with the position of the limit switches.

Mounted both  motors on CARME. Started 
with outside motor.

M6x160 screws proved slightly too long to 
fix bellows to rod and seal the vacuum. 
Had to be manually shortened by 1 cm 
roughly.

Mounted whole assembly but could not fix 
actuator rod to moving plate due to holes 
mis-alignment.

Mounted motor on CARME but could not align 
holes on flange and chamber without 
turning motor to the side... see attach 3 and 4


Got in contact with Peter over lunch and 
understood issue.

Mounted second motor with experience 
acquired. No issues. All screws go in. 
Motor box level. See attach 5

Will need to dismount / remount first 
motor box before pumping down. Currently 
secure on CARME thanks to adjustable 
support.

Note we forgot to buy DN 40 CF gaskets. ML 
supplied a few. Will buy more. Needed 
between bellows and rod.


Strain relief mentioned in previous ELOG 
entry was taken to workshop and hole re-
drilled. Workshop reports other hole is OK 
for M3 screw. Either screw used or hole 
were poorly machined. M3 screws should be 
correct. Re-machined part sent for 
cleaning.

Will try with clean parts again tomorrow 
unless we remount motors.

Activities for the day cut shorter due to 
SPARC meeting this week.

The motors were moved using the pneumatic only, in and out for several cycles to gauge the effect on the vacuum. For the left arm no change in pressure was observed. For the right arm a small increase of 0.04e-7 mbar was observed which was quickly reduced by the turbo. 

The detectors were then biased to see if movement had any effect on the leakage current (large effects were observed in 2021 beamtime). Movement of the pneumatics saw no observable change in the leakage current above normal fluctuations. 

Both motors arms had been sucked into the chamber by 5-10mm due to the vacuum before starting the movement cycle. After one movement cycle, the motors returned to their normal fully out positions and remained fully out. 

Prepared the motors to be mounted and 
found all components required. No issues. 
May have to partly dismount again to fix 
bellows depending on ease of mounting.

Note extremely long screws mentioned in 
previous Elog inventory are to be used for 
the motors.

Attempted to swap Pirani gauge with ours 
but realised it's CF and needs appropriate 
port without risk of baking. That port 
would be on the interaction chamber which 
is not going to be mounted. Will probably 
keep using ABD gauge for now.

Peter spotted mistake in mounting front 
foot of CARME which is back to front. 
Correct mount will gain us 2-3 cm but will 
conflict with current CARME position fixed 
by tube on the back. Will discuss options 
with ML.

Motor mounting to take place on Mon with 
local technical support. No activities 
tomorrow unless a solution for the clamps 
mentioned in previous entry is found (i.e. 
what size bolts go on top). Peter will 
check step file and we will consult with 
local engineers on Mon pm.

11:00

Changed slow comp threshold to 0x32 (500 keV). Checkloaded.

Temperature, Statistics fine (see screenshots).

System wide checks fine (see below).

All FEEs pass clock check.

All FEEs pass White Rabbit check.

ADC Calibration check:

FEE64 module aida02 failed
FEE64 module aida16 failed
Calibration test result: Passed 14, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

FPGA timestamp error check:

         Base         Current         Difference
aida09 fault      0x1 :      0xa :      9  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

09:10

Arrived at GSI. Leakage currents status same as before, DSSD4 is still fluctuating. DSSD1 and DSSD3 currents are definitely lower again compared to previous days. I am convinced they are decreasing over time, whether this will be enough for our measurement....

While in the cave I again removed the motors pins so that the detectors can can be shook in an attempt to perform magic. While in the cave I also connected the CRYRING ground cable to the coolant pipe (ground config 11). Will see effect on rates / pulser.

Ground config 11 - see attachments 1-8. Rates in aida03 are massively down by a factor 10! Rates now ~20k. Aida04 still around 300k. On attachment 4 we can see the noise from zero energy on aida03 is massively down and we can now see the pulser, although the peak is double. Similar low energy noise is now appearing on aida15 and aida16 of DSSD4. Is this the early phase of what happended to DSSD1 and DSSD3?

Note the waveforms shown in attachments 7 and 8 change quite alot when updating the spectrum browser. Wave structure very visible in 7 but after updating a few times this disappears before coming back again a bit later.

20 minutes later ...

DAQ continued for about 20 minutes, noticed the leakage current in DSSD4 was no longer fluctuating. Zero'd stats and histograms to look again.

Stats in aida03 have increased back to ~100k, still less than before but back up from ~20k. Check of the pulser widths shows they have decreased across the board from the previous test 20 mins ago.

Leaving DAQ to run for a bit longer, the rates in aida03 continue to increase and are now back ~200k (back where it was).

I am currently trying to cycle motors. No effect on leakage current when cycling pneumatic motor. Note when using single commands such as 'l p in' I have no problems. But when trying a longer command where I ask the code to wait before pulling the pneumatic out again the code crashes. ??? Crash not repeated by Carlo remotely, Perhaps it is my laptop, or a bug relating to windows. Will investigate back in Edinburgh.

Have performed the same bias test for DDSD4 as #1 and #2 in previous elogs. Same behaviour observed. No (0.01uA) leakage current with only nn and current ramps up and falls to zero with only pn. I did notice that the ion pumps were arcing, I have not noticed this in prevous days. They have now been turned off, I think we can leave them off as they do little for the vacuum anyway unless using heavy gas taget gas. VI bias curve looks identical to one performed at the start of the week.

I have moved the PC power to the 19 inch rack. Power cycled the FEEs. I noticed the leakage current on DSSD4 was fine during VI test but after starting the FEEs power cycle they started to fluctuate again. After about 5 minutes after the startup, DSSD4 leakage current returned to a normal stable level. I have seen spikes in leakage current for all detectors during power cycle but perhaps it could also cause the fluctuations in #4. Overnight I have left FEEs off due to uncertainty over the swagelock on aida08 and the fact I was changing ground a lot so a background didn't make sense. Perhaps leaving FEE power on will stabilise DSSD4? I will leave system to stabilise a bit now (after seeing rates and pulser widths change in previosu test) and check pulser widths. Going for an awful GSI coffee.

New configuration: See attachments 9-13. Rates on aida03 and aida04 still remain very high >200k. Pulser widths across the board look pretty good. All have reduced since moving the PC on same power supply as 19 inch rack. DSSD4 leakage current has been stable the last hour since FEE power on. Waveforms look improved too. No longer see such high amplitude oscillations. Updated several times, see no change in waveforms like previous test.

Attachment 14 for pulser widrths from tests this week. Note x means no value. This could be because the width was so very big, or the test input wasn't connected. These values aren't too bad. DSSD2 and 3 have avergae pulser width (1.8L only) of ~60 keV.

Tried ground cable to NIM bin, had no effect / maybe negative effect. Disconnected cable.

D+D tests in next elog

Finally a pulser walkthrough was completed. Run8 has pulser settings unchanged, slow comparator at 0x64, aida03 and aida04 set to MEC. Pulser amplitudes 9000,8000,7000,6000,5000,4000,3000,2000,1000

A second walkthrough (Run9) was done with slow comparator set to 0x14, aida03 and aida04 set to MEC. Pulser amplitudes 9000,8000,7000,6000,5000,4000,3000,2000,1000,700,400 (so that we can check ~BBN energies) - attachment 15

During second walkthrough, DSSD4 starting fluctuating in leakage current again. No-one was in the ring. No observable change. FEEs were ON and were currently acquiring. ?? See noise which in attachment 15 on aida15/16 that only started when current was fluctuating.

FEEs off. Bias on and monitoring. Pins re-inserted into motors. Compressed air off.

Attachments 1-5 - CARME shipment in CRYRING hall
                  XHV chambers nearest to YR09

Will remove top and sides of XHV chamber crate and install 4x SAES D-2000, 1x Leybold Mag Lev 400 turbo and 1x Leybold Ionvac IE514 gauge for basic vacuum test.

Attachments 6-7 - spare wire seals shipped from Edinburgh to GSI
                  poor packing and/or re-packing has resulted in bending of wire seals
                  wire seals shipped from STFC DL to GSI appear to be OK - see attachment 1
                  

10.00 FEE64 power ON
      CARME IE514 OFF

      Confirm DAQ SETUP restores ASIC settings 2024Apr24-14.14.03 for *all* FEE64s

      DSSSD bias & leakage current - attachment 1
       leakage currents remain high cf. June 2024 - status of vacuum group controlled ion gauge unknown

12.10 FEE64 power ON

      DSSSD bias & leakage current - attachment 2

      FEE64 temps OK - attachment 3

      All system wide checks OK

      WR timestamps OK - attachment 4

      per FEE64 Rate spectra - attachments 5-6

      ADC data item stats - attachment 7

13.45 ML says vacuum group have switched off YR09DD1AS
      DSSSD bias & leakage currents - attachment 8
       DSSSD #2 and DSSSD #4 leakage currents return to 'normal' values cf. June 2024
       DSSSD #1 and DSSSD #3 leakage currents reduce by c. 10-15% only and remain high cf. June 2024 

17.47 DSSSD bias OFF
      FEE6 Power OFF
   

Test + and test - daisy chain cabling extended from DSSSD #1 to #2 to DSSSD #1 to DSSSD #4
 terminated by 50 Ohm at end of daisy chain

Grounding cabling extended from DSSSD #1 ( only ) to DSSSD #1 to DSSSD #4
 grounding cables daisy-chained between 4x adaptor PCBs of each DSSSD - no inter-DSSSD grounding cables

DSSSD#1 bias cable SHV-Lemo 00.250 ( to p+n bias adaptor PCB )
DSSSD#2 - DSSSD #4 bias cables SHV - 2x Lemo 00.250 ( to p+n/n+n bias adaptor PCBs )

LK1 fitted for DSSSD #1 & #2 n+n Ohmic side bias adaptor PCBs
LK2 not fitted on any adaptor PCBs? to be checked

CAEN N1419ET internal jumper fitted ( non-floating outputs )? to be checked

16.54 DSSSD bias OK - see attachment 1

      FEE64 temperatures - attachment 2
       aida06 ASIC temp low

      System wide checks and WR - attachments 3-6
       global clock status aid04 0x6, aida06 0x4
       multiple FPGA timestamp and WR decoder errors for MACBs 4 and 5 ( of 5 )

       WR timestamps OK for aida01-aida08 ( w/ MACB firmware version 04/20 )

     ADC data item stats - attachment 7
      all rates consistent with 50Hz pulser *except* 
       aida11 global clock fail
       aida07 no boot
       aida16 no power
       aida06 ? 
       aida15 faulty asic #2

     per FEE64 Rate spectra - attachment 8

     per p+n and n+n FEE64 1.8.L spectra - attachments 9-12
      Pulser peak width p+n side 52 ch FWHM ( aida12 ), n+n side 70 ch FWHM ( aida14 )

     per p+n and n+n 1.8.W spectra - 20us FSR - attachments 13-16

09.50 CARME IE514 5.1e-11 mbar ( ambient temperature 16 deg C )

10.24 DSSD bias OK - attachment 1

10.40 MACB configuration

      1              2      3      4      5

mode  0x3            0x3    0x3    0x3    0x3

      from VETAR2    from 1 from 1 from 1 from 1
      to 2           aida01 aida05 aida09 aida13
      to 3           :      :      :      :
      to 4           :      :      :      :
      to 5           aida04 aida08 aida12 aida16

WR emulator cables connected to MACB #1 rear panel

DAQ reset and DAQ setup between each test below - the FEE64s were NOT power cycled between tests


      Per https://elog.ph.ed.ac.uk/CARME/478 test WR

      connecting/disconnecting HDMI cable from VETAR2 to MACB level 0 input was tested 20.1.24 per https://elog.ph.ed.ac.uk/CARME/477

      With VETAR2 abd all MACB mode sttings 0x3 observe same WR issues as previously

      Noted clock & sync cables from WR emulator were swapped
      ( pay attention to SMA cables cf. Zybo board *not* NIM module as shown in https://elog.ph.ed.ac.uk/DESPEC/517 )

      Change MACB level 0 mode from 0x3 to 0xd - all other MACBs mode 0x3
      HDMI cabkle from VETAR2 to MACB level 0 input disconnected

      System wide checks - global clocks all fail - see attachments 2-5


11.14 Reduce MAC config to MACB level 0 plus one MACB connected to FEE64s aida01-aida4
      All MACB modes 0x3

      System wide checks aida01-aida04 OK. Same problem observed with WR timestamp for aida01-aida04

      See attachments 6-9

11.22 Reduce MAC config to one MACB connected to FEE64s aida01-aida4
      MACB mode 0x3

      System wide checks aida01-aida04 OK. Same problem observed with WR timestamp for aida01-aida04

      See attachments 10-13

11.26 One MACB config using MACB 2 of 5 in NIM bin connected to FEE64s aida01-aida4
      MACB mode 0x3

      System wide checks aida01-aida04 OK. Same problem observed with WR timestamp for aida01-aida04

      See attachments 14-17

11.36 Connect VETAR2 output directly to aida09

      System wide checks for aida09 OK.Valid WR timestamp appears to be generated.

      See attachments 18-20

      Check NIM bin with MACBs etc
        +/-6V, 12V and 24V lines OK

11.42 Connect VETAR2 to MACB 1 of 5 ( mode 0x3 )
      Connect aida09 to MACB 1 of 5
      Global clock OK. WR decoder status error 0xd reported for aida09. Valid WR timestamp for aida09.

      See attachments 21-24

11.53 Connect VETAR2 to MACB 1 of 5 ( mode 0x3 )
      Connect aida09-aida12 to MACB 1 of 5
      Global clock OK. WR decoder status error 0xd reported for aida09-aida12. WR timestamps for aida09-aida12 very different - aida10 might be correct timestamp?

      See attachments 25-28

11.53 Connect VETAR2 to MACB 2 of 5 ( mode 0x3 )
      Connect aida09-aida12 to MACB 2 of 5
      Global clock OK. WR decoder status error 0xd reported for aida09-aida12. WR timestamps for aida09-aida12 very different - aida10 & 11 approx correct but 11 has 
lower timestamp than 10 so probably not quite right

      See attachments 29-32


15.00 Per https://elog.ph.ed.ac.uk/CARME/478 systematically test 1x MACB configurations to find 'working' MACBs

      Identify 3x MACB modules which produce sensible WR timestamps - all labelled ' firmware version 04/20 '.
      The other 5x MACBs are either unlabelled or labelled unknown.

      MACB configuration now uses 04/20 MACBs for MACBs 1-3 and unlabelled 2xMACBs fpr MACBs 4-5
      aida01-aida08 should produce WR timestamps OK

      Will attempt to update firmwre of 5x MACBs tomorrow pm with NH

15.51 1.8.W spectra 20ms, 2ms, 200us, 20us FSR - attachments 1-8

15.55 Lost control of motors - moving DSSSDs out

15.56 new run number 79

      ignore run number 80

17:58 new run number 81
      CB sets motion cycle from 20.28mm to 64.36mm

      Observe a second, higher energy peak during the part of the cycle where the DSSSD is moved out - see attachments 9-10

      Peak appears only when moving out. Does not appear when moving in/out manually on beam storage plateau. Does not appear when moving in synched with ring, waiting less than the storage time, and moving out.
      All seems to indicate it is due to beam injection and not moving out fast enough. Very weak halo piling up? Should be clear in the data depending on rate / strips affected.



18.45 new run number 82
      DSSSD position fixed 34.89mm

22.10 system wide checks OK *except* WR decoder status

	 Base 		Current 	Difference
aida01 fault 	 0xab60 : 	 0xab8c : 	 44  
aida02 fault 	 0x88d3 : 	 0x88fe : 	 43  
aida03 fault 	 0x23aa : 	 0x23d6 : 	 44  
aida04 fault 	 0x15d7 : 	 0x1603 : 	 44  
aida05 fault 	 0x2f03 : 	 0x2f0f : 	 12  
White Rabbit error counter test result: Passed 0, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 temperatures OK - attachment 11

      ADC data item stats OK - attachment 12 

      merger, tape server and console OK - no merger error messages since startup - attachments 13-15


DSSSD bias -100V leakage current-3.2360uA

     run82 - integrated x & y strip energy spectra - attachments 16-17

00.09 DAQ continues data file R3_21

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4

      Merger, TapeServer etc - attachments 5-7

04.50 DAQ continues data file R3_22

      DSSSD bias & leakage current OK - attachment 8

      FEE64 temps OK - attachment 9

      ADC data item stats OK - attachment 10

      per FEE64 Rate spectra - attachment 11

      Merger, TapeServer etc - attachments 12-14


09.41 DAQ continues data file R3_23

      DSSSD bias & leakage current OK - attachment 15

      FEE64 temps OK - attachment 16

      ADC data item stats OK - attachment 17

      per FEE64 Rate spectra - attachment 18

      Merger, TapeServer etc - attachments 19-21


09.43 DAQ stopped
      DSSSD bias OFF
      FEE64 power OFF

09.35 CARME IE514 1.95e-8 mbar, ambient temperature 20.8 deg C

      CARME bakeout monitor, ELOTECH R4000 temperature controller front panel and Grafana bakeout dashboard- attachments 1-3

      No change to ELOTECH R4000 set point

17.45 CARME IE514 1.91e-8 mbar, ambient temperature 20.8 deg C

      CARME bakeout monitor, ELOTECH R4000 temperature controller front panel and Grafana bakeout dashboard- attachments 4-6

      No change to ELOTECH R4000 set point

08.35 DSSSD#3 bias -149.95V leakage current -6.418uA temperature +19.6 deg C ( https://web-docs.gsi.de/~lestinsk/tempplot.php )

DSSSD#3 bias OFF

Cooling water manifolds


The manifolds were returned this afternoon.

Inflow manifold - blue hose, female Swagelok connectors

Attached to inflow hose and bled using bleed/drain valve.
Small leak observed at the 3/8" connection between the manifold and the bleed/drain valve - attachment tightened and leak fixed.
Inflow manifold installed.

Outflow manifold - red hose, male Swagelok connectors

Attached to outflow hose and bled using bleed/drain valve.
Small leak observed at the weld immediately *below* the weld that had been re-worked - this weld had not been re-worked.
Significant leak observed at one of the male Swagelok connectors in the lower half of the manifold - it does not seal correctly.


DR will return this manifold to the workshop for welding re-work Tuesday am.
The faulty male Swagelok will be replaced.


AIDA cabling


16x AIDA PSU, HDMI and RJ45 Cat 6 cables and 4x SHV-BNC cables required.

Available

8x AIDA PSU cables 
1x AIDA PSU cable ( in one of two white plastic storage boxs with 7x FEE64s returned from STFC DL ) now appears to be missing

16x AIDA HDMI cables 

8x RJ45 Cat 6 network cables
1x RJ45 Cat 6 network cable ( network switch to CRYRING network patch panel - to be replaced )
3x RJ45 Cat 6 network cables found in AIDA plastic box ( S4 Messehutte )

3x SHV-BNC 

Will check DESPEC store above ESR with NH tomorrow afternoon.