Run 93- New run at 150keV, background with beam

13:15 All checks ok

---9 July---

11:00

Turned detectors on - all stable see attachment 1 

AIDA started - all okay. See attachment 2 for FEE temps.  

All FEEs set to LEC (0x0) and slow comparator threshold set to 300keV (0x1e). See attachment 3. 

After some issues tape server and merger all set up, NetVar not working??? have tried to reboot things but still getting issues. See attachment 4 for tape service and attachment 5 for merger. 

Currently writing to data to G22-205/R98 (R97 empty), doing background run.

13:30

All checks okay apart from white rabbit, see attachment 6.

Attachment 7 - stats 

18:30 CEST

More issues with anydesk - now refuses to connect with error message desk_rt_ipc

May be as simple as the monitor being turned off. Will have to be checked in person

Connected to atppc025 via x2go and from there checked

- Temperatures are fine (attached)

- Detector bias / leakage current is fine (attached)

- Data rate from file size being written is around 500 kB/s (fine)

- A number of interesting features in the online background spectrum. Likely all non-physical (attached)

- Stats OK (attached)

20:50 CEST

All checks good.

22:28 CEST

All checks good.

Features in the XY plot rensemble beam hitting the detectors. Attached.

Leakage currents and rates are good - unlikely to be major detector damage. Attached.

Only beam in the ring between now and end of D+D was stable uranium.

Killed CARMEsort via terminal (still no access to anydesk).

Restarted CARMEsort. No obvious structures appear after a few minutes. Was there beam in the ring today?

23:09 CEST

All checks good

Features in the XY plot and Ey plots disappeared.

---10 July---

03:22 CEST

All checks good.

Features gone. Attached.

05:15 CEST

All checks good.

07:34 CEST

All checks good.

10:40 CEST

All checks good. 

12:35 CEST

All checks good. Some features in plots again appeared in last 2 hours, attached Ey plot (attachment 15). 

14:40 CEST

All checks okay. 

16:40 CEST

All checks okay. 

18:40 CEST

All checks okay. Attachment 16 leakage current stable. 

Attachment 17 Ey plot.

19:35 CEST

All checks OK.

Reset the Root spectra again - highly unlikely to have beam overnight without experiments running.

23:39 CEST

All checks OK.

Still seeing peak at 6 MeV like attachment 17, after resetting the Root spectra. Detector damage? Still seems implausible.

Events are focussed as in attachment 12.

Reset again. No obvious peak or pattern after a few minutes running.

From Claude K

"CRYRING was running with the Uranium beam from ESR until Tuesday (8 July) night around 22:00.

From Wednesday noon we started to inject D+ from the local source and had the D+ beam stored at various energies from 14:30 until 17:40. During the night from Wednesday (9 July) to Thursday (10 July) there was no beam.

We then continued measurements with stored D+ On Thursday from ca. 11:00 to 21:00. Since then CRYRING is in shutdown."

---11 July---

08:07 CEST

All checks OK.

Peak at 6 MeV did not re-appear overnight. Attached x2.

12:00 CEST

All checks good! All plots look good!

16:00 CEST 

All checks okay! 

19:20 CEST

All checks OK.

Background peaks almost visible.

22:53 CEST

All checks OK.

---12 July---

07:25 CEST

All checks OK.

10:00 CEST

All checks OK. Temps attached.

11:00 CEST

All checks OK.

19:00 CEST

All checks OK.

23:00 CEST

All checks OK.

---13 July---

07:42 CEST

All checks OK.

11:00 CEST

All checks OK.

ELOG down during the afternoon

--14 July--

11:00 CEST

All checks ok. 

14:40 CEST

Tape service and merger now working on atppc025

All checks ok, briefly went no storage at around 14:38 CEST for 1-2 minutes whilst getting tape service and merger up. 

17:40 CEST

All checks ok. 

21:30 CEST
All checks OK.

23:16 CEST

All checks OK. Several screenshots attached.

Calibration appears to hinders clear alpha peaks.

---15 July---

07:47 CEST

All checks OK.

11:55 CEST

All checks ok. 

14:35 CEST

Issue with atppc025, see attachment 29 for error message (desktop 2, detectors etc)

Got error to go, all checks ok. 

17:35 CEST

Same issue with atppc025, otherwise all checks ok. 

20:30 CEST
All checks OK.

23:07 CEST

All checks OK

---16 July---

07:41  CEST

All checks OK

12:00 CEST

All check ok. 

15:15 CEST

Some fees have high rates, all other checks ok. 

15:32 CEST

The above behaviour appears to be correlated with an increase in leakage current across all four detectors. Most likely light?

The background is not reliable -> No storage.

Unfortunately the leakage current logger crashed a few days ago on the server, and without anydesk we did not realise. Restarted it on atppc025.

Informed ML about issue. If it cannot be solved we could increase the thresholds and keep going, but the background may not be representative.

16:35 CEST

Issue persists. ML and ZA are not aware of any changes, except power-saving mode enabled for cave AC. No effect on FEE temperatures - not the issue.

aida04 rates shows 0. Checkloaded. Fine for a few seconds. Back to 0. Data event rate 100,000 may be too high.

Increased thresholds to 1 MeV. Data rates lowered, but e.g. on aida04 still 40,000. Goes back to zero. Checkloaded. OK for a short while, then back to zero.

Tape server stopped. DAQ stopped - all FEEs respond, good! Merger still going.

DSSDs off. Will retry tomorrow morning.

23:44 CEST

Temps OK.

Run 92- started at 09:00, background with beam

Same settings as run 91

See attachments for rates etc

All checks okay

09:58 Run stopped as now tuning 150keV beam

10:00 All checks ok

10:25 D2 bottle pack pressure 30 bar approximately

12:18 All checks ok

17:00

Scraper tests conducted. We tried moving in the scrapers both at a fixed position throughout and also moving with the injection cycle.

We were unable to find a good position with the scrapers that did not produce significant scattering in the detectors.

Therefore we have moved the scrapers back to the fully out position.

Run 42 started.

We have increased the storage time to 10 reps (1 minute) in order to investigate the shoulders on the 3He peaks online and whether it is likely from injection. I think it is likely just a calibration issue

D+ beam deuterium target. D+ beam energy is 0.75 MeV/u. (beam calc calculation is 0.7619 MeV/u). See previous entry for beamcalc.

Beam calc - Attachment 1

Target density is about 2e12 at/cm2

All four detectors positioned fully out and stationary.

Left potentiometer = 18.86 mm

Right potentiometer = 19.8 mm

Ring cycle time is 60 seconds. Note we see some hydrogen in in the spectrum, this disapears after a few seconds.

Target T1=30.0K, T2=31.6K.

Nozzle position x=-2.96, z=1.72 mm

YR09-Etarget pressure = 2.0e-10 mbar.

Cooler set value = 439 V, 15 mA. Read value = 439.81 V, 12.88 mA

All leakage currents look good

Beam position. Vertical =0.0mm , Horizontal=-1.5 mm

Beam intensity ~6e8 on bunched beam, 6e8 on coasting.

Data rate ~ 416kB/s

slow comparator threshold to 0x24 (360 keV)

Run 20 running

We start a new run since the laser guys will do some measurements.

Zoran will increase the beam intensity

Detectors are out position

All checks are OK!

13:30

All checks are ok!

7:00

Cooler voltage setpoint 247.4 V.

7:44

E1 Turbo at 808 Hz, 2.75 A

DSSSD bias & leakage current OK 

FEE64 temperatures OK 

ADC data item stats OK 

per FEE64 Rate spectra OK 

Beam intensity 7.5e+7 particles

Target density ~3E10 atoms/cm2

8:30

E1 pressure was too high (2.13E-1 mbar) and turbo frequency too low (807 Hz).

We increased T1 in steps of 0.2 K to 39.6 K, which improved in E1 (2E-1 mbar) and turbo freq. (813 Hz).

T1 was decreased again to 39.4 K.

After speaking to Nikos we slightly decreased the gas flow (~15 bar) as desribed in elog entry 631.

Values now: turbo (817 Hz), E1 pressure (1.78E-1 mbar) -> both improved

10:06

The gas flow was decreased in three steps, there is now change visible on the gauge (still ~15 bar).

The turbo frequency ist stable at 819 Hz and pressure at E1 is 1.45E-1 mbar.

The temperature is currently at 39.0 K and we will start decreasing it to try and bring the target density up.

DSSSD bias & leakage current

OK FEE64 temperatures OK

ADC data item stats OK

per FEE64 Rate spectra OK

Beam intensity 7.5e+7 particles

Target density ~2E10 atoms/cm2

11:50

All system checks OK

Target density is ~3e11 at/cm2.

Temoerature of the nozzle is curretnly 37.5K with E1=1.6e-1 mbar. Turbo running at 820 Hz

14:29

Run stopped

12:52

Started Run 58 - Beam energy 200 keV/u.

Slow comparator threshold set to 0xc (130 keV) for all channels. Fast discriminators turned OFF for all channels by setting enable mask in discrinator control to 0xfffff. See attachments. 

D+ beam deuterium target. D+ beam energy is now 0.2 MeV/u. (beam calc calculation is 0.2037 MeV/u).

All four detectors positioned fully out and stationary.

Ring cycle time is 5 seconds.

Cooler read value = 117.58 V, 0.00178 mA

Target around 8.9E12 atoms/cm2

All checks ok!

Beam on ionisation profile monitor is lost quite rapidly and looks to increase in size quite a lot due to target interactions. We may have to look at lowering target density

Attachment 25 shows IPM with no target and attachment 26 shows IPM with the target 8e12 at/cm2. Compare also to attachment 12 (IPM with target at 2.6e12).  

14:20

Checks are ok!

15:25

Fees 13-16 switched to MEC and bias on DSSD4 switched off

All other checks ok

17:00

Target dropped to 1-2E12 atoms/cm2

All checks okay- 17:25

17:45

Moved nozzle from x=-1500 to x=-2000, density went from 1E12 to 3E12

Moved nozzle again at 18:00 from x=-2000 to x=-3000 in steps of 500, density at 4E12

18:45

Turned bias ON for dssd4 stable at V=10 but unstable at V=20 so bias turned back OFF

19:30

All checks ok!

IPM attached, target density 2.6 atoms/cm2 (attachment 12)

20:00

Tried moving the nozzle more as target density dropped to 2-3E12 again

Moved in 500 steps either side of x=-2500 but didn't improve, then moved in 100 steps from y=1600 to y=2300 and target density now 6E12 at x=-2500 and y=2300

Tried dssd4 again but only stable to V=20 so bias OFF 

21:50- All checks good, target density stable at 8E12

22:50- All checks fine (see attachments), target density at 6E12

00:10

Checks OK. Have zero'd statistics to see pause and resume statements over an hour.

Rates looks OK. ~140 c/h 3He, and ~80 c/h for 3H and 1H each.

Target has been stable at 7e12 at/cm2 for the last hour. 

02:10

Checks OK. See pause statements after 2 hours - attachment 27, very few overall.

03:07

Checks OK. Target (9-11)E12

04:06

Checks OK. Target (9-11)E12

07:00

Checks OK. Target 9.8E12

page AIDA/Check/Check.tml.  failed

ptoblem fixed

09:00

Checks OK. Target 8.9E12

10:32

No target data displayed for 10 minutes, but jet density unchanged. Restored display from remote access by M.L.

11:22

Storage data stopped and Faraday cup inserted to check the leakage current of Detector #4. Not good, current increases above 10 uA already at 40 V. 

11:29

HV Detector #4 swithced off, Faraday cup removed, data stored again.

11:35

Check OK. Target 8.6E12

13:06

Check OK. Target 7.5E12

15:00

Temperature check - aida03 returns "No response". No response for rates either. Merger looks fine for 3? Likely just happened.

Attempted to STOP the DAQ. All FEEs still going (refused to stop) and 3 in error. Decided to just turn all FEEs off.

Restored DAQ with no issues. 3 is working. Temperatures fine. Rates fine. Unclear what had happened.

Run59 is EMPTY. New run = run 60.

16:00, Full check, everything working after the restart

17:35  Full check, everything is working smoothly!

18:35: Full check, status -Okay.  thickness 5.7E12,

19:35: full ckeck! Status - OKAY, target theckness ~5E12

20:50 : Full check! status - okay

22:00 : Full check, status -okay. Target thickness 6.4E12

22:55 : Full check, status -okay! ------ target thickness 5.8E12

01:00 : Full check, all okay! 

04:00 : Full check, all okay! 

07:00 : Full check, all okay!

10:00

Attempted to change cooling to improve the beam. Changed cooling by 0.5V. Significant improvement in the IPM, but Rutherford goes down. Target missed?

Changed orbit bumps. Increased Rutherford. IPM is back to how it was. Rates are possibly slightly higher? Will attempt more systematically in the afternoon, when we change beam energy again.

Attempted to delay target on by 1 second. No significant changes in beam lifetime or rates. Can try systematically again in the afternoon.

Restored settings to how they were. Started new run 61.

Temps OK

Rates OK

Merger OK

Online histos look OK (did not reset)

Checks

Clock status

FEE64 module aida16 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

ADC calibration

FEE64 module aida16 failed
Calibration test result: Passed 15, Failed 1

White Rabbit decoder

   
         Base         Current     Difference
aida14 fault      0xbf08 :      0xbf08 :      0  
aida14 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 16, Failed 0

NOTE: RunR55 is missing

Run56: Background run with 250 keV/u. Target off using the usual procedure.

All  checks ok!

We moved to nostorage mode and will change the run during the beam change

11:06

Run during beam change. We will go down to 200 keV/u

All the checks are ok.

No storage mode

Target off

We played with some of the slow comparator thresholds to judge the statistics and data rate. 

Deemed threshold at 100 keV too low for the current beam energy due to excessive rates observed on all FEEs. If required for lower energies we can go to this level. 

Data rate at 130 keV threshold is about 2 MB/sec compared to 8 MB/sec at 100 keV. Note beam energy is 400 keV so no significant benefit going to the lowest threshold. 

16:30

NOTE: Pulser is ON!

Check Leakage current: (see attachment 1)

Check NewMerger tab: (see attachment 2)

Check Fee temperatures: all green (see attachment 3)

Check statistics tab: ADC data items (see attachment 4)

                                 ADC coincidence-info #8 (see attachment 5)

                                 Good events (see attachment 6)

 Check Spectrum Browser tab: (see attachment 7)                 

Check the following plots: LowEnergyXYRateDSSD0 (see attachment 8)

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

                                         LowEnergyExTotal (see attachment 10)

                                         LowEnergyEyTotal (see attachment 11)

Check system wide checks tab: all okay.

All is good.

We start a new run with real data taking condititon: beam is ok, target density is 1e12 cm-2, detectors are moving in-out periodically.

Cooler voltage setpoint 1066.6 V. Read value 1068 V.

Current read = 0.01325 uA

Beam energy = 1.88 MeV/u (from HKR)

Attachment 1 - beamcalc with nominal values

Attachment 32 - beamcalc with actual values

Right arm at 19.80 mm

Left  arm IN at 77.56 mm (OUT 32.77 mm)

vertical beam position 0.0 mm

horizontal beam position  -3.0mm

Beam intensity ~1.8e+7 particles.

Target density ~1e12 at/cm2

Target temperatures: T1=31.4 K T2=35.1 K

17:48

aida01 fault      0x0 :      0x1 :      1  
FPGA Timestamp error counter test result: Passed 15, Failed 1

All checks OK.

The electron current from beamcalc is the setvalue.

18:51

aida01 fault      0x0 :      0x1 :      1  
FPGA Timestamp error counter test result: Passed 15, Failed 1

All checks OK.

20:20

aida01 fault      0x0 :      0x1 :      1  
FPGA Timestamp error counter test result: Passed 15, Failed 1

All checks OK.

22:02

         Base         Current         Difference
aida01 fault      0x0 :      0x1 :      1  
FPGA Timestamp error counter test result: Passed 15, Failed 1

All checks OK.

23:13

All checks OK. Target stabilized at ~6.7E11 at/cm^2

00:23

All checks good.

01:01

We are observing what seems to be a steady decrease of the target density, currently at around 5e11 at/cm2, whihc also seems to be following a decrease in the temperature T2.

1;40

All checks good. Spectra attached in att. 18. Target density still decreasing (3.7E11 at/vcm2)2

2:54

All checks ok. Attached rates and spectra in att. 19-20. target density decreased as continued, now around 2.5E11 at/cm^2.

We tried to move a bit around the nozzle/target, no effect.

Gone back to original value of x=1000, y=-4200.

Moreover, the screen with the program to move the detector seems to be bugged (gray background, a double window): however, the potentiometer value is chnaging and I see the usual effect of detector moving on the statistics, so i think it is fine

3:38

Checks Ok. Target 2.5E11 at/cm2

4:05

The bug disappeared

4:56

Target density is now 1.9E11 at/cm2. It is very low with respect to before, but our reasoning is that nothing can be done about it at night anyway (apart from moving the nozzle, which we tried without effect). O. pointed out that the beam lifetime is still such that the beam is gone before the next injection.

Checks ok. Attachments 21-22 spectra and xy plot. Some structure in top left?

06:00

All check ok. Target density 1.8E11 at/cm2

6:49

all goo, target 1.7E11 at/cm2

07:30

All checks ok, except timestamp (15 passed, 1 failed, see screenshot)

8:30

Tried moving the nozzle. Minor improvement in target density only. x=-1000 y=-4200

Stopped writing to disk while moving nozzle and stopped the motors. Restarted everything and moved to run 29 - no differences vs. 28.

09:11

Temps OK.

Nautilus does not respond to user commands. Attempint to make it work again without rebooting.

10:05

Stopped to reconsider which arm we are moving. Started to move arms around. Decided to stick with left arm.

Started run 30. Same as 28 and 29.

10:30

Temps OK - attach

Rates OK - attach

Stats OK (taken with arm in) - attach

Current histogram attached

11:40

Something happened to leakage currents. We see an increase in the top-right detector (which recovered) and a decrease in the bottom-right. Neither is moving.

No changes to left arm detectors, which are moving.

Temperature in the cave stable. No effects on the DAQ.

Temps OK

Rates in aida06 at 10k. Fast discriminator at 100k. Checkloaded. OK. Other rates fine.

Will keep monitoring.

11:50

aida06 fast discriminator over 100k. Checkloaded x2. OK now. Other rates fine.

No changes to leakage current.

13:20

Temps OK

Rates OK (aida 06 is no longer counting fast)

Leakage current stable

14:20

Run continues OK

Temps OK

Stats OK

15:30

Merger OK, Tape server OK

ADC data item statistics OK

FEE temperatures look OK

System checks OK

20:20

Merger OK, Tape server OK

ADC data item statistics OK

FEE temperatures look OK

System checks OK

At 19:10 the vacuum has increased suddenly everywhere in the ring, roughly a factor 1.5.

Merger, Tape server (WS3) checks: DONE, writing ~1 MB/s, refreshing is fine

Leakage current (WS4): 3.48 uA , stable current

DAQ (WS2): temperature checked,

     - clock status:
Clock status test result: Passed 5, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

     -  adc calibration:
Calibration test result: Passed 5, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   - white rabbit check:

         Base         Current     Difference
aida01 fault      0xc228 :      0xc272 :      74  
aida02 fault      0x5e98 :      0x5edf :      71  
aida03 fault      0x92d3 :      0x931d :      74  
aida04 fault      0x7a92 :      0x7adc :      74  
aida05 fault      0xedc0 :      0xedcc :      12  
White Rabbit error counter test result: Passed 0, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

DAQ (WS2): everything ok

Online monitoring (WS 5): done; first shots with something?

Merger, Tape server (WS3) checks: DONE, writing ~1 1273kB/s, refreshing is fine

Leakage current (WS4): 3.46 uA , stable current

DAQ (WS2): temperature checked,

     - clock status:

Clock status test result: Passed 5, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

     -  adc calibration:

Calibration test result: Passed 5, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   - white rabbit check:

         Base         Current     Difference
aida01 fault      0xc228 :      0xc272 :      74  
aida02 fault      0x5e98 :      0x5edf :      71  
aida03 fault      0x92d3 :      0x931d :      74  
aida04 fault      0x7a92 :      0x7adc :      74  
aida05 fault      0xedc0 :      0xedcc :      12  
White Rabbit error counter test result: Passed 0, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

DAQ (WS2): everything ok

Online monitoring (WS 5): done

At 19:10 Frank called us that vacuum in sections 08 and 10 got worse by about 1.5 times. There is a possible danger that vacuum valves can get closed if rest gas pressure is above the set treshhold.

Nikos warmed the cryostat by 0.5 degrees from 129.0 K to 129.5 K and the pressure became better than before. We cooled the system in steps of 0.1 degress down to 129.1 K and again observed the pressure rise.

We think we can stay with the present conditions. However, the pressure shall be monitored not to hit the treshhold.

Recently the pressure started to raise a bit, we have increased the temperature from 129.1 K to 129.2 K. Current pressure at S1 2.10e-7.

Merger, Tape server (WS3) checks: DONE, writing ~1 MB/s, refreshing is fine

Leakage current (WS4): 3.46 uA , stable current

DAQ (WS2): temperature checked, clock status:ok, adc calibration:ok, white rabbit check:ok

DAQ (WS2): everything ok

Online monitoring (WS 5): done

Pressure seems as before 19:10.

Merger, Tape server (WS3) checks: DONE, writing ~1 MB/s, refreshing is fine

Leakage current (WS4): 3.47 uA , stable current

DAQ (WS2): temperature checked, clock status:ok, adc calibration:ok, white rabbit check:ok

DAQ (WS2): everything ok

Online monitoring (WS 5): done