The startup procedure is the same as the regular startup as described in elog https://elog.ph.ed.ac.uk/CARME/666 with some important differences

x2go opens up the computer on the GSI network carme@atppc025

In order to access the carme computer and the CRYRING network we need to ssh. 

To access the carme-gsi computer use:

    ssh -Y npg@carme-gsi 

First the proxy on carme-gsi must be set up. On carme-gsi

    ssh -L 8080:proxy.gsi.de:8080 carme@atppc025   

Once the proxy is setup the firefox windows (i.e Relay system, localhost:8015 etc) can be opened on carme-gsi by using 

    firefox &

Use a single firefox window for all tabs (Relay, Run control, Tape server etc)

The startup via x2go does not allow the clicking of icons for the AIDA servers. To start these servers must instead be done by the command line. To start these servers from carme-gsi is done by

    ./MIDAS/Linux/startup/HTTPD

    ./MIDAS/Linux/startup/HTTPD@8115

    ./MIDAS/Linux/startup/NewMerger

    ./MIDAS/Linux/startup/TapeServer

Follow the procedure outlined in https://elog.ph.ed.ac.uk/CARME/666 with the extra step of the ssh'ing into carme-gsi first

12:00

Background run started via x2go - Run 6

We need to run for a while to gather sufficient stats on aida05, aida06, aida07 and aida08 channels to see which capacitance is best for the new adaptor cards.

All checks OK - aida16 ADC calibration failed not too bad. 

Leakage currents look the same as before.

DSSD1 115 V and 17.4 uA

DSSD2 100 V and 5.86 uA

DSSD3 100 V and 16.3 uA

DSSD4 100 V and 7.10 uA

14:00

All checks OK

17:00

Daq had stopped writing data to disk. Managed to recover system without much interuption.

Noticed very high rates in aida09 and aida10 (>200k) -> this was also seen in previous beamtime.

Increased bias on DSSD3 to 105 V as done during D+D and rates dropped to 100 Hz again. 

All checks OK.

New run R7 started. 

21:00

All checks OK. R7 continues

17/01/26 11:00

All checks OK R7 continues

14:00

All checks OK. R7 continues

23:00

All checks OK. R7 continues

18/01/26 08:00

All checks OK. R7 continues

12:00

All checks OK R7 continues

18:00

All checks OK R7 continues

22:00

All checks OK. R7 continues. 

19/01/26 08:20

All checks OK. R7 continues

12:00

All checks OK R7 continues

18:00

All checks OK. R7 continues

22:00

All checks OK. R7 continues

20/01/26 All checks OK. R7 continues

22:30

All checks OK R7 continues

21/01/26 All checks OK. R7 continues

13:00

All checks OK R7 continues

21:00

All checks OK R7 continues

22/01/26 All checks OK R7 continues

23/01/26 All checks OK. R7 continues

13:00 All checks OK. R7 continues

24/01/26 09:00

R7 stopped. All tabs running through x2go closed. 

R3 pulser walkthrough completed.

Pulser amplitudes from 90,000 to 10,000 in steps of 10,000

Pulser peaks widths taken again. Some channels show big improvement. Note leakage currents are higher today. See previos elog

Gleb is/was in the ring yesterday. Perhaps this has had some effect?

Turning CARME off to make electronics changes.

13:00

New adaptor cards have been installed on FEEs aida05 to aida08

Attachments 2-5 shows each new adaptor card mounted on CARME (attached in order of the FEEs i.e aida05,aida06,aida07,aida08).

Attachment 6 shows all adaptor cards mounted on CARME. 

Note aida05 has the new adaptor card design but with unchanged capacitor values (33 nF).

Aida06-08 have the new adaptor card design with many different capacitor values ranging from 47 pF up to 820 pF. The dot on the adaptor cards correspond to the end of the board which contains the capacitors with a value of 47 pF. 

Each board has 4 channels corresponding to each capacitor value. Attachment 7 shows how these are arranged on the board. 

16:00

R5 pulser walkthrough with new adaptor cards.

No effect on pulser widths (they look better!) - most likely other effects such as Gleb working in the ring

System shut down. Will restart the DAQ from Edinburgh via x2go

13/01/26

Arrived at GSI.

Biased DSSDs. Noticed higher leakage currents than previous experiment circa 19uA for DSSD#1 and #3 and 9uA for DSSD#2 and #4

Checked inside the cave. Edinburgh vacuum gauge was ON. Vacuum group gauge OFF. Turning off the Edinburgh gauge lowered leakage currents to those from D+D experiment. See attachment 1.

Noticed the water interlock was tripped reading 0 l/s. Water was ON. Interlock was not properly connected to the pipe anymore. We pushed it back onto the pipe and taped it on hard - hopefully it holds for the time being

Attempted to re-start the system, encountered a pop up error message when trying to power the FEEs.

Required reboot of both the carme-gsi computer and nnrpi1, likely as we had been accessing remotely via x2go as well.

AIDA fully started up again.

All DSSDs biased to 100V, except DSSD1 which requires 115V otherwise we get >100k on the rates. Leakage currents are 16.25uA, 5.4uA, 15.17 uA and 6.56uA. Current logging is ON

Data being saved in new file Jan26

All slow comparator thresholds set to 0x64 (1 MeV)

All system checks OK.

ADC data items rate on all FEEs is 100-200 Hz except aida15 which is 13k Hz. This is the same as in D+D. See attachment 2

R1 is just the startup file (empty). R2 is currently running as a background run. Data rate to disk is 100 kB/sec. 

We have looked at the pulser widths of all FEEs for 1.8L (layout 3 and 4). Note the pulser amplitude is 90,000.

See attachments 3 and 4 for pulser peaks

See attachment 5 for the rates across each FEE

See attachments 6,7 for the waveforms.

Pulser widths with original config:

Tomorrow we will swap out the adaptor cards for DSSD#2 (aida 5,6,7,8). The pulser widths look the best for these so it will make for the best test and also this DSSD has low leakage currents and no other obvious issues (i.e missing strips, fluctuating leakage current, curses)

Background run with unchanged configuration to be left overnight. R2

21:00

R2 continues. All checks OK.

Note the merger appears to no longer display properly. Appears to have frozen - see attachment 8

Daq is still running however, and data is being saved to disk still. Merger must therefore still be working and it is just the gui that is failing.

Attachment 9 shows energy spectra so far.

09:30

R2 continues. All checks OK

Leakage currents slightly higher than yesterday, periodic cycle probably. Attachment 10

Attachments 11 and 12 show the energy histogram and xy plot from the overnight run. Since all xy quadrants on each dssd are working all FEEs must be functional and merging even if merger gui isn't displaying correctly.

ML flicked switch on CAEN bias module to Enable, and restored remote control.

Attempted to bias the DSSDs. Leakage currents significantly higher than before, but apparently stable. See attach 1.

Likely a vacuum gauge turned on? Will have to investigate.

Bias off for safety for today. Will try again tomorrow.

26 August

14:40

Turned CARME on 

Everything started up okay, all detectors on and stable. 

Note aida16 clock and ADC are uncalibrated. Note dssd4 is on MEC

Detector 1 noisy. 

AIDA started - all okay. 

All FEEs set to LEC (0x0) and slow comparator threshold set to 300keV (0x1e). 

Currently writing to data to G22-205/R100, doing background run, ~400 kB/s. 

16:40

Checks OK. 

19:00

Checks OK

22:30

Temps OK, Rates OK

x2go window size weird on different monitor as well.

27th August

Stopped Run 100. We had accidently set the threshold to 140 keV (0xe) not 300 keV (0x1e) as intended. 

Either way we require data on the alphas for electronics testing in October and the data rate at this threshold is too high. Threshold set to 0x64 (1 MeV) to look only at the alphas. 

Run 101 started. See attached leakage currents

20:30

Checks OK

28th August

10:00

Checks OK

CARME daq crashed this afternoon. Aida14 had a disconnection error. Seemed to coincide with dssd4 tripping again. 

Rebooting FEEs. On reboot there are always some FEEs which seem to have some error. The FEE changes each time. i.e aida12 one time, aida09 next time. I will give them a little while to compose themselves before trying again. 

Unable to get the system writing data to disk again. I think the system is unhappy with the merger / tape server not being closed properly. Will have a proper look tomorrow. 

29th August

20:15

Running again R102. 0x64 slow comparator thresholds. DSSD4 off due to leakage current tripping -> MEC

30th August

Daq continues R102

31st August

Daq continues R102

11:30

R102 stopped. CARME shutdown. This week, potentially tomorrow morning, the Kasell group will dismount the VUV spectrometer. CARME off and bias off during these works. 

08:07 CEST

Temperatures ok - attached

DSSDs on. Leakage currents still high. - attached

If ML & co cannot find the reason, will turn off FEEs later today

FEEs turned off, leakage current still high (DSSD off now). 

Motors out. 

---9 July---

11:00

Turned detectors on - all stable see attachment 1 

AIDA started - all okay. See attachment 2 for FEE temps.  

All FEEs set to LEC (0x0) and slow comparator threshold set to 300keV (0x1e). See attachment 3. 

After some issues tape server and merger all set up, NetVar not working??? have tried to reboot things but still getting issues. See attachment 4 for tape service and attachment 5 for merger. 

Currently writing to data to G22-205/R98 (R97 empty), doing background run.

13:30

All checks okay apart from white rabbit, see attachment 6.

Attachment 7 - stats 

18:30 CEST

More issues with anydesk - now refuses to connect with error message desk_rt_ipc

May be as simple as the monitor being turned off. Will have to be checked in person

Connected to atppc025 via x2go and from there checked

- Temperatures are fine (attached)

- Detector bias / leakage current is fine (attached)

- Data rate from file size being written is around 500 kB/s (fine)

- A number of interesting features in the online background spectrum. Likely all non-physical (attached)

- Stats OK (attached)

20:50 CEST

All checks good.

22:28 CEST

All checks good.

Features in the XY plot rensemble beam hitting the detectors. Attached.

Leakage currents and rates are good - unlikely to be major detector damage. Attached.

Only beam in the ring between now and end of D+D was stable uranium.

Killed CARMEsort via terminal (still no access to anydesk).

Restarted CARMEsort. No obvious structures appear after a few minutes. Was there beam in the ring today?

23:09 CEST

All checks good

Features in the XY plot and Ey plots disappeared.

---10 July---

03:22 CEST

All checks good.

Features gone. Attached.

05:15 CEST

All checks good.

07:34 CEST

All checks good.

10:40 CEST

All checks good. 

12:35 CEST

All checks good. Some features in plots again appeared in last 2 hours, attached Ey plot (attachment 15). 

14:40 CEST

All checks okay. 

16:40 CEST

All checks okay. 

18:40 CEST

All checks okay. Attachment 16 leakage current stable. 

Attachment 17 Ey plot.

19:35 CEST

All checks OK.

Reset the Root spectra again - highly unlikely to have beam overnight without experiments running.

23:39 CEST

All checks OK.

Still seeing peak at 6 MeV like attachment 17, after resetting the Root spectra. Detector damage? Still seems implausible.

Events are focussed as in attachment 12.

Reset again. No obvious peak or pattern after a few minutes running.

From Claude K

"CRYRING was running with the Uranium beam from ESR until Tuesday (8 July) night around 22:00.

From Wednesday noon we started to inject D+ from the local source and had the D+ beam stored at various energies from 14:30 until 17:40. During the night from Wednesday (9 July) to Thursday (10 July) there was no beam.

We then continued measurements with stored D+ On Thursday from ca. 11:00 to 21:00. Since then CRYRING is in shutdown."

---11 July---

08:07 CEST

All checks OK.

Peak at 6 MeV did not re-appear overnight. Attached x2.

12:00 CEST

All checks good! All plots look good!

16:00 CEST 

All checks okay! 

19:20 CEST

All checks OK.

Background peaks almost visible.

22:53 CEST

All checks OK.

---12 July---

07:25 CEST

All checks OK.

10:00 CEST

All checks OK. Temps attached.

11:00 CEST

All checks OK.

19:00 CEST

All checks OK.

23:00 CEST

All checks OK.

---13 July---

07:42 CEST

All checks OK.

11:00 CEST

All checks OK.

ELOG down during the afternoon

--14 July--

11:00 CEST

All checks ok. 

14:40 CEST

Tape service and merger now working on atppc025

All checks ok, briefly went no storage at around 14:38 CEST for 1-2 minutes whilst getting tape service and merger up. 

17:40 CEST

All checks ok. 

21:30 CEST
All checks OK.

23:16 CEST

All checks OK. Several screenshots attached.

Calibration appears to hinders clear alpha peaks.

---15 July---

07:47 CEST

All checks OK.

11:55 CEST

All checks ok. 

14:35 CEST

Issue with atppc025, see attachment 29 for error message (desktop 2, detectors etc)

Got error to go, all checks ok. 

17:35 CEST

Same issue with atppc025, otherwise all checks ok. 

20:30 CEST
All checks OK.

23:07 CEST

All checks OK

---16 July---

07:41  CEST

All checks OK

12:00 CEST

All check ok. 

15:15 CEST

Some fees have high rates, all other checks ok. 

15:32 CEST

The above behaviour appears to be correlated with an increase in leakage current across all four detectors. Most likely light?

The background is not reliable -> No storage.

Unfortunately the leakage current logger crashed a few days ago on the server, and without anydesk we did not realise. Restarted it on atppc025.

Informed ML about issue. If it cannot be solved we could increase the thresholds and keep going, but the background may not be representative.

16:35 CEST

Issue persists. ML and ZA are not aware of any changes, except power-saving mode enabled for cave AC. No effect on FEE temperatures - not the issue.

aida04 rates shows 0. Checkloaded. Fine for a few seconds. Back to 0. Data event rate 100,000 may be too high.

Increased thresholds to 1 MeV. Data rates lowered, but e.g. on aida04 still 40,000. Goes back to zero. Checkloaded. OK for a short while, then back to zero.

Tape server stopped. DAQ stopped - all FEEs respond, good! Merger still going.

DSSDs off. Will retry tomorrow morning.

23:44 CEST

Temps OK.

Updated anydesk to 6.3.2 CentOS7 release.

Works now.

15:20

DAQ stopped

Merger & Tape Server stopped

FEEs off

All detectors off

Leakage current plot attached

Leakage current file moved to /TapeData/

Will restart for a long background when the next experiment is over.

00.31 DAQ continues data file G22-205/R96_27
      
      DSSSD bias & leakage currents - attachments 7-8
      FEE64 temps OK - attachment 6
      ADC data item stats - attachment 5
      per FEE64 Rate spectra - attachment 4
      Merger & TapeServer - attachments 2-3

      Online analysis - attachment 1

      Analysis data file R96_29 - attachment 9
       no timewarps, all FEE64 deadtimes <<1%

04.04 FEE64 temps OK - attachment 10
      ADC data item stats - attachment 11
      DSSSD bias & leakage currents - attachments 12 & 15
      per FEE64 Rate spectra - attachment 13
      TapeServer - attachment 14

08.26 DAQ continues data file G22-205/R96_64

      DSSSD bias & leakage currents - attachments 16-17
      FEE64 temps OK - attachment 18
      ADC data item stats - attachment 19
      per FEE64 Rate spectra - attachment 20
      TapeServer - attachment 21

11:03 All checks OK
EyTotal histogram attached.

12:25 Temps OK. Rates OK.

15:03 run stopped - beam time over
all checks ok
EyTotal hist attached

20:00 - Go to YR cave to see what is going on

20:20 - Beam back

20:28 - Set Tape to Storage, rate ca 2500 kB/s
Temperature and rate OK, System wide checks OK

Target density low (4e12/cm2)

21:00 - target density still 4e12, Ey rate histograms attached (attach 4)

21:20 - Target density increased to 6e12, we see more indication of reactions

22:30 - All checks OK, tape rate still 2500 kB/s
Target density ~7e12/cm2, beam intensity ~1e9pps

23:00 - Target density 7.5e12/cm2, beam intensity: 1e9 pps. Particle detector rate: 8 kHz at injection
 Gas bottle pressure still above first tick 
 Target position X=-1000, Y=1400
 All running smoothly. CARME DAQ running OK. (Figs 5-7)

Run 94

Background with beam at 150 keV and no target.

Note energy is 0.15% off vs. beam and target to have a more realistic orbit.

As prior run but thresholds changed to

DSSD1 - 0xc

DSSD2 - 0x8

DSSD3 - 0x8

DSSD4 - MEC

Run 95

DSSD 4 worked briefly. Tripped again a few minutes into this run. Ignore

Run 96 - as run 94

run stopped at 14:00

Zoran reset to with-target settings for later

The plotted readout for the G35 voltage divider over several injections for this run is attached. This kind of data can be obtained from Konstantin also after the experiment.

The average is something like 88.4808 V.

Run 93- New run at 150keV, background with beam

13:15 All checks ok

Run 92- started at 09:00, background with beam

Same settings as run 91

See attachments for rates etc

All checks okay

09:58 Run stopped as now tuning 150keV beam

10:00 All checks ok

10:25 D2 bottle pack pressure 30 bar approximately

12:18 All checks ok

23:00 DAQ restored and data now being written okay. Checks all okay, see screenshots below. Histograms reset. 40 bars in bottle @ 23:45 - should last 30 hours.

12:00 Checks all ok. No problems with aidas or writing to disk.

01:10 Checks all ok.

02:05 Checks all ok. See attatchment 6 for peak update after reset histograms 3 hours prior. Target okay, dropped a little to 6e12 but seems to be going up itself. Will monitor and then move position if needed.

03:10 During checks, noticed detector 4 had tripped so turned it off. Writing to disk increased alot, statistics increased to alost 250000. Put aida13-16 in low/medium energy. Started new run. (see attatchment 7-14 for before and afters)

RUN 91

03:15 Checks all ok after turning off detector 4. Moved target around and got abit more density but x is very sensitive and y did not seem to do much. Density 5.5e12.

04:30 Checks all ok. Target decreasing in density - monitoring it.

04:35 Target density back up to 1e13 after moving in y.

05:30 Checks all ok. Target density decreased slightly to 8e12

06:30 Checks all ok, target density stable for past hour

07:30 Checks all ok

08:30 Checks all ok, target density at 1e13 for past two hours

09:00 Run stopped

Found all Merger Links background black - inactive. Nontheless the tapeservice was working, the data rate to disk was normal, and the file was being written and the histograms were being updated.

Decided to stop DAQ, stop Merger, restart merger. Links rest to green background - but no more writing to disk!

Killed Merger, killed Tape Service. Restarted.

Netvar read out at 3 - set to 1. When the Output of the Merger is enabled Netvar returns to 3. Tapeservice not writing to disk

Killed Merger, killed Tape Service. Restarted x3. Same behaviour.

Not working. No blocks being written.

With Tape Server status Stopped Ticked No storage, Unticked No storage -> Now writing blocks ?!

Tape Server Go. Writing to disk. Merger OK. Histograms updated, observables as normal (see next ELOG).

Lost runs 84-89 in the process.

12:15 restarted measurement with target - Run 83

same settings as for runs 76, 77 etc

ROOT histograms NOT zeroed

Beam at 8e8, target density at 1.5e13, scintillator @ 0 deg. peak rate 5000

status screenshots attached

15:15: Checks all ok, peaks are peaky (attatchment 8)

16:45 Checks all ok.

17:45 Checks all ok. IMP Froze and has been restarted. Peaks good. Light turned off iin cave.

18:40: Checks all ok. Target great at 1e13.

22:20 Checks done - all okay except aida green-to-red was not working, but still writing data to disk. Carlo trying to figure it out now. All other checks were okay. Daedalus crashed, restarted and target still at 1e13. 

Background without beam while source is optimised.