Saturday 12 Sep 08:43

Remotely checked status of baking. See attached.

Baking appears to be proceeding OK. No thermocouples disconnected.

Speed is slower than previously observed, limited by thermal gradients observed across both octagon and CARME
vessel as a result of new thermocouple placement.
Octagon gradient may not be real, and could be a result of placement of Octagon side thermocouple?

Main concern is temperature recorded by CJC in Picolog2 (aka Room temperature 2) at 31 C, increasing. 
The second picolog may be too close to the tent. If the temperature keeps increasing it may stop working. Will
keep monitoring.

Sun 13 Sep 08:06

Baking reached 93 C yesterday afternoon at 16:18 and paused as a result of a percieved temperature gradient in
the octagon. 
See attached.

Left system overnight to check whether octagon gradient was going to equilibrate via convection. 
No changes this morning.
Thermocouple 1-8 labeled "Octagon (side)" likely is not in thermal contact with the octagon. 
It appears to have a temperature lower than room temperature. Possibly sits outside the tent? Gradient will not
decrease with time.

Temporarily overrode automatic mode, set temperature to 100 C, restarted automatic mode. Will keep monitoring.


Sun 13 Sep 10:00

Vessel reached 100 C, no gradients. All OK, except octagon gradient (ignoring).
Manually increased temperature to 110 C. Restored automatic mode.

Sun 13 Sep 12:00

Baking stopped automatically at 10:30 due to CJC temperature (aka room temperature 2) in TC-08 Picolog 2
exceeding 40 C.
Max operating temperature of picolog TC08 is 50 C, and accuracy is lost above 30 C.
https://www.picotech.com/data-logger/tc-08/usb-tc-08-specifications

Stopped baking. Removed thermocouple 1-8 "Octagon (side)" from Octagon gradient calculation and recompiled.
Restarted manually at T = 80 C

Please move picolog away from the tent tomorrow, and we can then increase temperature. Tent does not need to be
opened.

Sun 13 Sep 19:00
Vessel temperature at 80 C stable. 
Other temperatures stable. 
Leaving at 80 C overnight.

16 September

08:50
Baking continues OK. Temperatures stable, did not increase with respect to yesterday evening.
See attached.

15:00
Baking continues OK.
See attached.

20:22
Baking continues OK.
See attached.

17 September

08:20
Baking continues OK. Temperatures slightly decreasing?
See attached.

(NOTE: Raspberry Pi internal clock appears to be on GMT, rather than GMT+1 due to daylight saving.)

Baking started on Mon 28 Sep. Due to WiFi connection authorisation expiring, no remote access was possible until
Tue 28 afternoon.

JM found remotely that temperature had increase up to 80 C, and baking had then been shut off due to negative
gradients. Unclear what happened. System was already cooling.

Restarted baking in automatic mode. Baking shut down automatically after few hours. It is likely the automatic
mode was being confused by vessel cooling and being heated at the same time. 

29 SEP

20:30 Switched to manual mode (60 C). Left overnight

---

30 SEPTEMBER

08:10 Temperatures seem stable, but slight temperature gradient across vessel would prevent automatic mode from
working. See attached.
Unclear why RE72-1 let temperature reach 64 C when setpoint is 60 C. May be affected by RE72-2. Not worrying,
but may confuse automatic mode.
Ramped up to 70 C manually.

11:10 Temperatures stable. Ramped to 80 C manually. Gradient in CARME vessel unchanged at ~11 C. See attached.

14:18 Found baking had stopped shortly after last check. One floating thermocouple reported temperatures above 300 C. 
Unclear what happened. Possible fault? See attached. 
Restarted baking manually at 40 C. Waiting for temperature to stablise. Vessel still cooling - cannot activate
automatic mode.

15:00 Temperatures OK. Ramped manually to 50C

18:10 Temperatures stable. Ramped manually to 60 C. Left automatic mode on

20:20 Found temperature setpoint at 66 C. See attached.
Huge spikes in thermocouple 1-4 Floating (side-low). Same thermocouple that reported temperature spikes earlier. May be
faulty or touching heater directly. Seems to happen for T=65 C and above.
Set to manual mode. Temperature setpoint 60 C. Left overnight.
Will ask local people to remove thermocouple 4 from picologger 1 (aka #A0067/395, port 4) to continue.

---

1 OCTOBER

08:00 Temperatures stable. Interesting behaviour of problematic thermocouple 1-4. Trend looks real. Seems to confirm
hypothesis it is touching a heater.
Waiting for DL locals to disconnect it.

09:49 Thermocouple 1-4 disconnected. Increased temperature manually to 75 C / 70 C. 
It appears RE72-1 consistently reports 5C above its set point. It controls 3 heaters vs. 8 heaters controlled by RE72-2. 
This is most likely due to RE72-2 needing greater power output to reach its setpoint, increasing RE72-1 thermocouple
temperature as a result. 
RE72-2 may be doing very little at the moment.
Hopefully placing its setpoint 5 C above will force both heaters to work together. This may decrease temperature
gradients observed in the vessel.

16:20 Increased temperature manually in steps up to 95 / 80 C.
The uneven temperature setpoints associated with the two RE72 appears to have reduced the temperature gradient in the vessel.
Will ramp up to 100 later this evening and leave overnight.
See attached.

22:00 Temperature stable. Rates are showing unphysical rates (too high / too low) and occasionally going all to zero at the
same time. 
Looks like a previously observed bug that I thought had been sorted. Will have to debug again. Does not affect manual mode.
Ramped temperature to 105 / 95 C. Leaving overnight.
See attached.

---

2 OCTOBER

07:58 Temperatures stable. See attached. Slight gradient in CARME vessel (~11 C). Not worrying.
Ramped manually to 115 / 105 C

12:16 Temperatures stable. See attached. Valve temperature at 46 C.
Ramped manually to 125 / 115 C

17:34 Temperatures stable. See attached. We seem to be getting close to the upper limit of what the tent can achieve.
Ramped manually to 135 / 125 C
Will leave overnight.

---

3 OCTOBER

08:32 We have been baking at full power (heaters always on). Temperatures stable. 
Appears to be as high as we can go with this tent configuration.
Will leave in manual mode and monitor throughout the day.
See attached.

17:00 Temperatures stable. No changes from this morning.

21:00 Temperatures stable. See attached. Baking continues overnight.

---

4 OCTOBER

07:30 Found temperatures decreased overnight by around 10 C. See attached. 
We are still baking at full power. Room temperature has not changed.
Most likely explanation is part of the tent giving / moving due to the baking and a draft being developed.
Will keep monitoring during the day.

13:50 Temperatures stable, no changes from last check.

20:22 Temperatures stable. See attached. Leaving overnight.

Thu 22 October

22:15 Temperatures stables, slowly decreasing.
Bump - rising front corresponds to heaters turned on. Falling front to D2000 turned off?
See attached.

Fri 23 October
09:57 Temperatures stable. See attached.
13:09 Temperatures remain stable. No significant changes.

Sat 24 October
21:27 Temperatures stable. See attached.

Sun 25 October
*Daylight saving is in effect. Time on the Raspberry Pi is no longer incorrect*
07:57 Temperatures stable. See attached.
19:05 Temperatues mostly stable. See attached. Will leave overnight and move setpoint to 80 C early tomorrow morning

Mon 26 October
07:37 Temperatures mostly stable. See attached. Setpoint to 80 C.

Accessed Pi remotely 

( 1. Without the Internet, the Pi has no way of knowing what time is it.
In case of power loss make sure to set correct date via

sudo date -s 'YYYY-MM-DD HH:MM:SS'

Or all log files will have wrong timestamp )

EDIT: Pi now connected to GSI time server.


2. Temperature logging OK


3. Commands/events logging enabled. Logging to file Log.dat, automatically moved to Logs/$(timestamp)_log.dat on
gracefuly quit.
Cannot test due to TC08 / RE72 not being connected. Please test when possible.


4. Plot_temps.gp gnuplot script currently optimised for 1080p monitor. Monitor in use is 1024x768. 
Please optimise first line of script (size, position) to make it fit while we search for a better monitor.


5. Hostname changed to CARMEbakingpi

CARME pressure moved to

https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?e=hEoZt7


Some photos overnight are over-exposed and pressure cannot be read. 
Might be able to fix with image processing if needed, but likely not worth the time. 

Baking started: 23/09/2021 14:00	
Photo frequency changed to once per 10 minutes.

1. Connect to pi on CRYRING local network 
ssh pi@192.168.207.127

Password available on request


2. To check state of baking code

screen -x

please do not press any keyboard button if unsure how the code works. 
Pressing q instantly stops baking in case of emergencies

when done, to disconnect

CTRL+A 
d


3. To check log of commands / previous status of baking code

cd Programs/Baking
cat Log.dat


4. To check temperature plot

cd Programs/Baking
gnuplot Plot_temps.gp 

CTRL+C on terminal to quit

Please remember the Pi has limited computing power and the gnuplot window will consume some of that when open
One remote user is no problem, but multiple may give issues to the Pi


5. Check current pressure

Currently:
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?e=hEoZt7


(Pictures from webcam are in ~/Webcam on the pi
CM52 data also to be available online in real time soon)

Current temperature ~64 C

See plots attached

Current pressure ~ 2.5E-6 mbar

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug

and
http://web-docs.gsi.de/~lestinsk/vacplot.php

Current temperature ~90 C
However note setpoint is at 102 C. It appears we are baking at full power, but too many heat losses?
Temperature gradient still increasing. May just be a matter of time.

See plots attached


Current pressure ~ 1.9E-6 mbar decreasing

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug

and
http://web-docs.gsi.de/~lestinsk/vacplot.php

Baking automatically stopped due to negative gradient in on thermocouple for 10 minutes straight. 
Not surprising giving we were baking at full power bu making little progress. We should have switched to manual
mode.

Switched to manual mode for now at 60 C to recover temperature without excessive gradients. 
Will restart automatic mode later today. Should be back to full power tomorrow morning.

EDIT
21:42 CET - Temperature stabilised. Restarted automatic mode.

Current temperature ~87C

See plot and screen attached.
Note Baking code restarted yesterday evening.

Current pressure 2.1E-7 increasing

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Attached.

Current temperature ~87 C, increasing
See plot and screen attached.
Thermocouple on CARME foot side (1-4) may have come loose judging from sudden jump in temperature at 14:30?
Improvement to blanket seem to have significant effect.

Current pressure ~3.5E-8 increasing
See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Current temperature ~132 C (on manual mode RE72-1: 132 C, RE72-2: 130 C)
See plot and screen attached.

Current pressure 2.03E-7
See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Attached

Dry run complete.

YR09 section valves closed. Section isolated. Manual valve between electron target and CARME remain open.

There appear to be issues with Shahab's caenlogging program. It crashes occasionally, with an error indicating the number of columns in the terminal is incorrect. We will have to ask Shahab.

For now

1. Start the caen_screen.sh script on nnrp1 (aka CARMEMotorsPi -  why was it renamed? It should stay as CARMEMotorsPi to avoid duplicate names).

This starts connect to the caen power module via TTY inside a screen called caenhv. Currently on desktop 1

2. On nnrpi1, start the caenlogging program --settings caenlogger_settings_gsi.toml

3. On carme-gsi, in ~/Programs/caenlogger start the gnuplot script

If the caenlogger script crashes, restart it on the second terminal

More investigations required on the crash.

(ELOG does not allow more pictures to be added to previous entry)

0. Give split-funnels to OG or Grossmontage to clean. Mount back foot support for gate valves. Critical when we split.

1. Wait for vacuum division to come to install: IE514 cables to current CM52 module, ion pump cables + controls + turn on, RGA head + program + turn on

2. Vacuum division should also bring N2 gas to vent CARME. Ideally vent by Wed to avoid losing time. Remember to disable ion pumps and remove all cables when venting.

3. While waiting to vent, connect pneumatic hoses to motors box in CARME rack. There is one pneumatic in (central feedthrough) and four pneumatic out. See picture attached. Top two go to left pneumatic motor, bottom two go to right pneumatic motor. Of each set of two, one is for extending the motor and one is for retracting but unclear which is which. Try to connect and operate motor manually by pressing on buttons on top of pressure reducer to ensure everything works. Don't forget to remove the safety pin.

4. Some disassembly of the servo motors may be required to get them operational. Use the Windows ClearView program provided by Teknic to control servos and change the function of the limit switches. Download from https://teknic.com/downloads/ -> ClearPath -> Software -> ClearView Install.zip . There is a Linux version but I have never tested it. The code on the raspberry Pi cannot reproduce some of the functions in the official program due to library limitations (but can do other things the official program cannot do).

5. After venting CARME, split open at the COF600 flange close to the magnet. Open CARME by 1-2 cm, remove spherical washers from top of rail studs, and lift with crane. Make sure knife edges of the two halves do not touch. CARME currently has feet mounted and cannot easily be placed on a standard pallet. Either dismount the feet (not easy to remount), or find a pallet on which CARME can land on its belly only, or place CARME on pallet with feet and screw feet down? Discuss strategy with DR asap.

6. Install split-funnels on flanges/wire harnesses in use (no point installing on other flanges). This may require dismounting top flanges. Probably switch from 3 hole to 4 hole configuration with funnels.

7. Test detector support going to actuator rod without detector. Make sure you have all the parts, including bolt to secure support to rod. You may want to push the rod in via pneumatic motor (or by hand pushing the plate outside) to gain better access.

8. Take detector support on table, install *two* detectors on same support, and mount on actuator rod in XHV. Connect wires and close via jacking plates. Optionally, consider mouting empty support on other rod? May be useful for diagnostic/scraping?

9. Move XHV thermocouple feedthrough to top flange and connect one thermocouple per detector, one to the support, and leave one floating inside

10. If the motors are fully operational, check how far/close they can be actuated from chamber wall. If not, take some dimensions for future reference.

11. Lift CARME back in place. Close using Pfeiffer COF600 seal and ~72 Nm force per bolt. 80 Nm were required in DL. Probably get Grossmontage to do this. Use our long torque-wrench.

12. Connect to interaction chamber via bellows. Beware gaskets on one side are non standard. One appropriate gasket should be on top of bellows box.

13. Reconnect pumps, rebuild octagon, reconnect thermocouples, rebuild baking tent, bake. Make sure to label the NEG cables this time!

14. Activate NEG pumps following this recipe. ALWAYS monitor temperature and be ready to stop if detector temperature reaches 120 C. NEG temperature does not drop immediately, don't wait until it is too late.

If temperature is increasing too quickly (check rates/h), decrease activation power in increments of 25 W. Remember to flash ion pumps.

A. Start from D2000. Activate one at a time with T 400 C, t rise 30 min, t hold 90 min. Activating a second D2000 while the first has reached its hold state seems safe. Leave at 250 C to keep clean.

B. Continue with UVH1400 furthest from the detector (DSSD wall chamber). Activate at P 200 W, t rise 30 min, to hold 90 min. Finish one set of two before moving to the second. Leave at 50 W to keep clean.

C. Continue with UHV1400 in intermediate chamber. Activate at P 150 W, rest as above. Leave at 50 W to keep clean.

D. Continue with 2xUHV1400 closest to detectors. Activate at P 70 W, rest as above. Leave at 50 W to keep clean.

E. Finish with 3xUHV1400 closest to detectors. Activate at P 90 W, rest as above. Turn all NEGs off when activation of 3x is complete.

F. Let chamber cool. Activate ion pumps at room temperature. Section off turbo pump.