GSI vacuum division IE514 gauge connected.

Gleb reports gate downstream opened. Vacuum goes up and recovers as expected.

Saturday 12 Sep 08:43

Remotely checked status of baking. See attached.

Baking appears to be proceeding OK. No thermocouples disconnected.

Speed is slower than previously observed, limited by thermal gradients observed across both octagon and CARME
vessel as a result of new thermocouple placement.
Octagon gradient may not be real, and could be a result of placement of Octagon side thermocouple?

Main concern is temperature recorded by CJC in Picolog2 (aka Room temperature 2) at 31 C, increasing. 
The second picolog may be too close to the tent. If the temperature keeps increasing it may stop working. Will
keep monitoring.

Sun 13 Sep 08:06

Baking reached 93 C yesterday afternoon at 16:18 and paused as a result of a percieved temperature gradient in
the octagon. 
See attached.

Left system overnight to check whether octagon gradient was going to equilibrate via convection. 
No changes this morning.
Thermocouple 1-8 labeled "Octagon (side)" likely is not in thermal contact with the octagon. 
It appears to have a temperature lower than room temperature. Possibly sits outside the tent? Gradient will not
decrease with time.

Temporarily overrode automatic mode, set temperature to 100 C, restarted automatic mode. Will keep monitoring.


Sun 13 Sep 10:00

Vessel reached 100 C, no gradients. All OK, except octagon gradient (ignoring).
Manually increased temperature to 110 C. Restored automatic mode.

Sun 13 Sep 12:00

Baking stopped automatically at 10:30 due to CJC temperature (aka room temperature 2) in TC-08 Picolog 2
exceeding 40 C.
Max operating temperature of picolog TC08 is 50 C, and accuracy is lost above 30 C.
https://www.picotech.com/data-logger/tc-08/usb-tc-08-specifications

Stopped baking. Removed thermocouple 1-8 "Octagon (side)" from Octagon gradient calculation and recompiled.
Restarted manually at T = 80 C

Please move picolog away from the tent tomorrow, and we can then increase temperature. Tent does not need to be
opened.

Sun 13 Sep 19:00
Vessel temperature at 80 C stable. 
Other temperatures stable. 
Leaving at 80 C overnight.

16 September

08:50
Baking continues OK. Temperatures stable, did not increase with respect to yesterday evening.
See attached.

15:00
Baking continues OK.
See attached.

20:22
Baking continues OK.
See attached.

17 September

08:20
Baking continues OK. Temperatures slightly decreasing?
See attached.

(NOTE: Raspberry Pi internal clock appears to be on GMT, rather than GMT+1 due to daylight saving.)

Baking started on Mon 28 Sep. Due to WiFi connection authorisation expiring, no remote access was possible until
Tue 28 afternoon.

JM found remotely that temperature had increase up to 80 C, and baking had then been shut off due to negative
gradients. Unclear what happened. System was already cooling.

Restarted baking in automatic mode. Baking shut down automatically after few hours. It is likely the automatic
mode was being confused by vessel cooling and being heated at the same time. 

29 SEP

20:30 Switched to manual mode (60 C). Left overnight

---

30 SEPTEMBER

08:10 Temperatures seem stable, but slight temperature gradient across vessel would prevent automatic mode from
working. See attached.
Unclear why RE72-1 let temperature reach 64 C when setpoint is 60 C. May be affected by RE72-2. Not worrying,
but may confuse automatic mode.
Ramped up to 70 C manually.

11:10 Temperatures stable. Ramped to 80 C manually. Gradient in CARME vessel unchanged at ~11 C. See attached.

14:18 Found baking had stopped shortly after last check. One floating thermocouple reported temperatures above 300 C. 
Unclear what happened. Possible fault? See attached. 
Restarted baking manually at 40 C. Waiting for temperature to stablise. Vessel still cooling - cannot activate
automatic mode.

15:00 Temperatures OK. Ramped manually to 50C

18:10 Temperatures stable. Ramped manually to 60 C. Left automatic mode on

20:20 Found temperature setpoint at 66 C. See attached.
Huge spikes in thermocouple 1-4 Floating (side-low). Same thermocouple that reported temperature spikes earlier. May be
faulty or touching heater directly. Seems to happen for T=65 C and above.
Set to manual mode. Temperature setpoint 60 C. Left overnight.
Will ask local people to remove thermocouple 4 from picologger 1 (aka #A0067/395, port 4) to continue.

---

1 OCTOBER

08:00 Temperatures stable. Interesting behaviour of problematic thermocouple 1-4. Trend looks real. Seems to confirm
hypothesis it is touching a heater.
Waiting for DL locals to disconnect it.

09:49 Thermocouple 1-4 disconnected. Increased temperature manually to 75 C / 70 C. 
It appears RE72-1 consistently reports 5C above its set point. It controls 3 heaters vs. 8 heaters controlled by RE72-2. 
This is most likely due to RE72-2 needing greater power output to reach its setpoint, increasing RE72-1 thermocouple
temperature as a result. 
RE72-2 may be doing very little at the moment.
Hopefully placing its setpoint 5 C above will force both heaters to work together. This may decrease temperature
gradients observed in the vessel.

16:20 Increased temperature manually in steps up to 95 / 80 C.
The uneven temperature setpoints associated with the two RE72 appears to have reduced the temperature gradient in the vessel.
Will ramp up to 100 later this evening and leave overnight.
See attached.

22:00 Temperature stable. Rates are showing unphysical rates (too high / too low) and occasionally going all to zero at the
same time. 
Looks like a previously observed bug that I thought had been sorted. Will have to debug again. Does not affect manual mode.
Ramped temperature to 105 / 95 C. Leaving overnight.
See attached.

---

2 OCTOBER

07:58 Temperatures stable. See attached. Slight gradient in CARME vessel (~11 C). Not worrying.
Ramped manually to 115 / 105 C

12:16 Temperatures stable. See attached. Valve temperature at 46 C.
Ramped manually to 125 / 115 C

17:34 Temperatures stable. See attached. We seem to be getting close to the upper limit of what the tent can achieve.
Ramped manually to 135 / 125 C
Will leave overnight.

---

3 OCTOBER

08:32 We have been baking at full power (heaters always on). Temperatures stable. 
Appears to be as high as we can go with this tent configuration.
Will leave in manual mode and monitor throughout the day.
See attached.

17:00 Temperatures stable. No changes from this morning.

21:00 Temperatures stable. See attached. Baking continues overnight.

---

4 OCTOBER

07:30 Found temperatures decreased overnight by around 10 C. See attached. 
We are still baking at full power. Room temperature has not changed.
Most likely explanation is part of the tent giving / moving due to the baking and a draft being developed.
Will keep monitoring during the day.

13:50 Temperatures stable, no changes from last check.

20:22 Temperatures stable. See attached. Leaving overnight.

Thu 22 October

22:15 Temperatures stables, slowly decreasing.
Bump - rising front corresponds to heaters turned on. Falling front to D2000 turned off?
See attached.

Fri 23 October
09:57 Temperatures stable. See attached.
13:09 Temperatures remain stable. No significant changes.

Sat 24 October
21:27 Temperatures stable. See attached.

Sun 25 October
*Daylight saving is in effect. Time on the Raspberry Pi is no longer incorrect*
07:57 Temperatures stable. See attached.
19:05 Temperatues mostly stable. See attached. Will leave overnight and move setpoint to 80 C early tomorrow morning

Mon 26 October
07:37 Temperatures mostly stable. See attached. Setpoint to 80 C.

Accessed Pi remotely 

( 1. Without the Internet, the Pi has no way of knowing what time is it.
In case of power loss make sure to set correct date via

sudo date -s 'YYYY-MM-DD HH:MM:SS'

Or all log files will have wrong timestamp )

EDIT: Pi now connected to GSI time server.


2. Temperature logging OK


3. Commands/events logging enabled. Logging to file Log.dat, automatically moved to Logs/$(timestamp)_log.dat on
gracefuly quit.
Cannot test due to TC08 / RE72 not being connected. Please test when possible.


4. Plot_temps.gp gnuplot script currently optimised for 1080p monitor. Monitor in use is 1024x768. 
Please optimise first line of script (size, position) to make it fit while we search for a better monitor.


5. Hostname changed to CARMEbakingpi

CARME pressure moved to

https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?e=hEoZt7


Some photos overnight are over-exposed and pressure cannot be read. 
Might be able to fix with image processing if needed, but likely not worth the time. 

Baking started: 23/09/2021 14:00	
Photo frequency changed to once per 10 minutes.

1. Connect to pi on CRYRING local network 
ssh pi@192.168.207.127

Password available on request


2. To check state of baking code

screen -x

please do not press any keyboard button if unsure how the code works. 
Pressing q instantly stops baking in case of emergencies

when done, to disconnect

CTRL+A 
d


3. To check log of commands / previous status of baking code

cd Programs/Baking
cat Log.dat


4. To check temperature plot

cd Programs/Baking
gnuplot Plot_temps.gp 

CTRL+C on terminal to quit

Please remember the Pi has limited computing power and the gnuplot window will consume some of that when open
One remote user is no problem, but multiple may give issues to the Pi


5. Check current pressure

Currently:
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?e=hEoZt7


(Pictures from webcam are in ~/Webcam on the pi
CM52 data also to be available online in real time soon)

Current temperature ~64 C

See plots attached

Current pressure ~ 2.5E-6 mbar

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug

and
http://web-docs.gsi.de/~lestinsk/vacplot.php

Current temperature ~90 C
However note setpoint is at 102 C. It appears we are baking at full power, but too many heat losses?
Temperature gradient still increasing. May just be a matter of time.

See plots attached


Current pressure ~ 1.9E-6 mbar decreasing

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug

and
http://web-docs.gsi.de/~lestinsk/vacplot.php

Baking automatically stopped due to negative gradient in on thermocouple for 10 minutes straight. 
Not surprising giving we were baking at full power bu making little progress. We should have switched to manual
mode.

Switched to manual mode for now at 60 C to recover temperature without excessive gradients. 
Will restart automatic mode later today. Should be back to full power tomorrow morning.

EDIT
21:42 CET - Temperature stabilised. Restarted automatic mode.

Current temperature ~87C

See plot and screen attached.
Note Baking code restarted yesterday evening.

Current pressure 2.1E-7 increasing

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Attached.

Current temperature ~87 C, increasing
See plot and screen attached.
Thermocouple on CARME foot side (1-4) may have come loose judging from sudden jump in temperature at 14:30?
Improvement to blanket seem to have significant effect.

Current pressure ~3.5E-8 increasing
See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Current temperature ~132 C (on manual mode RE72-1: 132 C, RE72-2: 130 C)
See plot and screen attached.

Current pressure 2.03E-7
See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Attached

Dry run complete.

YR09 section valves closed. Section isolated. Manual valve between electron target and CARME remain open.

There appear to be issues with Shahab's caenlogging program. It crashes occasionally, with an error indicating the number of columns in the terminal is incorrect. We will have to ask Shahab.

For now

1. Start the caen_screen.sh script on nnrp1 (aka CARMEMotorsPi -  why was it renamed? It should stay as CARMEMotorsPi to avoid duplicate names).

This starts connect to the caen power module via TTY inside a screen called caenhv. Currently on desktop 1

2. On nnrpi1, start the caenlogging program --settings caenlogger_settings_gsi.toml

3. On carme-gsi, in ~/Programs/caenlogger start the gnuplot script

If the caenlogger script crashes, restart it on the second terminal

More investigations required on the crash.

(ELOG does not allow more pictures to be added to previous entry)