16 September

08:50
Baking continues OK. Temperatures stable, did not increase with respect to yesterday evening.
See attached.

15:00
Baking continues OK.
See attached.

20:22
Baking continues OK.
See attached.

17 September

08:20
Baking continues OK. Temperatures slightly decreasing?
See attached.

(NOTE: Raspberry Pi internal clock appears to be on GMT, rather than GMT+1 due to daylight saving.)

Baking started on Mon 28 Sep. Due to WiFi connection authorisation expiring, no remote access was possible until
Tue 28 afternoon.

JM found remotely that temperature had increase up to 80 C, and baking had then been shut off due to negative
gradients. Unclear what happened. System was already cooling.

Restarted baking in automatic mode. Baking shut down automatically after few hours. It is likely the automatic
mode was being confused by vessel cooling and being heated at the same time. 

29 SEP

20:30 Switched to manual mode (60 C). Left overnight

---

30 SEPTEMBER

08:10 Temperatures seem stable, but slight temperature gradient across vessel would prevent automatic mode from
working. See attached.
Unclear why RE72-1 let temperature reach 64 C when setpoint is 60 C. May be affected by RE72-2. Not worrying,
but may confuse automatic mode.
Ramped up to 70 C manually.

11:10 Temperatures stable. Ramped to 80 C manually. Gradient in CARME vessel unchanged at ~11 C. See attached.

14:18 Found baking had stopped shortly after last check. One floating thermocouple reported temperatures above 300 C. 
Unclear what happened. Possible fault? See attached. 
Restarted baking manually at 40 C. Waiting for temperature to stablise. Vessel still cooling - cannot activate
automatic mode.

15:00 Temperatures OK. Ramped manually to 50C

18:10 Temperatures stable. Ramped manually to 60 C. Left automatic mode on

20:20 Found temperature setpoint at 66 C. See attached.
Huge spikes in thermocouple 1-4 Floating (side-low). Same thermocouple that reported temperature spikes earlier. May be
faulty or touching heater directly. Seems to happen for T=65 C and above.
Set to manual mode. Temperature setpoint 60 C. Left overnight.
Will ask local people to remove thermocouple 4 from picologger 1 (aka #A0067/395, port 4) to continue.

---

1 OCTOBER

08:00 Temperatures stable. Interesting behaviour of problematic thermocouple 1-4. Trend looks real. Seems to confirm
hypothesis it is touching a heater.
Waiting for DL locals to disconnect it.

09:49 Thermocouple 1-4 disconnected. Increased temperature manually to 75 C / 70 C. 
It appears RE72-1 consistently reports 5C above its set point. It controls 3 heaters vs. 8 heaters controlled by RE72-2. 
This is most likely due to RE72-2 needing greater power output to reach its setpoint, increasing RE72-1 thermocouple
temperature as a result. 
RE72-2 may be doing very little at the moment.
Hopefully placing its setpoint 5 C above will force both heaters to work together. This may decrease temperature
gradients observed in the vessel.

16:20 Increased temperature manually in steps up to 95 / 80 C.
The uneven temperature setpoints associated with the two RE72 appears to have reduced the temperature gradient in the vessel.
Will ramp up to 100 later this evening and leave overnight.
See attached.

22:00 Temperature stable. Rates are showing unphysical rates (too high / too low) and occasionally going all to zero at the
same time. 
Looks like a previously observed bug that I thought had been sorted. Will have to debug again. Does not affect manual mode.
Ramped temperature to 105 / 95 C. Leaving overnight.
See attached.

---

2 OCTOBER

07:58 Temperatures stable. See attached. Slight gradient in CARME vessel (~11 C). Not worrying.
Ramped manually to 115 / 105 C

12:16 Temperatures stable. See attached. Valve temperature at 46 C.
Ramped manually to 125 / 115 C

17:34 Temperatures stable. See attached. We seem to be getting close to the upper limit of what the tent can achieve.
Ramped manually to 135 / 125 C
Will leave overnight.

---

3 OCTOBER

08:32 We have been baking at full power (heaters always on). Temperatures stable. 
Appears to be as high as we can go with this tent configuration.
Will leave in manual mode and monitor throughout the day.
See attached.

17:00 Temperatures stable. No changes from this morning.

21:00 Temperatures stable. See attached. Baking continues overnight.

---

4 OCTOBER

07:30 Found temperatures decreased overnight by around 10 C. See attached. 
We are still baking at full power. Room temperature has not changed.
Most likely explanation is part of the tent giving / moving due to the baking and a draft being developed.
Will keep monitoring during the day.

13:50 Temperatures stable, no changes from last check.

20:22 Temperatures stable. See attached. Leaving overnight.

Thu 22 October

22:15 Temperatures stables, slowly decreasing.
Bump - rising front corresponds to heaters turned on. Falling front to D2000 turned off?
See attached.

Fri 23 October
09:57 Temperatures stable. See attached.
13:09 Temperatures remain stable. No significant changes.

Sat 24 October
21:27 Temperatures stable. See attached.

Sun 25 October
*Daylight saving is in effect. Time on the Raspberry Pi is no longer incorrect*
07:57 Temperatures stable. See attached.
19:05 Temperatues mostly stable. See attached. Will leave overnight and move setpoint to 80 C early tomorrow morning

Mon 26 October
07:37 Temperatures mostly stable. See attached. Setpoint to 80 C.

Accessed Pi remotely 

( 1. Without the Internet, the Pi has no way of knowing what time is it.
In case of power loss make sure to set correct date via

sudo date -s 'YYYY-MM-DD HH:MM:SS'

Or all log files will have wrong timestamp )

EDIT: Pi now connected to GSI time server.


2. Temperature logging OK


3. Commands/events logging enabled. Logging to file Log.dat, automatically moved to Logs/$(timestamp)_log.dat on
gracefuly quit.
Cannot test due to TC08 / RE72 not being connected. Please test when possible.


4. Plot_temps.gp gnuplot script currently optimised for 1080p monitor. Monitor in use is 1024x768. 
Please optimise first line of script (size, position) to make it fit while we search for a better monitor.


5. Hostname changed to CARMEbakingpi

CARME pressure moved to

https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?e=hEoZt7


Some photos overnight are over-exposed and pressure cannot be read. 
Might be able to fix with image processing if needed, but likely not worth the time. 

Baking started: 23/09/2021 14:00	
Photo frequency changed to once per 10 minutes.

1. Connect to pi on CRYRING local network 
ssh pi@192.168.207.127

Password available on request


2. To check state of baking code

screen -x

please do not press any keyboard button if unsure how the code works. 
Pressing q instantly stops baking in case of emergencies

when done, to disconnect

CTRL+A 
d


3. To check log of commands / previous status of baking code

cd Programs/Baking
cat Log.dat


4. To check temperature plot

cd Programs/Baking
gnuplot Plot_temps.gp 

CTRL+C on terminal to quit

Please remember the Pi has limited computing power and the gnuplot window will consume some of that when open
One remote user is no problem, but multiple may give issues to the Pi


5. Check current pressure

Currently:
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?e=hEoZt7


(Pictures from webcam are in ~/Webcam on the pi
CM52 data also to be available online in real time soon)

Current temperature ~64 C

See plots attached

Current pressure ~ 2.5E-6 mbar

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug

and
http://web-docs.gsi.de/~lestinsk/vacplot.php

Current temperature ~90 C
However note setpoint is at 102 C. It appears we are baking at full power, but too many heat losses?
Temperature gradient still increasing. May just be a matter of time.

See plots attached


Current pressure ~ 1.9E-6 mbar decreasing

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug

and
http://web-docs.gsi.de/~lestinsk/vacplot.php

Baking automatically stopped due to negative gradient in on thermocouple for 10 minutes straight. 
Not surprising giving we were baking at full power bu making little progress. We should have switched to manual
mode.

Switched to manual mode for now at 60 C to recover temperature without excessive gradients. 
Will restart automatic mode later today. Should be back to full power tomorrow morning.

EDIT
21:42 CET - Temperature stabilised. Restarted automatic mode.

Current temperature ~87C

See plot and screen attached.
Note Baking code restarted yesterday evening.

Current pressure 2.1E-7 increasing

See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Attached.

Current temperature ~87 C, increasing
See plot and screen attached.
Thermocouple on CARME foot side (1-4) may have come loose judging from sudden jump in temperature at 14:30?
Improvement to blanket seem to have significant effect.

Current pressure ~3.5E-8 increasing
See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Current temperature ~132 C (on manual mode RE72-1: 132 C, RE72-2: 130 C)
See plot and screen attached.

Current pressure 2.03E-7
See
https://uoe-my.sharepoint.com/:x:/g/personal/cbruno2_ed_ac_uk/EVbsVOubNHRLj0PKiFIvd00BClMJ6WSJ1UDAyLA2F3R4jA?rtime=OK4kVZKB2Ug
http://web-docs.gsi.de/~lestinsk/vacplot.php

Attached

Dry run complete.

YR09 section valves closed. Section isolated. Manual valve between electron target and CARME remain open.

There appear to be issues with Shahab's caenlogging program. It crashes occasionally, with an error indicating the number of columns in the terminal is incorrect. We will have to ask Shahab.

For now

1. Start the caen_screen.sh script on nnrp1 (aka CARMEMotorsPi -  why was it renamed? It should stay as CARMEMotorsPi to avoid duplicate names).

This starts connect to the caen power module via TTY inside a screen called caenhv. Currently on desktop 1

2. On nnrpi1, start the caenlogging program --settings caenlogger_settings_gsi.toml

3. On carme-gsi, in ~/Programs/caenlogger start the gnuplot script

If the caenlogger script crashes, restart it on the second terminal

More investigations required on the crash.

(ELOG does not allow more pictures to be added to previous entry)

0. Give split-funnels to OG or Grossmontage to clean. Mount back foot support for gate valves. Critical when we split.

1. Wait for vacuum division to come to install: IE514 cables to current CM52 module, ion pump cables + controls + turn on, RGA head + program + turn on

2. Vacuum division should also bring N2 gas to vent CARME. Ideally vent by Wed to avoid losing time. Remember to disable ion pumps and remove all cables when venting.

3. While waiting to vent, connect pneumatic hoses to motors box in CARME rack. There is one pneumatic in (central feedthrough) and four pneumatic out. See picture attached. Top two go to left pneumatic motor, bottom two go to right pneumatic motor. Of each set of two, one is for extending the motor and one is for retracting but unclear which is which. Try to connect and operate motor manually by pressing on buttons on top of pressure reducer to ensure everything works. Don't forget to remove the safety pin.

4. Some disassembly of the servo motors may be required to get them operational. Use the Windows ClearView program provided by Teknic to control servos and change the function of the limit switches. Download from https://teknic.com/downloads/ -> ClearPath -> Software -> ClearView Install.zip . There is a Linux version but I have never tested it. The code on the raspberry Pi cannot reproduce some of the functions in the official program due to library limitations (but can do other things the official program cannot do).

5. After venting CARME, split open at the COF600 flange close to the magnet. Open CARME by 1-2 cm, remove spherical washers from top of rail studs, and lift with crane. Make sure knife edges of the two halves do not touch. CARME currently has feet mounted and cannot easily be placed on a standard pallet. Either dismount the feet (not easy to remount), or find a pallet on which CARME can land on its belly only, or place CARME on pallet with feet and screw feet down? Discuss strategy with DR asap.

6. Install split-funnels on flanges/wire harnesses in use (no point installing on other flanges). This may require dismounting top flanges. Probably switch from 3 hole to 4 hole configuration with funnels.

7. Test detector support going to actuator rod without detector. Make sure you have all the parts, including bolt to secure support to rod. You may want to push the rod in via pneumatic motor (or by hand pushing the plate outside) to gain better access.

8. Take detector support on table, install *two* detectors on same support, and mount on actuator rod in XHV. Connect wires and close via jacking plates. Optionally, consider mouting empty support on other rod? May be useful for diagnostic/scraping?

9. Move XHV thermocouple feedthrough to top flange and connect one thermocouple per detector, one to the support, and leave one floating inside

10. If the motors are fully operational, check how far/close they can be actuated from chamber wall. If not, take some dimensions for future reference.

11. Lift CARME back in place. Close using Pfeiffer COF600 seal and ~72 Nm force per bolt. 80 Nm were required in DL. Probably get Grossmontage to do this. Use our long torque-wrench.

12. Connect to interaction chamber via bellows. Beware gaskets on one side are non standard. One appropriate gasket should be on top of bellows box.

13. Reconnect pumps, rebuild octagon, reconnect thermocouples, rebuild baking tent, bake. Make sure to label the NEG cables this time!

14. Activate NEG pumps following this recipe. ALWAYS monitor temperature and be ready to stop if detector temperature reaches 120 C. NEG temperature does not drop immediately, don't wait until it is too late.

If temperature is increasing too quickly (check rates/h), decrease activation power in increments of 25 W. Remember to flash ion pumps.

A. Start from D2000. Activate one at a time with T 400 C, t rise 30 min, t hold 90 min. Activating a second D2000 while the first has reached its hold state seems safe. Leave at 250 C to keep clean.

B. Continue with UVH1400 furthest from the detector (DSSD wall chamber). Activate at P 200 W, t rise 30 min, to hold 90 min. Finish one set of two before moving to the second. Leave at 50 W to keep clean.

C. Continue with UHV1400 in intermediate chamber. Activate at P 150 W, rest as above. Leave at 50 W to keep clean.

D. Continue with 2xUHV1400 closest to detectors. Activate at P 70 W, rest as above. Leave at 50 W to keep clean.

E. Finish with 3xUHV1400 closest to detectors. Activate at P 90 W, rest as above. Turn all NEGs off when activation of 3x is complete.

F. Let chamber cool. Activate ion pumps at room temperature. Section off turbo pump.

EMERGENCY SHUTDOWNS

1. CARME motors: Q key on keyboard while on workspace 1 of appc218 (right of this double-monitor setup)

2. Internal target: Q on keyboard while on workspace 3 of appc218, go to server room, press "CLOSE" button on DUMP.

CARME DAQ

CARME computer has 6 workspaces (WS), two monitors for each WS.

• WS1 : left monitor - fee power

          right monitor - fee power server

• WS2 : left monitor - DAQ controls

          right monitor - CAEN bias control

Most important tabs (right hand monitor)

• Run control : shows daq status, stopped / going
• Fee temperatures : temperatures of fees labelled aida01, aida02, aida03, aida04, aida05
• Statistics : shows the statistics for each fee
• System wide checks : checks for the setup
• Spectrum browser : loads histograms for fees and fee channels from preset layouts
• Asic control : sets asic parameters

• WS3 : left monitor - Merge control, tape server

          right monitor - servers, data being written

• WS4 : leakage current plot
• WS5 : online data monitoring
• WS6 : browser, elog
• Motors are running on second computer

TO DO FOR THE SHIFTERS - CHECK EVERY HOUR AND POST TO ELOG WITH SCREENSHOTS

-->IMPORTANT NOTE: MIDAS DAQ can respond slowly. DO NOT try to change tab or give new command while the previous one is being executed. Check activity in right-hand part of tab currently in use. Do not double click on buttons.

Motors computer (Computer to the right)

• Check motors cycle is running. The code will cycle through the loaded motors instructions. When an instruction is being done it is highlighted in bold. Check the bold highlighting cycles through entire set of instructions.
• Check potentiometer is reproducing same max/min values for each cycle of instructions. 74.02 fully in, 29.5 fully out.
• Detectors are on LEFT arm. Nothing on RIGHT arm which should not move.
• No need for screenshots, but record in the ELOG you checked this.

Merger, Tape server (WS3)

• Check NewMerger tab. Hit reload button and see if 'current links with data' and 'current merged data rate' are updating. All links should go green, but not necessarily at the same time. Keep hitting refresh until you see all of them going green at least once.
• Data rate should be around 100,000. Take screenshot.
• Check tapeService tab. Hit reload button. Check data rate in Kbytes/sec box. Record data rate to elog. This is where the current run number is displayed. Take screenshot.
• On the right-hand monitor, bottom right terminal, check the current run fragment (sub-run) is being written and is slowly increasing in size. It refreshed automatically every 5 seconds.

Leakage current (WS4)

• Check leakage current plot is stable. Make sure it is updating and time is correct. Take screenshot. Check present values on WS2, CAEN window, channel 1.

DAQ (WS2)

• Check Fee temperatures tab. Hit reload, temperatures will update and should all be green. 1-2 C above safe limit (top) is OK. More than that - warn expert shifter immediately. Take screenshot.
• Check system wide checks tab. Click on 'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status' to run checks. Copy-paste test in the ELOG the first time you do it. If there are no changes throughout the shift, just record no changes.
• Check statistics tab. Hit Update All to see if the counter and rate are updating. Change counter being displayed using drop-down menu on the right hand side of the left monitor.
  • Good events - aida 1-4 should have similar counting rates. If any rate is a lot higher than the others, warn expert shifter.
  • ADC coincidence-info #8
  • Take screenshot of each for the ELOG
• Spectrum browser tab. Check it is displaying aida 1-4 rate (top right of each histogram). IF NOT: select layout 1 from 'layout ID' then restore layout from 'Arrange functions' drop down menu. Select log scale and change xmax as 128. This displays the data rate per strip. If any strips are a lot higher than the others, or very different from previous entry, make a note in the ELOG. Take screenshot.

Online monitoring (WS 5)

• Check the following plots are updating
  • LowEnergyXYTotal (right click - col - colz). This shows the total number of events, per pixel, since the code was started / re-started.
  • LowEnergyXYRate (right click - col - colz). This shows the rate over the last minute.
  • LowEnergyExTotal (double left click. Right click on y axis - Log scale on y). This shows the total energy deposited on all X strips since code restart.
  • LowEnergyEyTotal (as above). as above, Y strips.
• Please take screenshot of each

13:20 Stopped run.

Slow comp thresh 0x64 -> 0x32 on all FEEs

Checkload x2

New run - R55

Restarted Data Spy code

Zerod histograms

Data rate acceptable.

Dead channels seem to come back online

Please check for deadtime/idle time once we collect enough stats.