- water flow/interlock checks
   check water flow to/from inflow/outflow manifolds
   check all coolant lines to all FEE64s are connected
   check USB-controlled ac mains relay 'water' LED illuminated                                 water flow and dew point sensor interlocks OK

- carme-gsi network cabling
   from patch panel YR10/2
    yellow network cable to carme-gsi operating in 19" rack room (2.033)
    green network cable to carme-gsi operating in counting room (2.034)

- boot carme-gsi
   systemctl start dhcpd                                                                       start DHCP dynamic network configuration 
   ssh -L 8080:proxy.gsi.de:8080 carme@atppc025                                                establish proxy port for AnyDesk access
   ssh -X pi@nnrpi1
    /MIDAS/TclHttpd/linux-arm/TclHttpd-server &                                                start MIDAS server
   ssh -X pi@nnrpi1
    ./caen_screen.sh                                                                           start CAEN N1419ET control interface

- enable current logging
  open a new terminal
  ssh pi@nnrpi1
  cd Programs/caenlogger
  caenlogger --config caenlogger_config_gsi.toml
   
  open a new terminal
  cd Programs/caenlogger
  gnuplot plot_current.gp

- system log
   sudo tail -f /var/log/messages                                                              monitor system log e.g. filesystem mounts for FEE64 embedded processors
   (sudo does not work - use su instead. will have to fix)

- FEE64 power ON
   Start browser on nnrpi1:8015
   Power On

- Start MIDAS
   select server start icon (leftmost)                                                            1x terminal session will appear
   Start browser tab localhost:8015
 
   Data Acquisition Control Services
   Update (nothing should change)

   Act on all Data acquisition server TICKED   
   RESET
   SETUP
   
   Enable histograms and waveforms

  Open ASIC control
  Act on all FEE64 modules
  Act on all ASICS
  System functions -> Check ASIC control

  Check FEE64 temperatures (Reload)

  System wide checks
     Clock
     ADC Calibration
     White Rabbit
     Drop down -> collect FPGA errors
     Check FPGA
     Synchronise ASIC clocks

  FADC Align & Check - all FEE64s
  Drop down -> calibrate ADCs
  System wide checks
    Clock
    ADC calibration
  If any FEE fails, FADC calibrate it again

  DAQ GO

  WR timestamp                                                                  check WR timestamp monotonically increases aida01->aida16          


  To startup Merger and Tape Server

  DAQ STOP                                                 

  8115 server, tape server, merger -> from buttons on top 
         3x terminal sessions will appear

  Start browser tab localhost:8115
  New Merge control
  Tape Service
  NetVar Service

  Merger tab
   (top) setup
   go (ignore error message)
   enable output to data storage                                                Merger State = GOing : merge/running : xfer Links => Merger => Storage
  
  tape server
   Inquire Available Devices
   Allocate Device
   
   Enter volume name (if already defined)
   Mount volume
   Reload   
   Set run number
   Open file

   Open a new terminal
   Navigate to /TapeData/ and the volume name
   watch ls -lht                                                                

   Enable data transfer #1 (in Run Control)
   X should appear next to every FEE


  Check netvar variable NetVar.MERGE.RunOptions = 1

   DAQ GO
    check merger rate and activity of all FEE64s
    check tape server blocks written number is increasing
   Server state -> Go

1. Check Proxy server is correct

localhost : 8080

incoming connections 7070

Establish port

ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

2. Check a password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)

1) Select 'No Storage' [workspace 4, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode please ignore this step

2) STOP DAQ [workspace 3, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

3) Restart Merger by selecting the Merger icon [workspace 4, display 2]
    terminal window showing the startup will popup

4) Setup Merger by selecting 'SETUP' [workspace 4, display 1, browser tab 'NewMerger Control']

5) Start Merger by selecting 'GO' [workspace 4, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

6) Start DAQ by selecting 'GO' [workspace 3, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 4, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 4, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

8) Unselect 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode at the start of this procedure please ignore this step


   If you continue to have problems, please request expert assistance.

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5)   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

6) Start DAQ

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

8) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

9) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

10) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.

Interactive Motion Program [workspace 6, display 2]

c                         stop current command(s)
l s by -20 mm             confirm y at prompt
l s by 20 mm              confirm y at prompt
read FakeMotion.dat       confirm y at prompt

ONLY FOLLOW THESE INSTRUCTIONS if E1 turbo is below 820 Hz, pressure in E1 is above 2.5E-1 mbar and experts (Nikos, Andreas) cannot be reached.

Go to the red cabinet under the stairs.

Open rightmost door. See first photo. Do not press the button on the handle or the door will lock!

Turn down pressure regulator indicated in second photo by approximately 1 degree. Read regulator to know which way to turn to decrease. No immediate change should be obvious on the pressure regulator.

Check E1 pressure for 5 minutes. If no effect, turn pressure down again until an effect appears.

ML reports that baking of the upstream section (gas jet interaction chamber plus e-target) has commenced - initially to 50 deg C to check thermocouples. If OK baking 
temperature will be ramped slowly (which I think means c. 10 deg C per h) to limits established for specific components by the e-target.

Ragan and Oemer are currently threading a 10m heating tape helically around the CARME chamber.

16.20 ML reports baking of upstream section is proceeding OK - currently ramping to 100 deg C at 10 deg C / h.

16.25 RSS reports that he completed the install of the thermocouples this morning and will update the Elog with specific locations of each thermocuple and TC08 channel 
later today.

16:30 All CARME thermocouples <20 C. No temperature changes. Monitor.x left under screen. Please summon with screen -x and dismiss with CTRL+a d

17:40 One CARME thermocouple at 40C (CARME front)

(Wed 27 October)

Started with top-most flange. Attemptedto plug in MACOR lead in DSSD plug in preparation for jacking in. Realised that extender plate thickness causes plug to be rasied with respect to nominal position, which results in plug non entering straight. Removed extension plate.

Plugged MACOR lead in. OK. Inserted M6 jacking bolts and gently turned by hand. No resistance until jacking action starts. Jacked plug in with spanner and even 1/4 turns. Removed bolts. All OK.

Attempted same with bottom-most cable harness. Due to strain from top-most wire harness the plate is now rotated on the rod. Rotated back. Jacked in. Removed bolts. No issues.

MACOR plugs show small gap in rear part. Gap seems due to harness strain and disappears depending on plate orientation. Will check but pins are obviously connected.

(Thu 28 October)

Attempted to plug in top-left MACOR lead. Dog leg conflicts with jacking screw! Only solution is to slacken bolt, rotate strut, fasten again with detectors still on plate. Result attached. No clear if design, manufacturing or mounting oversight.

Had to attemping plugging in a few times before jacking bolts went in straight (i.e. no resistance felt). Jacked in both leads. No further issues.

Gap appears less significant in newly plugged leads. probably as a result of lack of rotational freedom.

Week ending friday 14/08/20

CARME chamber securely mounted upon aluminium frame and support skis. Neg-ion pumps and right angled flanges installed. Neg pumps installed in the wall detector section of the chamber. Neg pumps continuity checked and are electically isolated from the chamber. 

03.37 DAQ continues
      Data file APR25/R5_8

      DSSSD bias & leakage current - attachment 1
      
      FEE64 temps OK - attachment 2

      ADC data item stats - attachment 3

      Merger & Tape Server - attachment 4-5

10.23 New run 
      Data file APR25/R6
      alpha background - pulser OFF
      all histograms & stats zero'd

      DSSSD bias & leakage current - attachment 6
      
      FEE64 temps OK - attachment 7

      ADC data item stats - attachment 8

      Merger & Tape Server - attachment 9-10

      WR timestamps OK - attachment 11

Courtesy Nik Kurz, JG has installed a PEXARIA5 PCIe card adjacent to CARME - BuTIS single-mode fibre input
and HDMI output to root MACB input for distribution to the other MACBs and FEE64s.

The PEXARIA5 card provides equivalent functionality to the VETAR2 VME module that we have used to date. 

Further information https://www.gsi.de/fileadmin/EE/Module/EXPLODER/pexaria5_14.pdf 

Cooling water ON 

Via Anydesk startup MIDAS and FEE64s

Power ON
DSSSD HV OFF ( not required for WR test )

FEE64 temperatures OK - attachment 1

All system wide checks OK

Info code 4 & 5 stats OK - attachments 2 & 3

10:48:46 WR timestamps OK - attachment 4

         Timestamp (ns) converted to (s) corresponds to 09.49.22 GMT 6.3.24 which appears to be correct

11.00 Power OFF

9:40

DSSD bias and leakage current values:

-99.99 V

-13.0960 uA

Pulsar peak widths (ch FWHM)

aida01 ~ 97

aida02 ~ 103

aida03 ~ 134

aida04 ~ 131

aida05 ~ 22

Pretty similar to what was found yesterday

15.30

Per Elog 375

Measured FEE64 voltages are as follows

@ISOL 4 AIDA PSU   @FEE64 aida05      nominal

(top-botttom)               (right-left)

+6.05V                        +5.26V                     +5V

+6.05V                        +5.26V                     +5V

+6.05V                        +5.26V                     +5V

-6.49V                         -6.26V                      -6V

+7.78V                        +7.46V                     +7V

Adjusted PowerStax PSU voltage pots of AIDA PSU ISOL 4

@ISOL 4 AIDA PSU   @FEE64 aida05      nominal

(top-botttom)               (right-left)

+5.79V                        +4.99V                     +5V

+5.79V                        +4.99V                     +5V

+5.79V                        +4.99V                     +5V

-6.23V                         -6.00V                      -6V

+7.32V                        +7.00V                     +7V

Attatchment 1-12 = BEFORE change

Attatchment 13+ = AFTER change

Pulsar peak widths (ch FWHM) AFTER CHANGE

aida01 ~ 100

aida02 ~ 96

aida03 ~ 130

aida04 ~ 130

aida05 ~ 22

10.32

CARME chamber 5.8e-12mbar (see attachment 1)
Backing 1.6e-08mbar (Edwards vacuum cart w/RGA)

CARME temps - see attachment 2

All SAES D2000-10 & UHV1400 NEG power OFF
All SAES D2000-10 ion pump OFF
Gate valve open

11.01

All SAES D2000-10 ion pump ON

#  V/V       presssure/mbar
1  5012/5019  4-8e-11
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

11.04

CARME chamber 5.7e-12mbar
Backing 1.7e-08mbar (Edwards vacuum cart w/RGA)

12.12

CARME chamber 5.3e-12mbar
Backing 1.7e-08mbar (Edwards vacuum cart w/RGA)

All SAES D2000-10 & UHV1400 NEG power OFF
Gate valve open
All SAES D2000-10 ion pump ON

#  V/V       presssure/mbar
1  5012      < 8.9e-12
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

All ion pumps now stabilising wrt V/I/pressure

log (pressure) versus time - see attachment 3

16.00

CARME chamber 4.9e-12mbar - see attachment 4
Backing 1.7e-08mbar (Edwards vacuum cart w/RGA)

All SAES D2000-10 & UHV1400 NEG power OFF
Gate valve open
All SAES D2000-10 ion pump ON - see attachment 5

#  V/V       presssure/mbar
1  5012      < 8.9e-12
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

All ion pumps now stabilising wrt V/I/pressure

CARME temps - see attachment 6

Observing pressure transients from 5e-12mbar to c. 6-7e-12mbar which recover in c. 5-10 minutes 

21.00 CARME chamber 4.7e-12mbar

16.55 CARME IE514 8.2e-11 mbar

      Vacuum pressure increased slowly overnight from c. 6e-11 mbar yesterday to c. 8e-11 mbar today following low power startup tests of the bakeout controller heating 
the CARME interaction and detector chambers yesterday and today.


Attachment 1 - bakeout controller config courtesy Anton Kalinin

Attachments 2 & 3 - bakeout controller summary displays and CARME bakeout monitor display c. 17.00 today

The 3x CARME heaters are connected to controller #4 zones # 1-3, i.e. lower right display, zones 1-3, mains fuses F9-F11

17.30 3x ( of 4x ) SAES D-2000 ion pump controllers ( SIP modules ) switched off. Fourth controller was already switched off and SHV cable disconnected from rear panel of 
SIP module.

10.39

CARME XHV chamber pressure 6.0e-12mbar
Backing 2.1e-08mbar (Edwards vacuum cart w/RGA)

CARME temps - heaters OFF - chamber at ambient 
temperature

All SAES D2000-10 & UHV1400 NEG power OFF
All SAES D2000-10 ion pump ON

#  V/V       presssure/mbar
1  5012      < 8.9e-12
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

Gate valve open

attachment 1 - analog RGA scan (as found this am)

attachment 2 - scan mode 

c. 11.15

AH partially closes gate valve using c. 2 bar air pressure

VCARME XHV chamber pressure increases c. 7e-9mbar immediately

attachment 3 - RGA scan - no obvious change in RGA trends!

attachments 4 & 5 - VAT valve indicators - upper (partially open/closed) and lower (closed)

11.33 CARME XHV chamber pressure 6.6e-9mbar
Backing 2.2e-08mbar (Edwards vacuum cart w/RGA)

All SAES D2000-10 ion pump ON

#  V/V       presssure/mbar
1  5012      < 8.9e-12
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

All SAES D2000-10 ion pump OFF

11.40 CARME XHV chamber pressure 5.64e-9mbar

11.46 CARME XHV chamber pressure 5.01e-9mbar
No change in RGA trends!

12.01 CARME XHV chamber pressure 3.78e-9mbar
No change in RGA trends!

12.11 CARME XHV chamber pressure 3.10e-9mbar

12.42 CARME XHV chamber pressure 1.81e-9mbar
Backing 2.4e-08mbar (Edwards vacuum cart w/RGA)
No change in RGA trends!

13.20 CARME XHV chamber pressure 1.06e-9mbar
Backing 2.4e-08mbar (Edwards vacuum cart w/RGA)
No change in RGA trends!

13.36 CARME XHV chamber pressure 8.93e-10mbar
Backing 2.4e-08mbar (Edwards vacuum cart w/RGA)
No change in RGA trends!

13.59 CARME XHV chamber pressure 7.07e-10mbar
Backing 2.5e-08mbar (Edwards vacuum cart w/RGA)
No change in RGA trends!

14.00 Leybold turbo OFF - decelerating

14.30 Leybold turbo OFF - decelerating - 466Hz
CARME XHV chamber pressure 5.62e-10mbar
Backing 2.1e-08mbar (Edwards vacuum cart w/RGA)
No change in RGA trends!

14.40 1x SAES D2000-10 ion pump switched on briefly - settles in c. 10s - pressure < 8.9e-
12mbar
All 4x D2000-10 ion pumps OFF

15.01 Leybold turbo OFF - decelerating - 125Hz
CARME XHV chamber pressure 4.78e-10mbar
Backing 1.4e-08mbar (Edwards vacuum cart w/RGA)
No change in RGA trends!

15.10 Abrupt increase in RGA trends - see attachment 6
Leybold turbo OFF - decelerating - 60Hz
CARME XHV chamber pressure 4.53e-10mbar
Backing 2.6e-08mbar (Edwards vacuum cart w/RGA)

15.19 4x SAES D2000-10 ion pumps ON
RGA record 04.24.00 

15.24 see attachment 7 - RGA transient recovering - turbo stopped?

15.10 Leybold turbo OFF - stopped - 0Hz
CARME XHV chamber pressure 4.40e-10mbar
Backing 1.1e-08mbar (Edwards vacuum cart w/RGA)

All SAES D2000-10 ion pump ON

#  V/V       presssure/mbar
1  5012      < 8.9e-12
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

16.10 VAT gate valve partially closed
Leybold turbo OFF - stopped - 0Hz
CARME XHV chamber pressure 3.88e-10mbar
Backing 9.8e-09mbar (Edwards vacuum cart w/RGA)
RGA traces recovered to pre-transient levels and stable

All SAES D2000-10 ion pump ON

#  V/V       presssure/mbar
1  5012      < 8.9e-12
2  5012      < 8.9e-12
3  5012      < 8.9e-12
4  5012      < 8.9e-12

      Replace AIDA ASIC mezzanine of aida01

13.43 DSSSD bias & leakage current OK - attachment 1
      ambient temperature 16.5 deg C

      FEE64 temperatures OK - attachment 2

      All system wide checks OK 

      WR timestamps OK - attachment 3

      ADC data item stats OK - attachment 4

      per FEE64 Rate spectra - attachment 5
       c. 10 strips (of 1024) not producing data
       most missing strips are from aida05 - misaligned or damaged adaptor PCB?

      per p+n FEE64 1.8.L spectra - attachment 6
       aida01 pulser peak width 75 ch FWHM
       common x/y scale => peak height proxy for peak width

      per n+n FEE64 1.8.L spectra - attachment 7
       aida01 pulser peak width 70 ch FWHM

      per FEE64 1.8.W spectra - 20us FSR - attachments 8-11

Summary

All FEE64s appear to be working (except aida05 which has c. 8 missing channels) with pulser - which 
validates the FEE64, ASICs and adaptor PCB.

Electronic noise c. 50keV FWHM for aida01 & aida03 - other FEE64s better/worse.

Preamplifier noise appears to be similar for all FEE64s which may suggest an issue with the test signal chain

Note
1) e-target is starting up
2) have identified some instances of poor CARME cable management/routing see https://elog.ph.ed.ac.uk/CARME/545



EG&G Ortec 448 Pulser
Amplitude 50,000
Normalise 5.00
x5 attenuator IN
t_r 100ns
tau_d 50us
Polarity +
Frequency relay line (50Hz)

Cooknell SA1 Summing Amplifier
inverted output to test - daisy chain

Current ASIC settings saved to 2024Apr24-14.14.03


14.48 Pulser OFF
 
      All histograms & stats zero'd

      LEC slow comparator 0x64 -> 0xff
      HEC fast comparator 0x2 -> 0x5

      background alpha run to validate signal path from DSSSD to FEE64 adaptor PCB

22.18 Start Merger & TapeServer - attachments 12-14

      DSSSD bias & leakage current OK - attachment 21
      ambient temperature 16.0 deg C


      FEE64 temperatures OK - attachment 15

      WR timestamps OK - attachment 16

      ADC data item stats OK - attachment 17

      per FEE64 Rate spectra - attachment 18

      per p+n FEE64 1.8.L spectra - attachments 19-20

Connected Leybold Ionivac IE514 to Leybold Combivac CM52 Vacuum Gauge Controller

Rear panel - power ON

POST checks OK

Change parameters by

select PARA

select <-> to display Sen

select PARA to display S-on

select <-> to display HAnd

select PARA and hold for 7s

select CH to select CH3

select -> hold for 2s 

With CARME gate valve closed for c. 24h CARME chamber pressure is 6.43e-5mbar @ 13.34, 6.52e-5mbar @ 14.15


15.45 CARME Leybold turbo restarted

16.06 1.58e-6mbar, 3.5e-5mbar (RGA station), leak rate 3.6e-10 mbar.l/s

16.27 9.44e-7mbar, 3.3e-5mbar (RGA station), leak rate 3.6e-10 mbar.l/s

16.39 7.67e-7mbar

17.00 5.85e-7mbar, 2.5e-5mbar (RGA station), leak rate 2.60e-10 mbar.l/s

17.20 4.83-7mbar, 2.3e-5mbar (RGA station), leak rate 2.17e-10 mbar.l/s

Baking octagon moved into position along with the bottom insulation and heaters
 

Assorted screenshots of the CARME, CRYRING and gas jet target control systems in operation

Attachment 1 - detector motion controls

Attachment 2 - gas jet target pressures (blue proxy for target density, red proxy for gas load to CRYRING)

Attachment 3 - Schottky probe monitor of beam

Attachment 4 - CRYRING controls monitor

Attachment 5 - CRYRING vacuum pressures

Attachment 6 - CRYRING beam position and profile monitor 

CAEN N1419ET remote control via carme-gsi and nnrpi1 - see https://elog.ph.ed.ac.uk/CARME/472

15x ( of 16x ) AIDA power cables available, labelled #1 - #15, installed

16x ( of 16x ) RJ45 Cat 6 network cables available, 15x labelled #1 - #15

16x ( of 16x ) HDMI cables available, labelled #1 - #16

Attachments 1 & 2 - 2x SAES D-2000 re-fitted to side ports
                    all 4x SAES D-2000 now re-fitted to chamber

Will now fit 1x turbo and blank (currently) unused turbo port.

Attachments 3 & 4 - centre 2x78 pin micro-D feedthrough has some pins slightly bent - can probably be safely bent back into position 

ALWAYS check 78-pin feedthroughs for bent/broken pins before fitting connectors/cabling

Attachment 5 - currently unused turbo port

Attachments 6 & 7 - turbo port from above and side

Attachment 8 - turbo + NW16/NW25 adaptor + NW25 s/s hose + scroll pump + turbo controller + CMS52 gauge controller

Note we currently do not have a vent valve or backing line vacuum gauge (both functions provided by leak detector at STFC DL).