10:00

Check Leakage current: (see attachment 2)

Check NewMerger tab: (see attachment 5)

Check Fee temperatures: all green (see attachment 1)

Check statistics tab: ADC data items (see attachment 3)

                                 ADC coincidence-info #8 (see attachment 6)

                                 Good events (see attachment 7)

 Check Spectrum Browser tab: (see attachment 9)                 

Check the following plots:

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

                                         LowEnergyExTotal (see attachment 10)

                                         LowEnergyEyTotal (see attachment 11)

Check system wide checks tab: White Rabbit error counter test result: Passed 0, Failed 5 (see attachment 8)

Both GSI vacuum group and edinburgh ion gauges OFF

Fees power cycled. All system checks OK.

Writing to file, alpha background overnight.

Moved detector IN 2mm. Noise appears reduced with any distance moved in so overnight spectra to be taken while IN. Not moving further in as possible beam tomorrow.

22:44

Leakage current stable, slowly decreasing ~3.57 uA (attachment 1)

Statistics look OK

Temperatures OK

System wide check all pass - OK

Noise appears much reduced on spectrum Browser channels. (attachement 2). Compared to same plot on elog 197

XYtotal per pixel (attachment 3)

Ex histogram (attachment 4)

Edinburgh group ion gauge turned OFF -> decrease in leakage current from 10.05 uA to 6.5 uA

GSI vacuum group ion gauge turned OFF -> decrease in leakage current from 6.5 uA to 3.63 uA

GSI large ion pump turned ON -> no effect on leakage current

Motors moved IN the same as elog 201. No significant increase in leakage current as moved further into the chamber (attachment 1). Pulser widths follow similar path as previous tests. Aida1 and 2 statrt with larger pulser widths which reduce as are moved into the chamber. Aida3,4 start lower than aida1,2 and remain at same level throughout. (attachment2)

RF ring signal removed from aida5. Data rate dropped back to ~20 kb/sec, Merger rate ~80,000 items/sec

Noise peak on aida1 appears reduced after gauges switched OFF with detector fully out (attachment 3) coompare to elog 201

Noise peak is removed once detector is moved in. (attachment 4)

Ey histogram (attachment 5) shows multiple peaks for the same pulser amplitude. 

XY total pixels plot (attachment 6)

1.8 waveforms for aida 1,2,5 (attachment 7)

16:09

Check Leakage current: (see attachment 1)

Check NewMerger tab: (see attachment 3)

Check Fee temperatures: all green (see attachment 5)

Check statistics tab: ADC data items (see attachment 2)

                                 ADC coincidence-info #8 (see attachment 6)

                                 Good events (see attachment 4)

 Check Spectrum Browser tab: (see attachment 10)                 

Check the following plots:

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

                                         LowEnergyExTotal (see attachment 7)

                                         LowEnergyEyTotal (see attachment 8)

Check system wide checks tab: White Rabbit error counter test result: Passed 0, Failed 5 (see attachment 11)

Run stopped for pulser tests 16:13

14:00

Check Leakage current: (see attachment 1)

Check NewMerger tab: (see attachment 3)

Check Fee temperatures: all green (see attachment 5)

Check statistics tab: ADC data items (see attachment 2)

                                 ADC coincidence-info #8 (see attachment 6)

                                 Good events (see attachment 4)

 Check Spectrum Browser tab: (see attachment 10)                 

Check the following plots:

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

                                         LowEnergyExTotal (see attachment 7)

                                         LowEnergyEyTotal (see attachment 8)

Check system wide checks tab: White Rabbit error counter test result: Passed 0, Failed 5 (see attachment 11)

09:15

Check Leakage current: (see attachment 2)

Check NewMerger tab: (see attachment 5)

Check Fee temperatures: all green (see attachment 1)

Check statistics tab: ADC data items (see attachment 4)

                                 ADC coincidence-info #8 (see attachment 6)

                                 Good events (see attachment 7)

 Check Spectrum Browser tab: (see attachment 10)                 

Check the following plots:

                                         LowEnergyXYTotalDSSD0 (see attachment 3)

                                         LowEnergyExTotal (see attachment 8)

                                         LowEnergyEyTotal (see attachment 9)

Check system wide checks tab: White Rabbit error counter test result: Passed 0, Failed 5 (see attachment 11)

22:41

Check Leakage current: (see attachment 1) = 10.01 uA and falling

Check NewMerger tab: (see attachment 2)

Check Tape Server: (see attachment 3) Writing ~2500 kb/s

Check Fee temperatures: all green

Check statistics tab: ADC data items (see attachment 4)  -aida3 and aida4 reading zero. Appears to be low noise as stat histogram shows counts as does Ey and Ex low energy histograms

                                 ADC coincidence-info #8 (see attachment 5) aida1,2,5 correlation #8 signals counted. Do not see aida3 correlation 8 signals. was 70-90 previously (elog201)

                                 Good events (see attachment 6) see 10-20k on aida1-4.

Low energy histograms Ex, Ey (attachemnts 7,8)

Noise appears lower across all fees. Aida1 channels for asic 1 and 3 (noisiest) checked against elog 197, same channels are hot but at lower rate to elog 197.

On 15/2/22 it was noted that the leakage current increases when the motors are moved IN. This is contary to tests conducted on 12/2/22 (elog 192) where no increase was observed. 

16/2/22: Servo motor and pneumatic used to move in by different differences with leakage current observed to increase linearly with distance moved in, measured using the potentiometer. (attachment 10)

After each movement the pulser width was measured for each of aida1,2,3,4. No change observed for aida3,4 with pulser width ~220 channels. Pulser width decreased for aida1,2 when moved IN. (attachment 11)

Noise on aida 2,3,4 appeared low throughout tests. Aida1 has a large noise peak when in fully out position. When moved fully in, the noise peak disappears (attachments 1,2)

Cryring signals for ring RF, electron cooler voltage and schottky added to the motors signal and fed into mac b to be read in daq as aida correlation info #8 signal. 

cryring bnc 13  - ring rf - aida5 ~ 500,000 rate

cryring bnc 14 - e cooler voltage - aida3 ~ 80 rate

cryring bnc 15 - shottky -aida2 ~2000 rate

cryring bnc 16 motors - aida1 ~3 rate   

correlation info #8 appears to be added to the good events rate (attachments 4,5) ADC items (attachment 3)

Waveforms 1.8*W attached for 20us, 200us, 2ms, 20ms

16:30

NOTE: Pulser is ON!

Check Leakage current: (see attachment 1)

Check NewMerger tab: (see attachment 2)

Check Fee temperatures: all green (see attachment 3)

Check statistics tab: ADC data items (see attachment 4)

                                 ADC coincidence-info #8 (see attachment 5)

                                 Good events (see attachment 6)

 Check Spectrum Browser tab: (see attachment 7)                 

Check the following plots: LowEnergyXYRateDSSD0 (see attachment 8)

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

                                         LowEnergyExTotal (see attachment 10)

                                         LowEnergyEyTotal (see attachment 11)

Check system wide checks tab: all okay.

All is good.

13:30

Check Leakage current: (see attachment 1)

Check NewMerger tab: (see attachment 2)

Check Fee temperatures: all green (see attachment 3)

Check statistics tab: ADC data items (see attachment 4)

                                 ADC coincidence-info #8 (see attachment 5)

                                 Good events (see attachment 6)

 Check Spectrum Browser tab: (see attachment 7)                 

Check the following plots: LowEnergyXYRateDSSD0 (see attachment 8)

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

                                         LowEnergyExTotal (see attachment 10)

                                         LowEnergyEyTotal (see attachment 11)

Check system wide checks tab: all okay.

All is good.

09:15

Check Leakage current: (see attachment 1)

Check NewMerger tab: (see attachment 2)

Check Fee temperatures: all green (see attachment 3)

Check statistics tab: ADC data items (see attachment 4)

                                 ADC coincidence-info #8 (see attachment 5)

                                 Good events (see attachment 7)

 Check Spectrum Browser tab: (see attachment 6)                 

Check the following plots: LowEnergyExTotal (see attachment 8)

                                         LowEnergyEyTotal (see attachment 9)

                                         LowEnergyXYRateDSSD0 (see attachment 10)

                                         LowEnergyXYTotalDSSD0 (see attachment 11)

Check system wide checks tab: all okay.

All is good.

All fees and asics checked to find noisy channels. Channels are listed along with the noise peak height for the low energy plot at 22:00. Run started ~17:45. 

Attachment 1 shows 3*L for aida01 with x axis zoomed in on noise peak. Attachment 2 shows stat for aida01-04 in log scale. 

Aida01

• 1.0L ~3000
• 1.2L ~3500
• 1.7L ~3000
• 1.8L ~2000
• 2.11L ~2000
• 2.12L ~400
• 3.7L ~1000
• 3.10 ~60000
• 3.11L ~2000
• 3.12L ~15000
• 4.12L ~400
• 4.14L ~750 

Aida02

• 1.0L ~1000
• 1.3L ~750
• 1.7L ~600
• 1.15L ~600
• 4.15 ~125

Aida03

• 1.11L ~3000

nothing noisy of note on aida04

16:00

Check Fee temperatures: all green (see attachment 1)

Check statistics tab: ADC data items (see attachment 2)

                                 Good events (see attachment 3)

                                  ADC coincidence-info #8 (see attachment 4)

Check NewMerger tab: (see attachment 5)

Check Leakage current: (see attachment 6)

Check the following plots: LowEnergyExTotal (see attachment 7)

                                         LowEnergyEyTotal (see attachment 8)

                                         LowEnergyXYTotalDSSD0 (empty.)

Check Spectrum Browser tab: (see attachment 9)

Check system wide checks tab: all okay.

All is good.

17:00

Pulser walkthrough completed on run R38

290

280

270

250

240

230

220

210

210 / 10

230 / 10 (can't see)

210 / 5 (can't see)

190

90

150

130

70

50

10 (can't see)

30

15 (can't see)

Pulser to 70

11:30

Check Fee temperatures: all green (see attachment 1)

Check statistics tab: ADC data items (see attachment 2)

                                 Good events (see attachment 3)

                                  ADC coincidence-info #8 (see attachment 4)

Check NewMerger tab: (see attachment 5)

Check Leakage current: (see attachment 6)

Check the following plots: LowEnergyExTotal (see attachment 7)

                                         LowEnergyEyTotal (see attachment 8)

                                         LowEnergyXYTotalDSSD0 (see attachment 9)

Check Spectrum Browser tab: (see attachment 10)

Check system wide checks tab: all okay.

All is good.

EMERGENCY SHUTDOWNS

1. CARME motors: Q key on keyboard while on workspace 1 of appc218 (right of this double-monitor setup)

2. Internal target: Q on keyboard while on workspace 3 of appc218, go to server room, press "CLOSE" button on DUMP.

CARME DAQ

CARME computer has 6 workspaces (WS), two monitors for each WS.

• WS1 : left monitor - fee power

          right monitor - fee power server

• WS2 : left monitor - DAQ controls

          right monitor - CAEN bias control

Most important tabs (right hand monitor)

• Run control : shows daq status, stopped / going
• Fee temperatures : temperatures of fees labelled aida01, aida02, aida03, aida04, aida05
• Statistics : shows the statistics for each fee
• System wide checks : checks for the setup
• Spectrum browser : loads histograms for fees and fee channels from preset layouts
• Asic control : sets asic parameters

• WS3 : left monitor - Merge control, tape server

          right monitor - servers, data being written

• WS4 : leakage current plot
• WS5 : online data monitoring
• WS6 : browser, elog
• Motors are running on second computer

TO DO FOR THE SHIFTERS - CHECK EVERY HOUR AND POST TO ELOG WITH SCREENSHOTS

-->IMPORTANT NOTE: MIDAS DAQ can respond slowly. DO NOT try to change tab or give new command while the previous one is being executed. Check activity in right-hand part of tab currently in use. Do not double click on buttons.

Motors computer (Computer to the right)

• Check motors cycle is running. The code will cycle through the loaded motors instructions. When an instruction is being done it is highlighted in bold. Check the bold highlighting cycles through entire set of instructions.
• Check potentiometer is reproducing same max/min values for each cycle of instructions. 74.02 fully in, 29.5 fully out.
• Detectors are on LEFT arm. Nothing on RIGHT arm which should not move.
• No need for screenshots, but record in the ELOG you checked this.

Merger, Tape server (WS3)

• Check NewMerger tab. Hit reload button and see if 'current links with data' and 'current merged data rate' are updating. All links should go green, but not necessarily at the same time. Keep hitting refresh until you see all of them going green at least once.
• Data rate should be around 100,000. Take screenshot.
• Check tapeService tab. Hit reload button. Check data rate in Kbytes/sec box. Record data rate to elog. This is where the current run number is displayed. Take screenshot.
• On the right-hand monitor, bottom right terminal, check the current run fragment (sub-run) is being written and is slowly increasing in size. It refreshed automatically every 5 seconds.

Leakage current (WS4)

• Check leakage current plot is stable. Make sure it is updating and time is correct. Take screenshot. Check present values on WS2, CAEN window, channel 1.

DAQ (WS2)

• Check Fee temperatures tab. Hit reload, temperatures will update and should all be green. 1-2 C above safe limit (top) is OK. More than that - warn expert shifter immediately. Take screenshot.
• Check system wide checks tab. Click on 'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status' to run checks. Copy-paste test in the ELOG the first time you do it. If there are no changes throughout the shift, just record no changes.
• Check statistics tab. Hit Update All to see if the counter and rate are updating. Change counter being displayed using drop-down menu on the right hand side of the left monitor.
  • Good events - aida 1-4 should have similar counting rates. If any rate is a lot higher than the others, warn expert shifter.
  • ADC coincidence-info #8
  • Take screenshot of each for the ELOG
• Spectrum browser tab. Check it is displaying aida 1-4 rate (top right of each histogram). IF NOT: select layout 1 from 'layout ID' then restore layout from 'Arrange functions' drop down menu. Select log scale and change xmax as 128. This displays the data rate per strip. If any strips are a lot higher than the others, or very different from previous entry, make a note in the ELOG. Take screenshot.

Online monitoring (WS 5)

• Check the following plots are updating
  • LowEnergyXYTotal (right click - col - colz). This shows the total number of events, per pixel, since the code was started / re-started.
  • LowEnergyXYRate (right click - col - colz). This shows the rate over the last minute.
  • LowEnergyExTotal (double left click. Right click on y axis - Log scale on y). This shows the total energy deposited on all X strips since code restart.
  • LowEnergyEyTotal (as above). as above, Y strips.
• Please take screenshot of each

Alpha background gathered from 17:35 12/2/22 to 13:44 13/2/22 is run32. Run time ~20 hours. Some alphas visible in aida1 1*L histograms attached.

Noticed grounding of crimps that attched to cryring ground has come loose. Those crimps not ideal for our grounding cable so were loose. Grounding cable from each fee now meet at a single grounding cable joined by an m3 nut and bolt. This grounding cable then goes to the cryring ground and is properly secured. See attachment 1. No significant change to pulser widths with new grounding method from previous elog. 

Aida5 appeared non-responsive, no good events / data items etc and would not stop when daq was stopped. Fee power cycled. Aida 3 and 5 then showed no data items / good events or any stats. When stopping daq all fees then became non responsive. Full restart initiated, fee power off, all servers incl merger and tape server closed. Restarted system, all fees now responsive, all show full stats and are writing data. 

Alpha background started 17:06 13/2/22 to 09:41 14/2/22 is run35. Run time ~16 hours 40 mins.

Ex, Ey 1D histograms for run32 and run35 attached. Bin size 80 keV.

> Initial grounding see final grounding from elog 187. ASIC settings for all - LEC slow comparator 0x64. Note on 11/02/22 1.8*L widths with same settings were aida1-4: 300,240,250,260

ADC data items <10k

Good events fluctuate 10k-20k

1.8*L pulser widths aida1-4: 433,400,375,375(double peak)

Attachments 1-4

> New grounding implemented. 3mm grounding cable connect fee body to adaptor board. Each adaptor board is then connected directly to the cryring ground. Attachements 10-11

ADC data items <10k

Good events fluctuate 10k-20k

1.8*L pulser widths aida1-4: 340,288,288,295 (reduced double peak)

Noise in the system also seems to have decreaed in zoomed out 1.8*L plots (not attached). Aida1 still shows large noise peak channel 32000 on 1.8*L. 1.8*W no significant change, perhaps slightly reduced.

Attachements 5-9

Screencaps for following movement tests not taken as 1.8*L and 1.8*W spectra appeared the very similar as 5-9 except for slightly different widths.  

> Detectors moved in and out (pneumatic). Leakage current dropped by ~0.2 uA, torque now reads +2.8 previously was ~0. 

no change to ADC data items or good events. During movement aida3 saw momentary increase in good events to ~40k before returning to normal range. Rest of fees good events stats unaffected by movement.

1.8*L pulser widths aida1-4: 413,313,295,306

> servo motor (only) moved, torque reduced back to ~0

no change in good events stats for any fees during servo movement. 

1.8*L pulser widths aida1-4: 425,335,316,288

> Detector moved to IN position (pnuematic and servo). 19.3mm to 81.8mm on potentiometer. 

No change to good events. During movements aida3 saw momentary increase in good events to ~41k, all other fees unaffected. 

1.8*L pulser widths aida1-4: 302,221,305,293 

Noise peak at channel ~32000 on aida 1 (1.8*L) significantly reduced. little noise on other fees 

>Detector moved to fully OUT position (pneumatic and servo). 

no change in good events, during movement aida3 saw momentary increase to ~35k

1.8*L pulser widths aida1-4: 450,350,316,290

Noise peak at chn 32000 on aida 1 has increased to previous size. little noise on other fees. 

> Bias increased to 120V. Leakage current stable however slightly higher ~0.1 uA

no change in good events. 

1.8*L pulser widths aida1-4: 447,345,189,301

> Bias increased to 140V, increase in leakage current ~0.2 uA, appears stable. Increase to 150V, fluctuate in leakage current of +/-0.5uA then huge increase to >20uA trip. 

> Fee power cycle. Bias 100V, leakage current 9.9 uA ~0.1-0.2 uA higher than previous but stable. 

no change to good events

1.8*L pulser widths aida1-4: 468, 350,311, 296

> Some decrease seen with new grounding, however main source of noise still apparent and any decrease seems insignificant compared to changes due to position movement. 

> Water pipes have been zipped tied so they no loger touch motors, no change in pulser widths or noise levels however.  

To test effect of the motors on the leakage current, they were cycled once per hour overnight at approximately 50 minutes past the hour. - attachment 1
Leakgage current was seen to decrease slowly overnight before flattening out.

At 11:25 This morning the FEEs were powered on. From this point on the leakage current was observed to start to increase. - Attachment 2
Note the time recorded by the pi logging the leakage current is approximately 2 minutes ahead of the FEE.

15:40 I notice again that powering on the FEEs causes a momentary increase in the leakage current. - Attachment 3

08:52 Overnight the DAQ was left running to check if there was any individual radioactivity in the Si.
      This is stored in R7_75

      It was observed that even with no beam the detectors continued to increase in leakage current overnight but slowly - Attachment 1
      We then stopped the DAQ and gave the detector a single shake.
      This caused the leakage currents to decrease but not to the same level that they did last night - Attachment 2

13:45 Bias tripped as people removed the cover on the windows on the interaction chamber
      Iset was still at 11uA for safety
      Will leave off until they are finished

      

09:00 Detector still biased to 60V and leakage current has been mostly stable overnight.
      Only minor fluctuations observed - attachment 1

      FEE temps all ok - attachment 2
      Stats at 60V - attachment 3
      System wide checks all ok

      Increasing bias to 80V
      60V 9.45uA
      70V 9.7uA
      80V 80 9.82uA Still have diode response

10:15 In prep for taking data with beam tuning have tested the merger and tapeserver.
      To enable writing to file had to implement the method in https://elog.ph.ed.ac.uk/DESPEC/36
      Probably a corrupution caused in the difference between the carme system and the aida system.
      Now can write data to file. To limit the data of FEEs 01-04 have set threshold to 0xFF

10:33 Ion source was started this morning but there was no noticable effect on the leakage currents.

10:56 Leakage current is slowly rising continuing the trend at 60V. - Attachment 4
      This could just be the temperature shift during the day or linked to the ion source being powered on.

11:18 Pulser peaks for the FEEs using layout 3 and 4.
      Layout 3 - attachment 5
      Layout 4 - attachment 6
      The pulser peaks appear to be asymetric and in FEE 7 we are observing a double peak.
      Will check the circuit when we next have access.

11:23 Have started writing data to disk in advance of them putting beam into the ring.
      Writing to R4_8 at around 1.8MB/s. N.B. TapeServer shows the same behavior as AIDA with writing nothing to files R4_0->R4_7
      FEE 01-04 slow comp 0xFF
      FEE 05-08 slow comp 0x64
      Pulser running at 50Hz or so

14:50 Having been writing to DAQ since lunch.
      Leakage currents were mostly stable but we are now seeing an increase in leakage currents at a much greater rate - attachment 7

14:58 Attachment8 Updated leakage currents.

15:00 Due to the rapidly increasing bias we asked them to stop the beam.
      Motors were then cycled but we did not notice the leakage coming down.
      Bias was turned off and then motors cycled again.
      Bias was then set to 60V - Run R5
      Bias raised to 80V - R6
      Carlo requested we stop writing data to disk

16:25 Stopped writing to disk. Will retry bias at 80V and monitor.

18:13 Motors cycled before leaving for night no beam in ring - attachment 9
      to test if the increase is beam related we will leave at 80V overnight with the beam blocked all night.
      To avoid the rapidly increasing leakage current the iSet on the CAEN module is at 11uA with a trip time of 2s.