02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at around 3M items/sec
      tape service running at 26MB/sec

All attached to this entry with screenshots to 1-7

02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

All attached to this entry with screenshots to 1-7

04:05 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

All attached tp this entry with screenshots 7-14

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

07:46 all system wide checks ok

      good event statistics ok - attachment 1
  
      FEE64 temperatures ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:48 merger ok ~ 3 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

08:50 AIDA writing to file SecondDrive/TapeData/S480/R12_1524

09:56 all system wide checks ok

      good event statistics ok - attachment 4

      detector bias/leakage currents ok - attachment 5
  
      FEE64 temperatures ok - attachment 6

12:09 all system wide checks ok


12:13 all system wide checks ok

      good event statistics ok - attachment 7
  
      FEE64 temperatures ok - attachment 8

      detector bias/leakage currents ok, but ramping up - attachment 9


13:32 merger ok ~ 3 million data items/s

      TapeServer ok ~ 23 MB/s

      data forwarding to MBS ok

      Beam seems to be off

13:33 AIDA writing to file R12_1743

*update* checked DESPEC S480 Elog, issues with UNILAC, beam lost at 13:29

13:55 beam back

13:57 AIDA writing to file R12_1759

14:35 all system wide checks ok

      FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11
  
      detector bias/leakage currents ok, but ramping up - attachment 12

15:18 merger ok ~ 2.7 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

      AIDA writing to file R12_1283

16:00 1.8.H high energy spectra - attachments 13 & 14

      hit-rate spectra - attachment 15
 
16.22 all system wide checks okay 
      FEE temperatures okay (attachment 16)
      leakage currents okay and recorded to spreadsheet (attachment 17)
      good event stats okay (attachment 18)
      merger running at 3M items/sec
      tape service running at 26MB/sec

18.00 began compressing R7 files with command: nice -n 10 gzip -v R7_* &
      
      all system wide checks okay 
      FEE temperatures okay (attachment 19)
      leakage currents okay and recorded to spreadsheet (attachment 20)
      good event stats okay (attachment 21)
      merger running at 3M items/sec
      tape service running at 27MB/sec    

20.10 all system wide checks okay 
      FEE temperatures okay (attachment 22)
      leakage currents okay and recorded to spreadsheet, detectors 1 and 2 still rising (attachment 23)
      good event stats okay (attachment 24)
      merger running at 3M items/sec
      tape service running at 25MB/sec

      rates histogram (attachment 25)
      low energy histograms (attachment 26 and 27)
      high energy histograms (attachments 28 and 29)

      
22.07 all system wide checks okay 
      FEE temperatures okay (attachment 30)
      leakage currents okay and recorded to spreadsheet (attachment 31)
      good event stats okay (attachment 32)
      merger running at 3M items/sec
      tape service running at 25MB/sec
      currently on R12_2097

23.56 all system wide checks okay 
      FEE temperatures okay (attachment 33)
      leakage currents okay and recorded to spreadsheet (attachment 34)
      good event stats okay (attachment 35)
      merger running at 3M items/sec
      tape service running at 26MB/sec
      
finishing up now. night shift group have been informed how to check up on AIDA, there is a sheet on the table with relevant information
      

AIDA ELog filled out by DESPEC night shift members Matthew Brunet (MB), Shaheen Jazrawi (SJ), and Matthias Rudiger (MR).

0:05-0:08 Beam operators altering settings, beam off/sporadic during this period.

0:46
    All system wide checks passed
    Leak currents lower than previous measurement - attachment #1
    Implants look as previously - attachment #2
    Stats about where they have been - attachment #3
    FEE temperatures fine - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer writing about 25 MB/s

2:46

      All system wide checks passed
     Leak currents similar to previous - attachment #5 
     Implants look as previously - attachment #6
     Stats about where they have been - attachment #7
     FEE temperatures fine - attachment #8
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s
 

4:49

      All system wide checks passed
     Leak currents similar to previous - attachment #9
     Implants look as previously - attachment #10
     Stats good - attachment #11
     FEE temperatures fine - attachment #12
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s

6:49

      All system wide checks passed
     Leak currents no significant change - attachment #13
     Implants look as previously - attachment #14
     Stats good - attachment #15
     FEE temperatures fine - attachment #16
     Merger fine, 3 Mega items / sec
     TapeServer writing about 25 MB/s

8:15
    All system wide checks passed
    Leak currents are continuing to creep up - attachment #1
    FEE temperatures fine - attachment #2
    Stats about where they have been - attachment #3
    Implants look as previously (5ish counts per channel in SSD1 and SSD2) - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer still writing about 25 MB/s

09.37 analysis of data file S480/R12_443
      deadtime aida09 c. 5%, all other FEE64s <<1%
      See attachment #5

09.54.51
      biases, SSD1 and 2 over 2 uA each now -  attachment #6

11:20
    All  system-wide checks okay
    Merger okay, 3 Mega items / sec
    TapeServer normal, 27 MB/sec
    Biases as attachment #7
    Stats as #8
    Temps as #9
    Histos as #10

14:10
    All system wide checks okay
    Stats #11
    Temps #12
    Implants #13
    Leak currents are getting a bit higher #14

15:30
    All system wide checks passed
    Stats attachment  #15
    Temps attachment  #16
    Implants  attachment #17
    Biases / leak currents attachment  #18
    Merger okay, 3 Mega items / sec
    TapeServer okay, writing 27 MB/s

16.12 all online spectra zero'd

16.31 stat & rate spectra - attachments 19-22
      1.8.L spectra - attachments 23 & 24
      aida03 pulser peak 57 ch FWHM
      1.8.H spectra - attachments 25 & 26
      NewMerger & TapeServer - attachments 27 & 28

18.29

     system wide checks all okay
     FEE temperatures normal (attachment 29)
     Leakage currents okay and recorded to spreadsheet (attachment 30)
     Good event statistics okay (attachment 31)
     Merger running at 3M items/sec
     Tape server running at 25MB/sec

20.25
     system wide checks all okay
     FEE temperatures normal (attachment 32)
     Leakage currents okay and recorded to spreadsheet (attachment 33)
     Good event statistics okay (attachment 34)
     Merger running at 3M items/sec
     Tape server running at 26MB/sec

as of 20.39 just finished R12_961

21.36 beam is down confirmed by rates histogram (attachment 35), informed others on shift
      was down for maybe 5 minutes is back on at 21.40 (attachment 36)
      think between R12_1004 to R12_1008

22.10
     system wide checks all okay
     FEE temperatures normal (attachment 37)
     Leakage currents okay and recorded to spreadsheet (attachment 38)
     Good event statistics okay (attachment 39)
     Merger running at 3M items/sec
     Tape server running at 26MB/sec

23.03 someone comes to ask if we see an increase in implantation rate as they do their side
      I checked the rates histogram and good events and see nothing that would indicate an increase (attachment 40 and 41)
      FEE temperatures and leakage currents do not seem any different either
      believe they are calling someone about the issue

23.30 leaving now to catch bus but people on night shift have been told how to check on AIDA and know that CG is on call if needed

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00.00 DAQ momentarily set to NoStorage
      OH changes symbolic link such that AIDA is writing directly to mounted hard-drives ("SecondDrive" and "ThirdDrive")

00.02 DAQ now writing to file again - /media/SecondDrive/TapeData/S480/R12_0

00.30 all system wide checks ok

      FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias and leakage currents ok - attachment 3
 
00:43 merger ok - writing at ~ 3 million items/s

      TapeServer ok - writing at ~ 27 Mb/s

      Data forwarding to MBS ok

02:14 all system wide checks ok

      FEE64 temperatures ok - attachment 4

      good event statistics ok - attachment 5

      detector bias and leakage currents ok - attachment 6

02:17 Merger ok  ~ 2.7 million items/s

      TapeServer ok  ~ 27 Mb/s

      Data forwarding to MBS ok

02:18 currently writing to file /media/SecondDrive/TapeData/S480/R12_110

02:57 attachments 7 & 8 - high energy channel spectra

      attachment 9 - hit rate spectra

04:21  FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11

      detector bias and leakage currents ok - attachment 12

      system wide checks ok - except WR decoder status (see attachment 13)

04:26 check ASIC control

04:30 system functions -> select required function -> collect all WhiteRabbit status error counters for baseline
      WR decoder status check now ok

04:50 Merger ok  ~ 3 million items/s

      TapeServer ok  ~ 25 Mb/s

      Data forwarding to MBS ok

04:52 currently writing to file /media/SecondDrive/TapeData/S480/R12_231
 
06:11 all system wide checks ok

      FEE64 temperatures ok - attachment 14

      good event statistics ok - attachment 15

      detector bias and leakage currents ok - attachment 16


06:41 Merger ok  ~ 3.3 million items/s

      TapeServer ok  ~ 25 Mb/s

      Data forwarding to MBS ok

06:42 currently writing to file /media/SecondDrive/TapeData/S480/R12_315

07:25 hit-rate spectra - attachment 17

07:36 low energy channel spectra (1.8.L) - attachments 18 & 19



 


      

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

16:51 Bias and leakage currents ok but starting to increase in DSSD 1 predominantly - attachment 1
      Statistics ok - attachment 2
      FEE temp ok - attachment 3
      System wide checks all ok 
     
16:57 Histograms zeroed

17:17 ASIC control check done

17:40 Layout 1 - attachment 4
      Layouts 3 and 4 - attachments 4 and 5
      Layouts 5 and 6 - attachments 6 and 7

19:06 Earlier in the shift two extra 8TB drives were formatted and mounted
      Named SecondDrive and ThirdDrive
      TapeData folder created on SecondDrive
      Plan is before the end of this shift to move the symbolic link from the filling up drive to the new drive
      Current prediction is drive would otherwise be full by 12:20 tomorrow.
      Playing it safe and switching early

21:57 Bias and leakage ok - attachment 8
      Statistics ok - attachment 9
      FEE Temps ok - attachment 10

8:00 - See previous shift's elog entry.  All good.

9:00
    All system wide checks passed
    Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
    Tape Server okay - 27 MB/s
    Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
    Stats as usual - attachment #2
    Temps normal - attachment #3
    Biases and leaks fine -  attachment #4

Disk usage projection
    Space remaining: ~2 TB
    Fill rate: ~2 GB / 1 minute
    -> 1000 minutes =~ 17 hours

Still gzipping from R6_823

10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
     	Kbytes Written:  -1986382400	Kbytes/sec:  26816
        Rate is fixed at that value on update.
        However, actual data files are being updated so data has been coming.
        DK and NH consulted and believe the situation is fine.
        We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.

10:25 Now we are on R10 after start and stop the TapeServer.  Problem is resolved.

11:30
      Implants: attachment #5
      Stats: attachment #6
      Biases / leaks: attachment #7
      Temps: attachment #8
      System wide checks performed, situation normal.

Labelled PID plots.  Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.

There was an error in the printouts for Cd isotopes, but the below attachments are corrected.

The Z-depth (color, not nuclear charge) is log scale.

   Shift file starting R9_639

   system wide checks okay
   FEE temperatures okay (attachment 1)
   leakage currents normal and recorded to spreadsheet (attachment 2)
   good event stats are normal (attachment 3)
   merger running at 3.5 M items/sec
   tape running at 26 MB/sec
   
   rates histogram (attachment 4)
   low energy histograms (attachment 5 and 6)
   high energy histograms (attachment 7 and 8)

02.02 
   system wide checks okay
   FEE temperatures okay (attachment 9)
   leakage currents normal and recorded to spreadsheet (attachment 10)
   good event stats are normal (attachment 11)
   merger running at 3.5 M items/sec
   tape running at 27 MB/sec

04.09 
   system wide checks okay
   FEE temperatures okay (attachment 12)
   leakage currents normal and recorded to spreadsheet (attachment 13)
   good event stats are normal (attachment 14)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
   writing to file R9_840

06.07 
   system wide checks okay
   FEE temperatures okay (attachment 15)
   leakage currents normal and recorded to spreadsheet (attachment 16)
   good event stats are normal (attachment 17)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
 
07.59 
   system wide checks okay
   FEE temperatures okay (attachment 18)
   leakage currents normal and recorded to spreadsheet (attachment 19)
   good event stats are normal (attachment 20)
   merger running at 3 M items/sec
   tape running at 26 MB/sec

as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space

16:30
    All system wide checks passed
    Temps shown in attachment #1 
    Histos shown in attachment #2
      NB: Zoomed in, linear scale
    Biases shown in attachment #3
    Stats shown in attachment #4
    Merger rates fine 3 Mega items / sec
    TapeServer looks happy, 25 MB/s

Paddy was asking us to give a rough estimate for the number of implants per spill.
I zoomed in on the rates histogram (attachment #2) and it looks like roughly 5 implants per channel in SSD 1 and SSD2.
So this is 5*128*2=1280.  Note that I don't know the rate window size (e.g., 1 second, 1 spill?)  Probably the window does not catch a full spill.
This might be an over-estimate because an implant in SSD2 may leave a signal in SSD1.
Paddy was getting around 1k or 2k fragments per spill, but to first order it seems consistent.

18:36
   Near R9_372 they slightly opened some FRS slits, as there is a belief we may be blocking some of 94Pd.
   Implant rate seems similar to before, see attachment #5
   Biases did not change more than normal, see attachment #6

20:15
   All system wide checks passed
   Temps shown in attachment #7
   Stats shown in attachment #8
   Biases shown in attchment #9

22:12 
   All system wide checks okay
   Histo #10
   Bias #11
   Stat #12
   Temps #13

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

09:00 Bias and leakage currents ok - attachment 1
      Statistics ok - attachment 2
      FEE temperatures ok - attachment 3
      System wide checks all ok

09:08 No beam, not sure exactly how long it has been down.
      FRS team are working on it

09:28 still no beam. issues with SIS. 
      DAQ set to NoStorage mode
      most recent file R7_522

10.27 beam is back
      writing to file R9_0

10.43 attachment 4 - hit rate spectra

11:09 all system wide checks ok
      FEE64 temps ok - attachment 5
      good event statistics ok - attachment 6
      detector bias and leakage currents - attachment 7

11.18 merger ok, ~ 2.7 million data items per second
      TapeServer ok, ~ 25 Mb/s data rate
      Data forwarding to MBS ok

11.20 writing to file R9_39

11.22 beam off

11.24 beam back online (R9_42)

11.41 FRS has reverted to an older setting

13:04 hit rate spectra - attachment 8

13:19 all system wide checks ok

13:30 FEE64 temps ok - attachment 9

      good event statistics ok - attachment 10

      detector bias and leakage currents ok - attachment 11


15:31 good event statistics ok - attachment 13

      detector bias and leakage currents ok - attachment 14

      FEE64 temps ok - attachment 15

      all system wide checks ok

      beam seems to have stopped during 2pm meeting

15.35 TapeServer ok, data rate ~ 23 Mb/s

      Merger ok, rate of 2.5 - 3 million data items per second

      data forwarding to MBS ok

15.38 AIDA writing to file R9_233

16:02 beam back - AIDA writing to file R9_250







      

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00:44 Implantation range in AIDA - attachment 1

01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
      During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
      the AIDA boxes and begin moving files onto that

01:35 System wide checks all ok
      Bias and leakage current ok - attachment 2
      Stats all ok - attachment 3
      FEE temperatures ok - attachment 4

      TapeServer rate around 25-29MB per second
      Merger rate around 2.6-3.2 million items per second

04:11 System wide checks all ok
      Bias and leakage currents ok - attachment 5
      Stats all ok - attachment 6
      FEE temp ok - attachment 7
06:38 System wide checks all ok
      Bias and leakage currents ok - attachment 8
      Stats all ok - attachment 9
      FEE temp ok - attachment 10

18.25 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - OK

WR timestamp control - attachments 4 & 5

merger & TapeServer OK - attachments 6-8

rate spectra - attachment 9

1.8.L spectra - attachments 10 & 11
  aida03 72 ch FWHM

20:36 DSSSD biases & leakage currents OK - attachment 12
      FEE64 temps OK - attachment 13
      good events statistics OK - attachment 14

      all system wide checks ok

21.50 beam starts back up
      DAQ running, TapeServer now writing to file R7_0
      data rate ~ 25 Mb/s
      data forwarding to MBS ok
      merger ok, data item rate ~ 2.7 million items/sec
      spill time length is now 3 seconds
      1500 count rate per spill, ~ 300 Hz implant rate in AIDA

22.35 FEE64 temps OK - attachment 15
      good event statistics ok - attachment 16
      detector bias/ leakage currents ok - attachment 17

      all system wide checks ok

22.47 attachments 18 & 19 - high energy spectra for all FEE64 (1.8.H)

      attachment 20 - hit rate spectra for all FEE64

23.02 beam down
      on file R7_55

23.11 beam back online
      on file R7_61

23.15 system wide checks all ok

      detector bias/ leakage currents ok - attachment 21

      good event statistics ok - attachment 22

      FEE64 temperatures ok - attachment 23      

17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1

17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2

17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3

17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4

00:15 Shift change near AIDA R6_475

  We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.

  All system-wide checks passed successfully
  Temperatures ok (aida01 still a bit hot) - see attachment #1
  Stats as normal (SSD3 still noisy) - see attachment #2
  Bias and pulser settings remain sane - see attachment #3
  Writing data to disk at ~26 MB/s as yesterday - see attachment #4
  Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
  Rates look not unreasonable for expected implantation in short time window - see attachment #6

2:26
   We are near R6_574
   All system checks are fine
   Temps are fine - attachment #7
   Stats look as before - attachment #8
   Biases and leaks typical - attachment #9
   Merger fine, 3 x 10^6 items / sec (skip screencap)
   TapeServer still sits near 25 MB/s (skip screencap)
   Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10

2:44 R6_587 or R6_588
   The spills seem to have stopped based on the beeping scalars.  Confirmed with other group there are no scint counts.
   See attachment of AIDA rates where no implants can be seen.  See attachment #11

2:46 Spill beeps heard briefly, but they cut again.  R6_590

2:49 Beam seems to be back R6_592
  Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants.  No problem.  See attachment #12 for implants.

4:35 
  System checks all pass (except expected error on master clock)
  Temps okay - attachment #13
  Biases and leak currents normal - attachment #14
  Stats as before - attachment #15
  Merger going, 3 mega items per sec
  TapeServer going, 25 MB/s

5:59 Beam is gone again, near R6_738.  As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
   See attachment #16 which shows no implants in the rate.


   

Contact phones for Control Room: 2222

We called, they are aware of a problem with the magnets and they are working to resolve it.  So we will await them.

Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.  

We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off

7:14  Beam is still off, but I make some notes
   System wide checks all okay
   Temps as normal (see attachment #17)
   Biases and leak currents okay (see attachment #18)
   Stats as normal (see attachment #19)
   Merger going at 3 mega cps
   TapeServer on, around 25 MB/sec
   Implants not seen in histogram rate as expected (see attachment #20)

7:32 We called the operators to confirm that they don't change anything before calling us first.
     Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.

8.00 shift change at file R6_823
     beam is back on, no settings changed.
     particles per spill is similar to before (2k~3k per spill)
     implants now being seen in rate histogram (attachment 21)

8.32 system wide checks all okay
     FEE temperatures are normal (attachment 22)
     leak currents are okay and have been recorded to sheet (attachment 23)
     good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
     merger rate is running around 3M items/sec
     tape service rate is running around 26MB/sec

9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
     problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
     remaking each histogram and saving layout6 seems to be fixed (attachment 27)

Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
    invoked from within
"SpecDetails 1.8.h"
    (in namespace inscope "::urn:SpectrumService" script line 1)
    invoked from within
====

10.05 beam is being stopped to change some thresholds

10.34 system wide checks all okay
      FEE temperatures are normal (attachment 28)
      leak currents are okay and have been recorded to spreadsheet (attachment 29)
      good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24MB/sec
      implants not seen in histogram rate as expected as beam still down (see attachment 30)

10.42 beam still down but taking data, currently file finishing R6_946
      as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
      have zeroed all histograms so will take screenshots at 11.00

11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks

11.25 beam back file S480/R6_975

12.00 analysis file S480/R6_1001 - attachment 33
      FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
      all HEC data rate c. 1kHz
      all ADC data rate c. 1.2MHz

12.23 system wide checks all okay
      FEE temperatures are normal (attachment 34)
      leak currents are okay and have been recorded to spreadsheet (attachment 35)
      good event stats are okay (attachment 36)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 26MB/sec

13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
      seems to be intermittent. file R6_1051
      
13.10 beam back on, no one was called. currently on file R6_1061

13.22 beam off as going to higher intensity beam. file R6_1069

14.42 system wide checks all okay
      FEE temperatures are normal (attachment 38)
      leak currents are okay and have been recorded to spreadsheet (attachment 39)
      good event stats are okay (attachment 40)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24M/sec
14.48 on file R6_1131

16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
      file R6_1193

16:35 compression of files R6_[200-999] commenced (nice -20)

16:40 system wide checks all okay
      FEE temperatures are normal (attachment 41)
      leak currents are okay and have been recorded to spreadsheet (attachment 42)
      good event stats are okay (attachment 43)

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


Every 4h

1) Desktop 1 - 1x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 2 = DSSSD # 3

   Select the terminal and obtain a screenshot (Fn-Alt-PrtScn) of DSSSD #1 = #3 detector
   bias voltages and leakage currents
   Upload screenshots to Elog (elog.ph.ed.ac.uk/DESPEC)
   Update leakage current spreadsheet (https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023)

   If S4 ambient temperature information is available please add this information to the Elog

   Compare DSSSD leakage currents (and temperatures) with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select 'Reload'

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C


   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0

   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows

   Desktop 3 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off'

   and *now* seek expert help


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H W X' tags

4) Desktop 1 - Firefox web browser - Statistics

   Check 'good event' statistics - screenshot to Elog - compare with previous Elog entries

5) Desktop 1 - Firefox web browser - System Wide Checks

   Check all system wide checks - compare results with porevious Elog entries

6) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~2-3M data items/s) consistent with value observed at start of run

   Observe active spinners in NewMerger and TapeServer server terminal windows adjacent to web browser

   Chweck MBS terminal data rate is around 20000kB/s Bottom right terminal on desktop 2

7) Desktop 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check data rate consistent with value observed at start of run

A range scan was performed using AIDA.
See the GSI elog: https://elog.gsi.de/despec/S480/19

c 09.00 aida03 drops Histogram enable at DAQ GO
        NH determined this was due to Options DB corruption and fixed

09.55 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23 (different from the December ones yesterday which experienced corruption)
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - attachments 4-11
    aida06, 08, 10, 12 re-calibrated OK, all others OK
    aida03 failed WR decoder status - collect all WR errors for baseline - all FEE64s pass WR decoder status
    sync all ASIC clocks using ReSync

merger & TapeServer OK - attachments 12-14

13.29 DAQ to disk file S480/R6

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa
     aida01 LEC/MEC fast comparator 0x64, all other FEE64s 0xff

14.57 fast comparator thresholds LEC/MEC changed to 0xff for all FEEs

15.01 FEE temperatures normal, aida1 virtex temperature still slightly over 65 degrees (attachment 15)
      leakage currents recorded to spreadsheet (attachment 16)
      good events rate similar to previous, high in DSSD3 and seems to be returning to normal in DSSD1 except in one FEE (attachments 17 and 18)
      system wide checks OK
      data rate around 22000 KB per second (attachment 19)
as of 15.12 on file R6_80

attachment 20 is total event stats shows events are being implanted
attachment 21 and 22 show low energy spectrum left most peak is pulser and rightmost is central noise. middle peak in aida1
attachment 23 and 24 show high energy spectrum, aida4 histogram not being produced unsure why.
attachment 25 and 26 show waveforms, only aida3 histogram working

16.00 checked ASIC control, all OK

16.35 spill is now changed to 10s with beam intensity increase by factor of 10. 3000 implants per spill.

16.42 leakage currents recorded to spreadsheet

17.10 system wide checks ok
      fee64 temperatures ok - attachment 27
      detector bias/leakage currents ok - attachment 28
      good event statistics ok - attachment 29

17.15 moved slit at S2 - rate drop to try remove some 96Pd

17.20 writing to file R6_166

17.44 merger/tapeserver ok. Data rate ~ 25000 kb/s

18:10 Fine tuning implantation profile
      Removing 0.25mm Al degrader (60 mg/cm^2) to get 96Pd slightly further in to AIDA.  
      DESPEC is closing the current file, R6_203 was just finished on AIDA on typing this.
      (Maybe the end  of run S480f004_ on DESPEC side but that elog was not updated since 16:40?)
      Decision based on assumption of 6 GeV 96Pd having a range 1 mm in silicon.

19.16 system wide checks ok
      good event statistics ok - attachment 30
      detector bias/leakage currents ok - attachment 31
      fee64 temperatures ok - attachment 32
      
19.39 TapeServer/Merger ok. Writing to file R6_265. Data rate ~ 25000 kb/s

20.07 1.8.H high energy spectra front and back side (attachments 33 + 34)

20.36 S4 timing change, increased by factor of 2

20.38 Merger/TapeServer ok. Data being forwarded to MBS.

20.38 writing to file R6_308 at rate of ~ 25000 kb/s

21:21 system wide checks ok
      good event statistics ok - attachment 35
      fee64 temperatures ok - attachment 36
      detector bias/leakage currents ok - attachment 37

21:57 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_368 at rate of ~26000 kb/s

23:10 system wide checks ok
      good event statistics ok - attachment 38
      fee64 temperatures ok - attachment 39
      detector bias/leakage currents ok - attachment 40

23:24 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_435 at rate of 27000 kb/s

23:42 rate spectra - attachment 41











 

14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?

15:06 Systems checks fine.
      Master clock failed (no master clock)
      all checks passed
      memory checks all around 38k as normal

      Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees

      Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k

      Attachment 4: leakage currents seem normal at expected values

15:16 Merger working for all FEEs
      tape server is on but no storage
      data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)

ASIC controls checked and are all OK

15.48 low energy pulsar peak FWHM

      fee   FHWM
      1     89.08
      2     132.06
      3     70.73
      4     81.84
      5     62.86
      6     79.92
      7     122.13
      8     73.31
      9     198.39
      10    115.53
      11    100.55
      12    144.04

ALL FEEs in downstream SSD are high

16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)

16:25 included rates, note not to common scale in y (attachment 9)

ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
  Did not seem completely reproducible?
  We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad

Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)


19:15 Update

There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.  
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems

One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space 
  This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)

Now we have brought back up the AIDA systems and should run through the checks

Most system checks look good (except some calibration errors)

Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool

Bias and leak currents attached as #10

Temps as #11

Stats as #12

19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.

21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
      A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
      Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET

21:51 AIDA running to no storage again