Labelled PID plots.  Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.

There was an error in the printouts for Cd isotopes, but the below attachments are corrected.

The Z-depth (color, not nuclear charge) is log scale.

   Shift file starting R9_639

   system wide checks okay
   FEE temperatures okay (attachment 1)
   leakage currents normal and recorded to spreadsheet (attachment 2)
   good event stats are normal (attachment 3)
   merger running at 3.5 M items/sec
   tape running at 26 MB/sec
   
   rates histogram (attachment 4)
   low energy histograms (attachment 5 and 6)
   high energy histograms (attachment 7 and 8)

02.02 
   system wide checks okay
   FEE temperatures okay (attachment 9)
   leakage currents normal and recorded to spreadsheet (attachment 10)
   good event stats are normal (attachment 11)
   merger running at 3.5 M items/sec
   tape running at 27 MB/sec

04.09 
   system wide checks okay
   FEE temperatures okay (attachment 12)
   leakage currents normal and recorded to spreadsheet (attachment 13)
   good event stats are normal (attachment 14)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
   writing to file R9_840

06.07 
   system wide checks okay
   FEE temperatures okay (attachment 15)
   leakage currents normal and recorded to spreadsheet (attachment 16)
   good event stats are normal (attachment 17)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
 
07.59 
   system wide checks okay
   FEE temperatures okay (attachment 18)
   leakage currents normal and recorded to spreadsheet (attachment 19)
   good event stats are normal (attachment 20)
   merger running at 3 M items/sec
   tape running at 26 MB/sec

as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space

16:30
    All system wide checks passed
    Temps shown in attachment #1 
    Histos shown in attachment #2
      NB: Zoomed in, linear scale
    Biases shown in attachment #3
    Stats shown in attachment #4
    Merger rates fine 3 Mega items / sec
    TapeServer looks happy, 25 MB/s

Paddy was asking us to give a rough estimate for the number of implants per spill.
I zoomed in on the rates histogram (attachment #2) and it looks like roughly 5 implants per channel in SSD 1 and SSD2.
So this is 5*128*2=1280.  Note that I don't know the rate window size (e.g., 1 second, 1 spill?)  Probably the window does not catch a full spill.
This might be an over-estimate because an implant in SSD2 may leave a signal in SSD1.
Paddy was getting around 1k or 2k fragments per spill, but to first order it seems consistent.

18:36
   Near R9_372 they slightly opened some FRS slits, as there is a belief we may be blocking some of 94Pd.
   Implant rate seems similar to before, see attachment #5
   Biases did not change more than normal, see attachment #6

20:15
   All system wide checks passed
   Temps shown in attachment #7
   Stats shown in attachment #8
   Biases shown in attchment #9

22:12 
   All system wide checks okay
   Histo #10
   Bias #11
   Stat #12
   Temps #13

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

09:00 Bias and leakage currents ok - attachment 1
      Statistics ok - attachment 2
      FEE temperatures ok - attachment 3
      System wide checks all ok

09:08 No beam, not sure exactly how long it has been down.
      FRS team are working on it

09:28 still no beam. issues with SIS. 
      DAQ set to NoStorage mode
      most recent file R7_522

10.27 beam is back
      writing to file R9_0

10.43 attachment 4 - hit rate spectra

11:09 all system wide checks ok
      FEE64 temps ok - attachment 5
      good event statistics ok - attachment 6
      detector bias and leakage currents - attachment 7

11.18 merger ok, ~ 2.7 million data items per second
      TapeServer ok, ~ 25 Mb/s data rate
      Data forwarding to MBS ok

11.20 writing to file R9_39

11.22 beam off

11.24 beam back online (R9_42)

11.41 FRS has reverted to an older setting

13:04 hit rate spectra - attachment 8

13:19 all system wide checks ok

13:30 FEE64 temps ok - attachment 9

      good event statistics ok - attachment 10

      detector bias and leakage currents ok - attachment 11


15:31 good event statistics ok - attachment 13

      detector bias and leakage currents ok - attachment 14

      FEE64 temps ok - attachment 15

      all system wide checks ok

      beam seems to have stopped during 2pm meeting

15.35 TapeServer ok, data rate ~ 23 Mb/s

      Merger ok, rate of 2.5 - 3 million data items per second

      data forwarding to MBS ok

15.38 AIDA writing to file R9_233

16:02 beam back - AIDA writing to file R9_250







      

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00:44 Implantation range in AIDA - attachment 1

01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
      During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
      the AIDA boxes and begin moving files onto that

01:35 System wide checks all ok
      Bias and leakage current ok - attachment 2
      Stats all ok - attachment 3
      FEE temperatures ok - attachment 4

      TapeServer rate around 25-29MB per second
      Merger rate around 2.6-3.2 million items per second

04:11 System wide checks all ok
      Bias and leakage currents ok - attachment 5
      Stats all ok - attachment 6
      FEE temp ok - attachment 7
06:38 System wide checks all ok
      Bias and leakage currents ok - attachment 8
      Stats all ok - attachment 9
      FEE temp ok - attachment 10

18.25 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - OK

WR timestamp control - attachments 4 & 5

merger & TapeServer OK - attachments 6-8

rate spectra - attachment 9

1.8.L spectra - attachments 10 & 11
  aida03 72 ch FWHM

20:36 DSSSD biases & leakage currents OK - attachment 12
      FEE64 temps OK - attachment 13
      good events statistics OK - attachment 14

      all system wide checks ok

21.50 beam starts back up
      DAQ running, TapeServer now writing to file R7_0
      data rate ~ 25 Mb/s
      data forwarding to MBS ok
      merger ok, data item rate ~ 2.7 million items/sec
      spill time length is now 3 seconds
      1500 count rate per spill, ~ 300 Hz implant rate in AIDA

22.35 FEE64 temps OK - attachment 15
      good event statistics ok - attachment 16
      detector bias/ leakage currents ok - attachment 17

      all system wide checks ok

22.47 attachments 18 & 19 - high energy spectra for all FEE64 (1.8.H)

      attachment 20 - hit rate spectra for all FEE64

23.02 beam down
      on file R7_55

23.11 beam back online
      on file R7_61

23.15 system wide checks all ok

      detector bias/ leakage currents ok - attachment 21

      good event statistics ok - attachment 22

      FEE64 temperatures ok - attachment 23      

17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1

17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2

17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3

17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4

00:15 Shift change near AIDA R6_475

  We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.

  All system-wide checks passed successfully
  Temperatures ok (aida01 still a bit hot) - see attachment #1
  Stats as normal (SSD3 still noisy) - see attachment #2
  Bias and pulser settings remain sane - see attachment #3
  Writing data to disk at ~26 MB/s as yesterday - see attachment #4
  Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
  Rates look not unreasonable for expected implantation in short time window - see attachment #6

2:26
   We are near R6_574
   All system checks are fine
   Temps are fine - attachment #7
   Stats look as before - attachment #8
   Biases and leaks typical - attachment #9
   Merger fine, 3 x 10^6 items / sec (skip screencap)
   TapeServer still sits near 25 MB/s (skip screencap)
   Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10

2:44 R6_587 or R6_588
   The spills seem to have stopped based on the beeping scalars.  Confirmed with other group there are no scint counts.
   See attachment of AIDA rates where no implants can be seen.  See attachment #11

2:46 Spill beeps heard briefly, but they cut again.  R6_590

2:49 Beam seems to be back R6_592
  Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants.  No problem.  See attachment #12 for implants.

4:35 
  System checks all pass (except expected error on master clock)
  Temps okay - attachment #13
  Biases and leak currents normal - attachment #14
  Stats as before - attachment #15
  Merger going, 3 mega items per sec
  TapeServer going, 25 MB/s

5:59 Beam is gone again, near R6_738.  As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
   See attachment #16 which shows no implants in the rate.


   

Contact phones for Control Room: 2222

We called, they are aware of a problem with the magnets and they are working to resolve it.  So we will await them.

Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.  

We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off

7:14  Beam is still off, but I make some notes
   System wide checks all okay
   Temps as normal (see attachment #17)
   Biases and leak currents okay (see attachment #18)
   Stats as normal (see attachment #19)
   Merger going at 3 mega cps
   TapeServer on, around 25 MB/sec
   Implants not seen in histogram rate as expected (see attachment #20)

7:32 We called the operators to confirm that they don't change anything before calling us first.
     Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.

8.00 shift change at file R6_823
     beam is back on, no settings changed.
     particles per spill is similar to before (2k~3k per spill)
     implants now being seen in rate histogram (attachment 21)

8.32 system wide checks all okay
     FEE temperatures are normal (attachment 22)
     leak currents are okay and have been recorded to sheet (attachment 23)
     good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
     merger rate is running around 3M items/sec
     tape service rate is running around 26MB/sec

9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
     problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
     remaking each histogram and saving layout6 seems to be fixed (attachment 27)

Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
    invoked from within
"SpecDetails 1.8.h"
    (in namespace inscope "::urn:SpectrumService" script line 1)
    invoked from within
====

10.05 beam is being stopped to change some thresholds

10.34 system wide checks all okay
      FEE temperatures are normal (attachment 28)
      leak currents are okay and have been recorded to spreadsheet (attachment 29)
      good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24MB/sec
      implants not seen in histogram rate as expected as beam still down (see attachment 30)

10.42 beam still down but taking data, currently file finishing R6_946
      as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
      have zeroed all histograms so will take screenshots at 11.00

11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks

11.25 beam back file S480/R6_975

12.00 analysis file S480/R6_1001 - attachment 33
      FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
      all HEC data rate c. 1kHz
      all ADC data rate c. 1.2MHz

12.23 system wide checks all okay
      FEE temperatures are normal (attachment 34)
      leak currents are okay and have been recorded to spreadsheet (attachment 35)
      good event stats are okay (attachment 36)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 26MB/sec

13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
      seems to be intermittent. file R6_1051
      
13.10 beam back on, no one was called. currently on file R6_1061

13.22 beam off as going to higher intensity beam. file R6_1069

14.42 system wide checks all okay
      FEE temperatures are normal (attachment 38)
      leak currents are okay and have been recorded to spreadsheet (attachment 39)
      good event stats are okay (attachment 40)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24M/sec
14.48 on file R6_1131

16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
      file R6_1193

16:35 compression of files R6_[200-999] commenced (nice -20)

16:40 system wide checks all okay
      FEE temperatures are normal (attachment 41)
      leak currents are okay and have been recorded to spreadsheet (attachment 42)
      good event stats are okay (attachment 43)

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


Every 4h

1) Desktop 1 - 1x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 2 = DSSSD # 3

   Select the terminal and obtain a screenshot (Fn-Alt-PrtScn) of DSSSD #1 = #3 detector
   bias voltages and leakage currents
   Upload screenshots to Elog (elog.ph.ed.ac.uk/DESPEC)
   Update leakage current spreadsheet (https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023)

   If S4 ambient temperature information is available please add this information to the Elog

   Compare DSSSD leakage currents (and temperatures) with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select 'Reload'

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C


   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0

   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows

   Desktop 3 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off'

   and *now* seek expert help


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H W X' tags

4) Desktop 1 - Firefox web browser - Statistics

   Check 'good event' statistics - screenshot to Elog - compare with previous Elog entries

5) Desktop 1 - Firefox web browser - System Wide Checks

   Check all system wide checks - compare results with porevious Elog entries

6) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~2-3M data items/s) consistent with value observed at start of run

   Observe active spinners in NewMerger and TapeServer server terminal windows adjacent to web browser

   Chweck MBS terminal data rate is around 20000kB/s Bottom right terminal on desktop 2

7) Desktop 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check data rate consistent with value observed at start of run

A range scan was performed using AIDA.
See the GSI elog: https://elog.gsi.de/despec/S480/19

c 09.00 aida03 drops Histogram enable at DAQ GO
        NH determined this was due to Options DB corruption and fixed

09.55 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23 (different from the December ones yesterday which experienced corruption)
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - attachments 4-11
    aida06, 08, 10, 12 re-calibrated OK, all others OK
    aida03 failed WR decoder status - collect all WR errors for baseline - all FEE64s pass WR decoder status
    sync all ASIC clocks using ReSync

merger & TapeServer OK - attachments 12-14

13.29 DAQ to disk file S480/R6

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa
     aida01 LEC/MEC fast comparator 0x64, all other FEE64s 0xff

14.57 fast comparator thresholds LEC/MEC changed to 0xff for all FEEs

15.01 FEE temperatures normal, aida1 virtex temperature still slightly over 65 degrees (attachment 15)
      leakage currents recorded to spreadsheet (attachment 16)
      good events rate similar to previous, high in DSSD3 and seems to be returning to normal in DSSD1 except in one FEE (attachments 17 and 18)
      system wide checks OK
      data rate around 22000 KB per second (attachment 19)
as of 15.12 on file R6_80

attachment 20 is total event stats shows events are being implanted
attachment 21 and 22 show low energy spectrum left most peak is pulser and rightmost is central noise. middle peak in aida1
attachment 23 and 24 show high energy spectrum, aida4 histogram not being produced unsure why.
attachment 25 and 26 show waveforms, only aida3 histogram working

16.00 checked ASIC control, all OK

16.35 spill is now changed to 10s with beam intensity increase by factor of 10. 3000 implants per spill.

16.42 leakage currents recorded to spreadsheet

17.10 system wide checks ok
      fee64 temperatures ok - attachment 27
      detector bias/leakage currents ok - attachment 28
      good event statistics ok - attachment 29

17.15 moved slit at S2 - rate drop to try remove some 96Pd

17.20 writing to file R6_166

17.44 merger/tapeserver ok. Data rate ~ 25000 kb/s

18:10 Fine tuning implantation profile
      Removing 0.25mm Al degrader (60 mg/cm^2) to get 96Pd slightly further in to AIDA.  
      DESPEC is closing the current file, R6_203 was just finished on AIDA on typing this.
      (Maybe the end  of run S480f004_ on DESPEC side but that elog was not updated since 16:40?)
      Decision based on assumption of 6 GeV 96Pd having a range 1 mm in silicon.

19.16 system wide checks ok
      good event statistics ok - attachment 30
      detector bias/leakage currents ok - attachment 31
      fee64 temperatures ok - attachment 32
      
19.39 TapeServer/Merger ok. Writing to file R6_265. Data rate ~ 25000 kb/s

20.07 1.8.H high energy spectra front and back side (attachments 33 + 34)

20.36 S4 timing change, increased by factor of 2

20.38 Merger/TapeServer ok. Data being forwarded to MBS.

20.38 writing to file R6_308 at rate of ~ 25000 kb/s

21:21 system wide checks ok
      good event statistics ok - attachment 35
      fee64 temperatures ok - attachment 36
      detector bias/leakage currents ok - attachment 37

21:57 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_368 at rate of ~26000 kb/s

23:10 system wide checks ok
      good event statistics ok - attachment 38
      fee64 temperatures ok - attachment 39
      detector bias/leakage currents ok - attachment 40

23:24 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_435 at rate of 27000 kb/s

23:42 rate spectra - attachment 41











 

14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?

15:06 Systems checks fine.
      Master clock failed (no master clock)
      all checks passed
      memory checks all around 38k as normal

      Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees

      Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k

      Attachment 4: leakage currents seem normal at expected values

15:16 Merger working for all FEEs
      tape server is on but no storage
      data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)

ASIC controls checked and are all OK

15.48 low energy pulsar peak FWHM

      fee   FHWM
      1     89.08
      2     132.06
      3     70.73
      4     81.84
      5     62.86
      6     79.92
      7     122.13
      8     73.31
      9     198.39
      10    115.53
      11    100.55
      12    144.04

ALL FEEs in downstream SSD are high

16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)

16:25 included rates, note not to common scale in y (attachment 9)

ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
  Did not seem completely reproducible?
  We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad

Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)


19:15 Update

There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.  
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems

One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space 
  This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)

Now we have brought back up the AIDA systems and should run through the checks

Most system checks look good (except some calibration errors)

Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool

Bias and leak currents attached as #10

Temps as #11

Stats as #12

19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.

21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
      A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
      Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET

21:51 AIDA running to no storage again

Noted using my AIDA event builder code, a timewarp was reported and I noticed the ADC event had a different timestamp LSB to the "WR" markers preceding it, which I didn't think was possible in the merger as it stands

 aida event 875015fa 0f68098c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68098c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68098c
 aida event c1f6851b 0f68098c | ADC 08:54 L 851b
 event time: 15fa4b164f68098c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1ef8238 0f68115c | ADC 08:47 L 8238
 event time: 15fa4b164f68115c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1f884b0 0f68115c | ADC 08:56 L 84b0
 event time: 15fa4b164f68115c
 aida event 8a5015fa 0f681382 | INFO 5 SYNC6348 11 015fa
 aida event 8a44b164 0f681382 | INFO 4 SYNC4828 11 4b164
 event time: 15fa4b164f681382
 aida event c2817dc8 0f681382 | ADC 11:01 L 7dc8
 event time: 15fa4b164f681382
 aida event 875015fa 0f68192c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68192c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68192c
 aida event c1f9849c 0f68192c | ADC 08:57 L 849c
 event time: 15fa4b164f68192c
 aida event 815015fa 0f681f94 | INFO 5 SYNC6348 02 015fa
 aida event 8144b164 0f681f94 | INFO 4 SYNC4828 02 4b164
 event time: 15fa4b164f681f94
 aida event c0568084 04b6854a | ADC 02:22 L 8084
 event time: 15fa4b1644b6854a
AIDA Timewarp (15fa4b1644b6854a before 15fa4b164f68192c)
Event reported errors, skipping this file...
Events: 506459240   506459240             (0 errors)

Unsure if the cause, easily handled by ignoring such events but should be "impossible"

Circa 700 GB of data

A few channels in DSSD 1 missing? Otherwise seems OK

Looking at the HV bias in a scope to see if it is the source of noise especially on the p+n side

Channel 1: HV Core channel 4
Channel 2: HV Braid Channel 4
Math: Ch1 - Ch2

100V bias (100 uA leakage current for 1 MOhm)

No obvious frequency components or large frequency components?
I do not know how this compares to RIKEN or STFC-DL

Updated to put the right frequency range (misread before)

Fig 3-4: Normalish looking data 

Fig 5-6: Examples of noise appearing randomly on math line

Fig 7: 1ms long trace, noise stopped appearing

Update 26.02.2020

Fig 8: Trace with all 4 HV channels off

Fig 9: Trace with only oscilloscope biased

Fig 10: Trace with braids unplugged from all 3 FEE64s (only channel 4 connected)

Things to try:

Joining clean earth & chassis ground?

Checking HV jumper?

Finding 'scope probes and testing NIM socket?

Attachment 1 - layout 2 - stat spectra
Attachment 2 - detector biases & leakage currents - OK
Attachment 3 - FEE64 temperatures - OK
Attachments 4-7 - TapeServer, Merger, Merger Statistics, good event statistics - OK

10:40 ASIC check load x2


13.12 DAQ stopped c. 10.50
      unable to connect messages for multiple FEE64s

      After lunch powwer-cycled FEE64s and restarted DAQ
      aida03 crashed (see https://elog.ph.ed.ac.uk/DESPEC/131 ) and rebooted by issuing reboot command as root

      Data file 20FEB20/R1
      ASIC settings 2019Oct31-13.24.23
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      continue alpha background run

Overnight alpha data

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Feb 20 17:46:47 2020
Last event : Fri Feb 21 10:07:57 2020
Duration   : 58869.9 seconds
--------------------------------------
MBS Events   (M):     1248648 (21 Hz)
AIDA Words   (W): 10222680133 (27 kHz)
ADC Words    (A):  3407431088 (-15076 Hz)
-- Decays    (D):  3407017078 (-15083 Hz)
-- Implants  (I):      414010 (7 Hz)
Info Words   (F):  6815249045 (-30146 Hz)
-- SYNC48    (S):  3407559635 (-15074 Hz)
-- SYNC63    (s):  3407559635 (-15074 Hz)
-- PAUSE     (P):         614 (0 Hz)
-- RESUME    (R):         614 (0 Hz)
-- Discrim   (D):      128547 (2 Hz)
-- Unknown   (U):           0 (0 Hz)
Timewarps    (T):          79 (0 Hz)
Dead Time    (X): 26.8658 s (0.04563588%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |           W           A           D           I           F           S           s     P     R       D     U     T  X
   1 |        5025        1246        1093         153        3779        1675        1675     0     0     429     0     0  0
   2 |        6180         991         851         140        5189        2060        2060     0     0    1069     0     0  0
   3 |  7692461404  2564028610  2564028286         324  5128432794  2564153392  2564153392   614   614 	124782     0    58 26.8658
   4 |        2277         642         581          61        1635         759         759     0     0     117     0     0  0
   5 |        6174        1249        1010         239        4925        2058        2058     0     0     809     0     0  0
   6 |     2467062      822244      822191          53     1644818      822354      822354     0     0     110     0     0  0
   7 |        2484         740         701          39        1744         828         828     0     0      88     0     0  0
   8 |   358445004   119481621   119069026      412595   238963383   119481668   119481668     0     0      47     0     1  0
   9 |        7134        1530        1243         287        5604        2378        2378     0     0     848     0     0  0
  10 |  2168031885   722677295   722677295           0  1445354590   722677295   722677295     0     0       0     0    20  0
  11 |        3981        1252        1190          62        2729        1327        1327     0     0      75     0     0  0
  12 |     1241523      413668      413611          57      827855      413841      413841     0     0     173     0     0  0
**** DESPECAnalysis: PostLoop

A few FEEs running "hot", maybe check. 3 had some alignment issues may need rechecking.
Timewarps somehow in aida03 and aida10 but may be code mistake - must check.
Only aida03 ran fast enough to have dead time.

aida10 shows decay data now - DSSD seems ok

Fig 1: DSSD 1D hit pattern 

Fig 2: DSSD Energy

Fig 3: Single channel in aida03 that is hot

Fig 4: Channels 48-64 in aida08 are hot (wiring?)

Fig 5: Channels 17, 19 (crosstalk?) and 27, 33, 38, 39

All otehr channels quiet and OK. 

DSSD#3 (most downstream) was replaced

Noted that old DSSD wafer was not sitting properly - detached - in PCB, PCB flexing and/or inadequate epoxy gluing

New DSSD#3 - 2998-22 - 1006 um - 100 V - ~4 uA

Old Detector removed was 3208-14

Fig 1. HV Control

Fig 2. Temps

Fig 3. Stats

Fig 4. Rates

Fig 5-6: Waves

Fig 7-8: Pulser Peaks

Start alpha run at 17:48, 20FEB20, R1
 Slow comparator 0x64
 Pulser off

Fig 9 : Alpha stats

Fig 10: Merger stats

Fig 11: Merger main page

Fig 12: Timestamp errors in merger console

> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
> 
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
> 
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>  
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> 
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5 
> 
> Attach 5: Timestamp shown by merger

Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of 
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a 
logical "rollover" of the lower 14 bits of this 10nS timestamp.


Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?


It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.

The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be 
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.

If this occurs again then the system wide check results would be interesting.