08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)

This was because the statistics page had been changed to transfer buffers. The rates were ok

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).

This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

All seems ok  (apart for my virgin network connection usually quite reliable but not so tonight !?!).

Statistics seems ok.

Temperatures appears stable

Leakage current ok.

System Wide Checks:

Clock check: Passed= 14: - Failed=: 0

ADC calibration check:

FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES

White rabbit decode status: Passed=14 ; Failed=0

FPGA timestamp check: Passed=14 ; Failed=0

Memory information: Page loading failed with the error message below:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"

The rate is very low in all aida, not sure .

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida*

Attachment 1 - top aida06

Attachment 2 - top aida01

Attachment 3 - top aida-gsi (AIDA server)

Attachment 4 - TapeServer

Attachment 5 - NewMerger stats block 6 (aida06)

Attachment 6 - NewMerger stats block 1 (aida01)

Attachment 7 - NewMerger stats block 0 (all FEE64s)

Attachment 8 - NewMerger options

Attachment 9 - NewMerger link stats

Attachment 10 - NewMerger

Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats

Attacgment 15 - Network traffic

08.10 AIDA DAQ status
    
      14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
      AD9222 OFF - waveforms disabed
      disc - all disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23

      BNC PB-5
      amplitude 1.000V
      attenuation x1
      tau_d 1ms (tail pulse)
      polarity +
      frequency 22Hz

08.14 all histograms zero'd
2
08.15 system wide check counter baseline

08.17 check ASIC control - all FEE64s all ASICs

Attachment 1 - grafana DSSSD bias & leakage current - OK

Attachment 2 - per FEE64 stat spectra

Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips

Attachments 5 & 6 - per FEE64 1.8.L spectra
 pulser peak width aida01 124ch FWHM, aida04 293ch FWHM

Attachment 7 - per FEE64 rate spectra

Attachments 8-12 - system wide checks - all OK

Attachments 13-16 - statistics - adc, pause, resume & correlation data

Attachment 17 - FEE64 temps OK

Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends

Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages

08.44 DAQ continues - file NOTAPE/R18 - no storage mode

08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc

08.53 no storage mode enabled

Attachment 21 - analysis of file NOTAPE/R18_20

Attachment 22- analysis of file NOTAPE/R19_19
               AIDA12 slow comp up to 0xd improved deadtime drastically for FEE

10:54 Beam on detector. Take another run to test deadtime - attachment 23
      Deadtimes ranging from 10-40% in n+n FEEs

Attachment 23 & 24 aida06 aida02 1.*.H spectra


14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
        Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime)

01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
      That is now complete.
      Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc

      R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
      Will lower 2 and 4 to 200 and raise 6 and 8 to 300

      R17 2 good, 4 shakey 6 and 8 poor - attachment 2

      R18 2 and 4 at 220 and 6 and 7 at 350
      2 and 8 low deadtime, 4 and 6 could be better. - attachment 3


      Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
      Statistics at this rate - attachment 4




02:18 Statistics - attachment 7
      Temperatures - attachment 6
      Bias & leakage current - attachment 5

      Clock Status - OK
      White Rabbit - OK
      The other 3 checks currently aren't working




04:11 Statistics - attachment 10
      Temperatures - attachment 9
      Bias & leakage current - attachment 8

      Clock Status - OK
      White Rabbit - OK




06:05 Statistics - attachment 13
      Temperatures - attachment 12
      Bias & leakage current - attachment 11

      Clock Status - OK
      White Rabbit - OK

20:19 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

22:12 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
      It is currently sending data pause requests and data push requests but nothing else - attachment 1
      telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
      FEE appears to have hung. - attachment 2
      Still passes all system wide checks.
      When performing a check load can see the tclsh process take up CPU usage

09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
      AIDA14 recovered but ASIC2 still missing
    
      Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
      Given access would change the adapter board on AIDA05.
      Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
      Would retain the option of a speedybacktrack

      Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
      A threshold of 0x4 reduced the rate in the HEC channels considerably
      A threshold of 0x6 removes entirely

      Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned

09:59 Had access so replaced the adapter card on AIDA05
      Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
      This could also have been the cause


10:09 Restarted DAQ and there is no change to the behavior of AIDA05
      Still see events in HEC
      Again raising threshold to 0x5 in ASIC1 largely removes them. 

13.10 Attachments 3-5
      DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
      ASIC check load
      ASIC slow comparator all FEE64s all ASICS 0xa
      ADC data item stats

13.17 Attachments 6-9
      DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
      ADC data item stats
      NewMerger/Tape Server

18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
      It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2

      To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
      After powering on it was found there was no improvement and possible worse noise than before

      We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14

8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam

Reseat aida05 ribbon cable on adapter board while in...

No difference to rates, aida05 still shows hot asic01
aida14 seems seated OK, still asic02 basically empty

Noise level still high, threshold set to 0x20 (320 keV) which has decent rate

Start forwarding to MBS with 0x20 thresholds

10.12 slow comparator 0x20 -> 0xa

Attachments 1-5

Attachment 6 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 7 FEE64 temps OK

Attachment 8 DSSSD bias & leakage currents OK


13.30 system checks

      N.B. aida05 ASIC temp low and pause/resume data rate high

Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK

Attachments 10-15 spectra

Attacahments 16-19 system wide checks OK *except* WR error counter

Attachment 20 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 21 FEE64 temps OK

Attachment 22 DSSSD bias & leakage currents OK

Attachments 23-29 newMerger stats

20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable

22:26 - No implantation test although we had some brief beam through AIDA.
      - FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow
        Will keep DAQ on 0x64 to eliminate deadtime issues for implantation

09.32 Re-start AIDA

Attachments 1 & 2 1.8.W spectra 20us FSR

Attachments 3 & 4 1.8.L spectra
 aida10 98ch FWHM
 aida06 384 ch FWHM

Attachment 5 per FEE64 rate spectra

Attachments 6-13 system wide checks - all OK

Attachment 14 ADC data item stats
 n+n FEE64s c. 250k -> c. 370k
 aida01, 7 & 12 c. 200-400k 

Attachment 15 FEE64 temps OK

Attachment 16 DSSD bias & leakage current OK

Attachment 17 per FEE64 sys console status summary



10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
      ASIC check load

Attachments 18 ADC data item stats
 no change to n+n FEE64s, very high p+n FEE64s now at lower rates

Attachments 19 & 20 1.8.W spectra 20us FSR


18.27 all ADC data rates very high
      aida05, asic #1 is again producing HEC data
      aida14, asic #2 producing very little data

Attachments 23 & 24 1.8.W spectra 20us FSR

Attachments 25 & 26 1.8.L spectra
 aida10 165ch FWHM

Attachment 27 per FEE64 rate spectra

Attachment 28 ADC data item stats

Attachment 29 FEE64 temps OK

Attachment 30 DSSD bias & leakage current OK

Attachments 21 & 22 1.8.L spectra
 aida10 65ch FWHM
 aida06 394ch FWHM

Attachment 31 Rate with threesholds at 0x64

14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change. 


14.30 Replace FEE64 aida05
      old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
      old dhcpd.conf file backed up

AIDA FEE64 layout with respect to the beam.

DSSSD bias and FEE64 power OFF during Ge install



Photographs of AIDA (triple) + DEGAS setup as of Sunday pm.

Attachments 1-4 side/front/oblique views - note AIDA snout droops and twists

Attachments 5 & 6 - position of the AIDA stand vertical adjustment threads (brass) 

The asymmetry upstream-downstream pre-dates this setup. To lift the end of the snout clear of the of lower DEGAS detectors all 3x 1/4 turns clockwise (viewed from above) were applied to each of the four threads.

Adaptor PCB link settings

n+n FEE64s

aida02, 4, 6 & 8 - LK1 fitted, HV Lemo-00 not connected - left/right wafer n+n side bias ring -> gnd

p+n FEE64s

   top - aida09, 1, 10 & aida13, 5, 14 - LK2 & LK4 fitted - front/back FP -> gnd
bottom - aida11, 3, 12 & aida15, 7, 16 - LK2 & LK4 fitted - front/back FP -> gnd

   top - aida01, 5 - LK3 fitted - middle wafer n+n side bias ring -> gnd
bottom - aida03, 7 - LK3 fitted - DSSSD PCB gnd -> gnd

AIDA PSU

   top 1-2, 3-4, 9-10, 11-12
bottom 5-6, 7-8, 13-14, 15-16


ASIC settings 2021Apr29-13-16-00
     slow comparator 0x64 -> 0xa

BNC PB-5 settings
    amplitude 1.000V
    attenuation x1
    tail pulse
    tau_d 1ms
    polarity +
    frequency 25Hz