 Sat May 14 03:14:00 2022, PP, Shift Checks 8x
|
4:13
AIDA stats OK
Leakage current OK
Temperatures OK
grafana OK
ucesb rates OK
System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Screenshot 1: statistics
Screenshot 2: temperatures
Screenshot 3: scalers
Screenshot 4: Bias and leakage current
6:36
AIDA stats OK
Leakage current OK
Temperatures OK
grafana OK
ucesb rates OK
System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Screenshot 5: statistics
Screenshot 6: temperatures
Screenshot 7: scalers
Screenshot 8: Bias and leakage current
|
|
Sat May 14 01:52:03 2022, ML, 2h-Shift Checks
|
AIDA stats ok
Leakage current ok
Temperatires ok
grafana ok
ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.
System wide check done and same results as earlier:
aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR |
|
Fri May 13 23:37:59 2022, ML, 2h-Shift Checks
|
Took over remotely from CB about an hour ago
At 0h40 (CET):
Stats still ok - attach 1
Leakage still OK - Attach 2
Temps - attach 3
Grafana OK
UCESB rates ok - attach 5
System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR |
|
Fri May 13 15:00:23 2022, CB, 13 March - 16:00-24:00 shift 10x
|
16:00 Took over remotely from OH, TD who remain in GSI
Stats OK - attach 1
Stats & Leakage OK - attach 2
Temps OK
Grafana OK - attach 3
System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
17:29 All good, but no beam.
17:30 Beam is back
17:58 No beam again
18:10 Beam is back. Stats OK.
Ucesb OK - attach 4
Stats & Leakage currents ok - attach 5
Temps OK - attach 6
Grafana OK - attach 7
System-wide checks as before.
19:36 No beam again.
20:37 Beam is back
Stats OK - attach 8
Leakage OK
Temps OK
Grafana OK - attach 9
Ucesb OK - attach 10
System-wide checks as before
21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.
22:51 Stats & Leakage current OK - attach 11
Temps OK
Grafana OK - attach 12
Ucesb OK - attach 13
System-wide checks as before |
|
Fri May 13 07:36:18 2022, OH, TD, Friday 13 May 35x
|
08:36 DAQ Still running smoothly
Currently on run R1_328
At current data rates we have just over 4 days of space left on the current disk
Analysis of R1_327 - attachment 1
14.15 all histograms zero'd
baseline system wide checks counters
DAQ continues file S450/R4_33
netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
netint flushenable 60 - not changed
ASIC settings
slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc
BNC PB-5 settings
amplitude 1.000V
attenuation x1
frequency 2Hz
tau_d 1ms (tail pulse)
+ polarity
Attachments 2 & 3 - ucesb
Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.
Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK
Attachments 6-8 - NewMerger stats blocks 0, 1, & 6
Attachment 9 - NewMerger link stats
Attachments 10-11 - TapeServer/NewMerger
Attachment 12 - iptraf aida-gsi network interfaces
Attachments 13-14 - per FEE64 1.8.L spectra
pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM
Attachments 15-18 - per FEE64 rates and stat spectra
Attachments 19-22 - system wide checks
aida09 clock status 6, WR decoder status errors
Attachments 23-26 - adc, pause, resume correlation sclaer data item stats
Attachment 27 - FEE64 temps OK
Attachment 28 - DSSSSD bias and leakage currents OK
14.30 check ASIC load
14.47 attachment 29 - analysis S450/R4_38
15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate) |
|
Thu May 12 22:49:39 2022, BA, MA, Low Rates
|
The rate is very low in all aida, not sure . |
 Fri May 13 06:56:14 2022, OH, BA, MA, Low Rates
|
| Quote: |
|
The rate is very low in all aida, not sure .
|
This was because the statistics page had been changed to transfer buffers. The rates were ok |
|
Fri May 13 06:47:33 2022, ML, 2h-Shift Checks
|
It has been a quiet night. AIDA has been stable.
Stats appears stable appear
Temperature also stable
Leakage current slowly dropping still.
System wide checks done. Status quo here again. |
|
Fri May 13 04:45:23 2022, ML, 2h-Shift Checks
|
AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.
Stats seems ok
Temperatures stable
Leakage current: ok
A systwm wide check gives same result as for the fist check on this shift at ~1am |
|
Fri May 13 02:56:36 2022, ML, 2h-Shift Checks
|
All seemnormal on AIDA side.
Stats as before
Temperatures stable
HV-LV ok
System wide check : all results as in the previous checks |
|
Fri May 13 02:28:36 2022, ML, beam dropped
|
We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).
This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too. |
|
Fri May 13 00:50:16 2022, ML, 2h-Shift Checks
|
All seems ok (apart for my virgin network connection usually quite reliable but not so tonight !?!).
Statistics seems ok.
Temperatures appears stable
Leakage current ok.
System Wide Checks:
Clock check: Passed= 14: - Failed=: 0
ADC calibration check:
FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES
White rabbit decode status: Passed=14 ; Failed=0
FPGA timestamp check: Passed=14 ; Failed=0
Memory information: Page loading failed with the error message below:
Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]" |
|
Thu May 12 21:43:49 2022, BA, MA, Rates
|
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
|
|
Thu May 12 16:30:47 2022, NH, OH, TD, Notes for mounting a second switch
|
Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida* |
|
Thu May 12 10:10:51 2022, TD, OH, AIDA data throughput 15x
|
Attachment 1 - top aida06
Attachment 2 - top aida01
Attachment 3 - top aida-gsi (AIDA server)
Attachment 4 - TapeServer
Attachment 5 - NewMerger stats block 6 (aida06)
Attachment 6 - NewMerger stats block 1 (aida01)
Attachment 7 - NewMerger stats block 0 (all FEE64s)
Attachment 8 - NewMerger options
Attachment 9 - NewMerger link stats
Attachment 10 - NewMerger
Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats
Attacgment 15 - Network traffic |
|
Thu May 12 07:09:43 2022, TD, Thursday 12 May 08:00-24:00 27x
|
08.10 AIDA DAQ status
14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
AD9222 OFF - waveforms disabed
disc - all disabled
ASIC settings 2021Apr29-13-16-00
slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23
BNC PB-5
amplitude 1.000V
attenuation x1
tau_d 1ms (tail pulse)
polarity +
frequency 22Hz
08.14 all histograms zero'd
2
08.15 system wide check counter baseline
08.17 check ASIC control - all FEE64s all ASICs
Attachment 1 - grafana DSSSD bias & leakage current - OK
Attachment 2 - per FEE64 stat spectra
Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips
Attachments 5 & 6 - per FEE64 1.8.L spectra
pulser peak width aida01 124ch FWHM, aida04 293ch FWHM
Attachment 7 - per FEE64 rate spectra
Attachments 8-12 - system wide checks - all OK
Attachments 13-16 - statistics - adc, pause, resume & correlation data
Attachment 17 - FEE64 temps OK
Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends
Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages
08.44 DAQ continues - file NOTAPE/R18 - no storage mode
08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc
08.53 no storage mode enabled
Attachment 21 - analysis of file NOTAPE/R18_20
Attachment 22- analysis of file NOTAPE/R19_19
AIDA12 slow comp up to 0xd improved deadtime drastically for FEE
10:54 Beam on detector. Take another run to test deadtime - attachment 23
Deadtimes ranging from 10-40% in n+n FEEs
Attachment 23 & 24 aida06 aida02 1.*.H spectra
14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime) |
|
Thu May 12 00:37:52 2022, OH, AM, Thursday 12 May 13x
|
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
That is now complete.
Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc
R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
Will lower 2 and 4 to 200 and raise 6 and 8 to 300
R17 2 good, 4 shakey 6 and 8 poor - attachment 2
R18 2 and 4 at 220 and 6 and 7 at 350
2 and 8 low deadtime, 4 and 6 could be better. - attachment 3
Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
Statistics at this rate - attachment 4
02:18 Statistics - attachment 7
Temperatures - attachment 6
Bias & leakage current - attachment 5
Clock Status - OK
White Rabbit - OK
The other 3 checks currently aren't working
04:11 Statistics - attachment 10
Temperatures - attachment 9
Bias & leakage current - attachment 8
Clock Status - OK
White Rabbit - OK
06:05 Statistics - attachment 13
Temperatures - attachment 12
Bias & leakage current - attachment 11
Clock Status - OK
White Rabbit - OK |
|
Wed May 11 19:08:41 2022, CB, PP, Wednesday 11th May - evening shift
|
20:19 no beam yet.
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
All ADC failed calibration. Clock and white rabbit all passed.
22:12 no beam yet.
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
All ADC failed calibration. Clock and white rabbit all passed. |
|
Wed May 11 07:04:13 2022, OH, NH, TD, Wednesday 11th May 9x
|
08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
It is currently sending data pause requests and data push requests but nothing else - attachment 1
telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
FEE appears to have hung. - attachment 2
Still passes all system wide checks.
When performing a check load can see the tclsh process take up CPU usage
09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
AIDA14 recovered but ASIC2 still missing
Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
Given access would change the adapter board on AIDA05.
Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
Would retain the option of a speedybacktrack
Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
A threshold of 0x4 reduced the rate in the HEC channels considerably
A threshold of 0x6 removes entirely
Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned
09:59 Had access so replaced the adapter card on AIDA05
Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
This could also have been the cause
10:09 Restarted DAQ and there is no change to the behavior of AIDA05
Still see events in HEC
Again raising threshold to 0x5 in ASIC1 largely removes them.
13.10 Attachments 3-5
DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
ASIC check load
ASIC slow comparator all FEE64s all ASICS 0xa
ADC data item stats
13.17 Attachments 6-9
DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
ADC data item stats
NewMerger/Tape Server
18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2
To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
After powering on it was found there was no improvement and possible worse noise than before
We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14 |
|
Tue May 10 07:41:28 2022, NH TD, Tuesday 10 May 29x
|
8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam
Reseat aida05 ribbon cable on adapter board while in...
No difference to rates, aida05 still shows hot asic01
aida14 seems seated OK, still asic02 basically empty
Noise level still high, threshold set to 0x20 (320 keV) which has decent rate
Start forwarding to MBS with 0x20 thresholds
10.12 slow comparator 0x20 -> 0xa
Attachments 1-5
Attachment 6 ADC data items
c. 150-180k per FEE64, c. 21Mb to disk
Attachment 7 FEE64 temps OK
Attachment 8 DSSSD bias & leakage currents OK
13.30 system checks
N.B. aida05 ASIC temp low and pause/resume data rate high
Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK
Attachments 10-15 spectra
Attacahments 16-19 system wide checks OK *except* WR error counter
Attachment 20 ADC data items
c. 150-180k per FEE64, c. 21Mb to disk
Attachment 21 FEE64 temps OK
Attachment 22 DSSSD bias & leakage currents OK
Attachments 23-29 newMerger stats
20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable
22:26 - No implantation test although we had some brief beam through AIDA.
- FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow
Will keep DAQ on 0x64 to eliminate deadtime issues for implantation |
|
Mon May 9 08:36:23 2022, TD, NH, Monday 9 May 31x
|
09.32 Re-start AIDA
Attachments 1 & 2 1.8.W spectra 20us FSR
Attachments 3 & 4 1.8.L spectra
aida10 98ch FWHM
aida06 384 ch FWHM
Attachment 5 per FEE64 rate spectra
Attachments 6-13 system wide checks - all OK
Attachment 14 ADC data item stats
n+n FEE64s c. 250k -> c. 370k
aida01, 7 & 12 c. 200-400k
Attachment 15 FEE64 temps OK
Attachment 16 DSSD bias & leakage current OK
Attachment 17 per FEE64 sys console status summary
10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
ASIC check load
Attachments 18 ADC data item stats
no change to n+n FEE64s, very high p+n FEE64s now at lower rates
Attachments 19 & 20 1.8.W spectra 20us FSR
18.27 all ADC data rates very high
aida05, asic #1 is again producing HEC data
aida14, asic #2 producing very little data
Attachments 23 & 24 1.8.W spectra 20us FSR
Attachments 25 & 26 1.8.L spectra
aida10 165ch FWHM
Attachment 27 per FEE64 rate spectra
Attachment 28 ADC data item stats
Attachment 29 FEE64 temps OK
Attachment 30 DSSD bias & leakage current OK
Attachments 21 & 22 1.8.L spectra
aida10 65ch FWHM
aida06 394ch FWHM
Attachment 31 Rate with threesholds at 0x64
14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change.
14.30 Replace FEE64 aida05
old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
old dhcpd.conf file backed up |