ID |
Date |
Author |
Subject |
|
438
|
Fri May 13 02:56:36 2022 |
ML | 2h-Shift Checks |
All seemnormal on AIDA side.
Stats as before
Temperatures stable
HV-LV ok
System wide check : all results as in the previous checks |
|
437
|
Fri May 13 02:28:36 2022 |
ML | beam dropped |
We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).
This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too. |
|
436
|
Fri May 13 00:50:16 2022 |
ML | 2h-Shift Checks |
All seems ok (apart for my virgin network connection usually quite reliable but not so tonight !?!).
Statistics seems ok.
Temperatures appears stable
Leakage current ok.
System Wide Checks:
Clock check: Passed= 14: - Failed=: 0
ADC calibration check:
FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES
White rabbit decode status: Passed=14 ; Failed=0
FPGA timestamp check: Passed=14 ; Failed=0
Memory information: Page loading failed with the error message below:
Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]" |
|
435
|
Thu May 12 22:49:39 2022 |
BA, MA | Low Rates |
The rate is very low in all aida, not sure . |
|
434
|
Thu May 12 21:43:49 2022 |
BA, MA | Rates |
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
|
|
433
|
Thu May 12 16:30:47 2022 |
NH, OH, TD | Notes for mounting a second switch |
Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida* |
|
432
|
Thu May 12 10:10:51 2022 |
TD, OH | AIDA data throughput |
Attachment 1 - top aida06
Attachment 2 - top aida01
Attachment 3 - top aida-gsi (AIDA server)
Attachment 4 - TapeServer
Attachment 5 - NewMerger stats block 6 (aida06)
Attachment 6 - NewMerger stats block 1 (aida01)
Attachment 7 - NewMerger stats block 0 (all FEE64s)
Attachment 8 - NewMerger options
Attachment 9 - NewMerger link stats
Attachment 10 - NewMerger
Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats
Attacgment 15 - Network traffic |
|
431
|
Thu May 12 07:09:43 2022 |
TD | Thursday 12 May 08:00-24:00 |
08.10 AIDA DAQ status
14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
AD9222 OFF - waveforms disabed
disc - all disabled
ASIC settings 2021Apr29-13-16-00
slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23
BNC PB-5
amplitude 1.000V
attenuation x1
tau_d 1ms (tail pulse)
polarity +
frequency 22Hz
08.14 all histograms zero'd
2
08.15 system wide check counter baseline
08.17 check ASIC control - all FEE64s all ASICs
Attachment 1 - grafana DSSSD bias & leakage current - OK
Attachment 2 - per FEE64 stat spectra
Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips
Attachments 5 & 6 - per FEE64 1.8.L spectra
pulser peak width aida01 124ch FWHM, aida04 293ch FWHM
Attachment 7 - per FEE64 rate spectra
Attachments 8-12 - system wide checks - all OK
Attachments 13-16 - statistics - adc, pause, resume & correlation data
Attachment 17 - FEE64 temps OK
Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends
Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages
08.44 DAQ continues - file NOTAPE/R18 - no storage mode
08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc
08.53 no storage mode enabled
Attachment 21 - analysis of file NOTAPE/R18_20
Attachment 22- analysis of file NOTAPE/R19_19
AIDA12 slow comp up to 0xd improved deadtime drastically for FEE
10:54 Beam on detector. Take another run to test deadtime - attachment 23
Deadtimes ranging from 10-40% in n+n FEEs
Attachment 23 & 24 aida06 aida02 1.*.H spectra
14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime) |
|
430
|
Thu May 12 00:37:52 2022 |
OH, AM | Thursday 12 May |
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
That is now complete.
Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc
R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
Will lower 2 and 4 to 200 and raise 6 and 8 to 300
R17 2 good, 4 shakey 6 and 8 poor - attachment 2
R18 2 and 4 at 220 and 6 and 7 at 350
2 and 8 low deadtime, 4 and 6 could be better. - attachment 3
Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
Statistics at this rate - attachment 4
02:18 Statistics - attachment 7
Temperatures - attachment 6
Bias & leakage current - attachment 5
Clock Status - OK
White Rabbit - OK
The other 3 checks currently aren't working
04:11 Statistics - attachment 10
Temperatures - attachment 9
Bias & leakage current - attachment 8
Clock Status - OK
White Rabbit - OK
06:05 Statistics - attachment 13
Temperatures - attachment 12
Bias & leakage current - attachment 11
Clock Status - OK
White Rabbit - OK |
|
429
|
Wed May 11 19:08:41 2022 |
CB, PP | Wednesday 11th May - evening shift |
20:19 no beam yet.
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
All ADC failed calibration. Clock and white rabbit all passed.
22:12 no beam yet.
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
All ADC failed calibration. Clock and white rabbit all passed. |
|
428
|
Wed May 11 07:04:13 2022 |
OH, NH, TD | Wednesday 11th May |
08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
It is currently sending data pause requests and data push requests but nothing else - attachment 1
telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
FEE appears to have hung. - attachment 2
Still passes all system wide checks.
When performing a check load can see the tclsh process take up CPU usage
09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
AIDA14 recovered but ASIC2 still missing
Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
Given access would change the adapter board on AIDA05.
Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
Would retain the option of a speedybacktrack
Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
A threshold of 0x4 reduced the rate in the HEC channels considerably
A threshold of 0x6 removes entirely
Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned
09:59 Had access so replaced the adapter card on AIDA05
Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
This could also have been the cause
10:09 Restarted DAQ and there is no change to the behavior of AIDA05
Still see events in HEC
Again raising threshold to 0x5 in ASIC1 largely removes them.
13.10 Attachments 3-5
DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
ASIC check load
ASIC slow comparator all FEE64s all ASICS 0xa
ADC data item stats
13.17 Attachments 6-9
DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
ADC data item stats
NewMerger/Tape Server
18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2
To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
After powering on it was found there was no improvement and possible worse noise than before
We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14 |
|
427
|
Tue May 10 07:41:28 2022 |
NH TD | Tuesday 10 May |
8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam
Reseat aida05 ribbon cable on adapter board while in...
No difference to rates, aida05 still shows hot asic01
aida14 seems seated OK, still asic02 basically empty
Noise level still high, threshold set to 0x20 (320 keV) which has decent rate
Start forwarding to MBS with 0x20 thresholds
10.12 slow comparator 0x20 -> 0xa
Attachments 1-5
Attachment 6 ADC data items
c. 150-180k per FEE64, c. 21Mb to disk
Attachment 7 FEE64 temps OK
Attachment 8 DSSSD bias & leakage currents OK
13.30 system checks
N.B. aida05 ASIC temp low and pause/resume data rate high
Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK
Attachments 10-15 spectra
Attacahments 16-19 system wide checks OK *except* WR error counter
Attachment 20 ADC data items
c. 150-180k per FEE64, c. 21Mb to disk
Attachment 21 FEE64 temps OK
Attachment 22 DSSSD bias & leakage currents OK
Attachments 23-29 newMerger stats
20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable
22:26 - No implantation test although we had some brief beam through AIDA.
- FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow
Will keep DAQ on 0x64 to eliminate deadtime issues for implantation |
|
426
|
Mon May 9 08:36:23 2022 |
TD, NH | Monday 9 May |
09.32 Re-start AIDA
Attachments 1 & 2 1.8.W spectra 20us FSR
Attachments 3 & 4 1.8.L spectra
aida10 98ch FWHM
aida06 384 ch FWHM
Attachment 5 per FEE64 rate spectra
Attachments 6-13 system wide checks - all OK
Attachment 14 ADC data item stats
n+n FEE64s c. 250k -> c. 370k
aida01, 7 & 12 c. 200-400k
Attachment 15 FEE64 temps OK
Attachment 16 DSSD bias & leakage current OK
Attachment 17 per FEE64 sys console status summary
10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
ASIC check load
Attachments 18 ADC data item stats
no change to n+n FEE64s, very high p+n FEE64s now at lower rates
Attachments 19 & 20 1.8.W spectra 20us FSR
18.27 all ADC data rates very high
aida05, asic #1 is again producing HEC data
aida14, asic #2 producing very little data
Attachments 23 & 24 1.8.W spectra 20us FSR
Attachments 25 & 26 1.8.L spectra
aida10 165ch FWHM
Attachment 27 per FEE64 rate spectra
Attachment 28 ADC data item stats
Attachment 29 FEE64 temps OK
Attachment 30 DSSD bias & leakage current OK
Attachments 21 & 22 1.8.L spectra
aida10 65ch FWHM
aida06 394ch FWHM
Attachment 31 Rate with threesholds at 0x64
14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change.
14.30 Replace FEE64 aida05
old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
old dhcpd.conf file backed up |
|
425
|
Sun May 8 17:43:57 2022 |
NH | AIDA FEE64 Layout |
AIDA FEE64 layout with respect to the beam. |
|
424
|
Sun May 8 15:10:45 2022 |
TD | Sunday 8 May |
DSSSD bias and FEE64 power OFF during Ge install
Photographs of AIDA (triple) + DEGAS setup as of Sunday pm.
Attachments 1-4 side/front/oblique views - note AIDA snout droops and twists
Attachments 5 & 6 - position of the AIDA stand vertical adjustment threads (brass)
The asymmetry upstream-downstream pre-dates this setup. To lift the end of the snout clear of the of lower DEGAS detectors all 3x 1/4 turns clockwise (viewed from above) were applied to each of the four threads. |
|
423
|
Sat May 7 13:27:39 2022 |
TD | AIDA configuration |
Adaptor PCB link settings
n+n FEE64s
aida02, 4, 6 & 8 - LK1 fitted, HV Lemo-00 not connected - left/right wafer n+n side bias ring -> gnd
p+n FEE64s
top - aida09, 1, 10 & aida13, 5, 14 - LK2 & LK4 fitted - front/back FP -> gnd
bottom - aida11, 3, 12 & aida15, 7, 16 - LK2 & LK4 fitted - front/back FP -> gnd
top - aida01, 5 - LK3 fitted - middle wafer n+n side bias ring -> gnd
bottom - aida03, 7 - LK3 fitted - DSSSD PCB gnd -> gnd
AIDA PSU
top 1-2, 3-4, 9-10, 11-12
bottom 5-6, 7-8, 13-14, 15-16
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 -> 0xa
BNC PB-5 settings
amplitude 1.000V
attenuation x1
tail pulse
tau_d 1ms
polarity +
frequency 25Hz |
|
422
|
Sat May 7 09:55:53 2022 |
TD | Saturday 7 May |
Attachments 1 & 2 per FEE64 1.8.L spectra
pulser peak widths aida01 121 ch FWHM, aida02 517 ch FWHM
Attachments 3 & 4 per FEE64 1.8.W spectra 20us FSR
Attachment 5 per FEE64 rate spectra
Attachment 6 - grafana DSSSD bias/leakage current for previous 7 days showing effect of day/night temp variations
Attachments 7 & 8 - merger & tape server c. 14Mb/s to disk - no errors reported by servers
Attachments 9-12 - system wide checks - all OK
Attachment 13 ADC data item stats
Attachment 14 FEE64 temps OK
Attachment 15 DSSSD bias & leakage currents OK
16.15 aida05 zero ADC data items stats
merger restart, DAQ STOP/Go cycle to recover
activity is S4 area near AIDA DSSSDs as install of DEGAS detectors commences
19.50 DEGAS install continues
DSSSD bias and FEE64 power OFF
|
|
421
|
Fri May 6 09:31:41 2022 |
TD NH | Friday 6 May |
10:30 DAQ appears to have stopped
Attachment 2 - DSSSD bias and leakage current OK
Attachments 1, 3-5 - zero rate adc data items all FEE64s - note aida08 adc data item counter zero
RESET/SETUP/GO aida08 *only*
DAQ STOP *all* FEE64s
Attachment 6 - FEE64 temperatures OK
Attachments 7-23 - system wide checks - all OK
10:48 DAQ GO *all* FEE64s
all spectra zero'd
attachment 29 - ADC data items
n+n FEE64s aida02, 4, 6 & 8 c. 250k
p+n FEE64s aida07, 9 & 12 c. 100-150k, all other FEE64s < 100k
attachments 26 & 27 1.8.L spectra
aida01 peak width 112ch FWHM
aida02 peak width 485 ch FWHM
attachments 24 & 25 1.8.W spectra 20us FSR
attachment 30 - aida05 1*H spectra - aida05 asic#1 is producing HEC data
attachment 31 - aida05 also producing disc data
attachments 32 & 33 merger and tape server OK - 14MB/s to disk - no data to MBS currently?
18.36 Disconnect ribbon cabling from adaptor PCB - continue to observe HEC events from aida05 asic#1
Disconnect test inputs from adaptor PCB - continue to observe HEC events from aida05 asic#1
Conclude aida05 asic#1 has probably failed in operation and should be replaced. Other FEE64s are currently in CRYRING hall and measurement is in progress - ML says we should be able to access Monday. |
|
420
|
Wed May 4 16:50:04 2022 |
NH | AIDA 4th May |
bPlas has been mostly cabled in now
Reseat LAN cable (AIDA Switch -> AIDA-3) and reachieve 1 gbit/s speed as needed.
CHeck DAQ with MBS, working well (16 MB/s). Performance seems better than before on MBS side too
Note implant counts in aida05, observable in rate spectra (fig 1)
Also a few early channels in aida10 seem empty
ASIC check-loads didn't fix
Try powercycle... also didn't fix
Tomorrow will inspect adapter board and ribbon cable, perhaps it is loose.
Absolute rates otherwise seem OK (fig 2) aida01 and aida07 are both very hot.
aida01 is usually quieter but aida07 has always been observed running fast |
|
419
|
Tue May 3 14:15:17 2022 |
NH | AIDA Updates 2 May |
PAUSE/RESUME stats are dominated by 2x FEE64s - aida07 and aida16
These FEE64s also have the highest rates of discriminator data?
> The snout endcap has been sealed with new copper tape over the red tape
>
> Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
> It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive
>
>
>
> -
>
> the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
> When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s
>
> -
>
> Preliminary alpha background analysis (over weekend, 90 hours statistics)
>
> Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04
>
> ======================================
> AIDA Unpacker Analysis
> --------------------------------------
> First event: Thu Apr 28 19:50:20 2022
> Last event : Mon May 2 14:04:03 2022
> Duration : 324823 seconds
> --------------------------------------
> MBS Events (M): 912899 (2 Hz)
> AIDA Words (W): 7472078760 (-3441 Hz)
> ADC Words (A): 2742871865 (-4778 Hz)
> -- Decays (D): 2728717223 (-4821 Hz)
> -- Implants (I): 14154642 (43 Hz)
> Info Words (F): 4729206895 (1 kHz)
> -- SYNC48 (S): 2093328 (6 Hz)
> -- SYNC63 (s): 2093328 (6 Hz)
> -- PAUSE (P): 3823161 (11 Hz)
> -- RESUME (R): 3823161 (11 Hz)
> -- Discrim (D): 1390520565 (4 kHz)
> -- Scaler (C): 3326853352 (3 kHz)
> -- Unknown (U): 0 (0 Hz)
> Timewarps (T): 20 (0 Hz)
> Dead Time (X): 5024.43 s (1.54682%)
> ======================================
> ======================================
> Individual FEE Overview
> ======================================
> FEE | W A D I F S s P R D C U T X
> 1 | 9593 6985 6628 357 2608 2 2 0 0 2604 0 0 0 0
> 2 | 31134221 31115668 31115166 502 18553 8183 8183 0 0 2187 0 0 0 0
> 3 | 717765720 17634 17218 416 717748086 216426 216426 0 0 1653 717313581 0 0 0
> 4 | 3467468322 2684820687 2684820157 530 782647635 1293089 1293089 0 0 62747129 717314328 0 20 0
> 5 | 177575 108631 107352 1279 68944 21 21 20 20 68862 0 0 0 0.00102142
> 6 | 8371118 8361524 8358540 2984 9594 1350 1350 23 23 6848 0 0 0 10.6775
> 7 | 1396478942 4860615 57719 4802896 1391618327 346048 346048 1286994 1286994 446838393 941513850 0 0 3071.3
> 8 | 954933629 3978543 3975110 3433 950955086 118195 118195 5 5 7093 950711593 0 0 5.47318
> 9 | 9877 7923 7429 494 1954 4 4 0 0 1946 0 0 0 0
> 10 | 8076 6236 5917 319 1840 4 4 0 0 1832 0 0 0 0
> 11 | 26792 24399 24029 370 2393 3 3 0 0 2387 0 0 0 0
> 12 | 18271 16025 15382 643 2246 6 6 0 0 2234 0 0 0 0
> 13 | 1094646 51444 48305 3139 1043202 195 195 1 1 1042810 0 0 0 0.179207
> 14 | 7314376 152823 84786 68037 7161553 906 906 18158 18158 7123425 0 0 0 32.882
> 15 | 438092 22117 20290 1827 415975 56 56 251 251 415361 0 0 0 4.91581
> 16 | 886829510 9320611 53195 9267416 877508899 108840 108840 2517709 2517709 872255801 0 0 0 4970.9 |