 Tue Jun 22 13:01:19 2021, NH, AIDA Data on lustre
|
The following runs are now copied to lustre
If necessary they can be removed from AIDA HDDs for space.
There are also spare HDDs that can (and should) be swapped for the current ones
/lustre/gamma/DESPEC_S452_FILES/AIDA
S452
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58
/lustre/gamma/DESPEC_S460_FILES/AIDA
S460
R1
R2
R4
R5
R7
R9
R11
R13
R14
R16
R20
R21
R23
R24
R25
R26
R27
R30
R31
R32
R33
R34
R35
R36
R38
R40
R45
R46
R48
R50
R51
R52
/lustre/gamma/DESPEC_S496_FILES/AIDA
S496
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58 |
|
Fri Apr 16 00:47:36 2021, OH-ML, AIDA DAQ Reset
|
1:18 (CET) No rate in AIDA 07.
Oscar did a AIDA DAQ reset and a quick wide check.
(Power cycle
All back to normal. |
|
Sat Apr 24 03:13:05 2021, ML, AIDA DAQ Reset
|
AIDA PC crashe completely at ~3:30am CET.
Tom was called and he Restarted the AIDA system at ~4am with Run 52.
Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.
|
|
Tue Feb 1 15:00:28 2022, NH, AIDA CentOS 7 Startup
|
Starting up AIDA on CentOS 7 to confirm things work (based on some CARME experience)
NFS settings:
/etc/exports copied over from old disk
line to old /MIDAS_130718 export removed (commented) to see if necessary to copy over
nfsv2 enabled in nfs.conf
NFS service enabled and started (systemctl enable nfs-server ; systemctl start nfs-server)
DHCP setting:
DHCP was disabled, enabled and started (systemctl enable dhcpd ; systemctl start dhcpd)
Raspberry Pis pick up IP and become responsive again
Firwall setting:
Interface p4p1 (AIDA switch #1) is changed to zone 'trusted' so that FEEs can connect to NFS
(panicked before unable to get RPC port)
See FEEs starting to boot!
Getting both / and /MIDAS successfully (note confirming that old /MIDAS_130718 is not needed)
FEEs 4 and 9 take longer to boot others (observed before, seems to be PSU slots 4/5 in first take a while to provide voltage)
FEE9 seems to panic after mounting root, waiting to see if it reboots itself (no serial)
... Nope but power cycle fixes it
All FEEs online and Reset/Setup completed... Temps OK
Note 1: CentOS 7 uses systemd, monitoring log is best done as
`journalctl -f` as root.
npg not in sudoers (should be done)
Note 2: System used as graphical terminal for Edinburgh double-alpha experiment until Feb. 11
AIDA operated in background and using port forwarding to access MIDAS/Web mostly
UI has weird workspace configuration by default needs changing to better cope with AIDA windows
Note 3:
Only 4 FEEs are producing serial data to the Pi_Monitor
Probably 2 of the USB hubs were knocked lose (from the USB extender cables) when the Fatima/HPGe cables were removed last year
S4 access is limited but will try to reconnect ASAP
Note 4:
AIDA setup unfortunately quite limited in functionality due to other running experiments:
WR is unavailable as VETAR VME crate was powered off
Not all FEEs are connected to pulser or grounded. Some cables were removed during testing last year.
Some ground cables removed and pulser inverter removed as needed at CARME and S4 access is now limited for WASA tests |
|
Mon Oct 28 17:02:11 2019, TD, AIDA @ DESPEC cable length estimates
|
AIDA snout length 37cm (from end of AIDA snout support assembly)
downstream scintillator (assembly depth 1cm) 36cm
downstream dsssd (#1) 35.2cm
(#2) 34.3cm
(#3) 33.4cm
upstream dsssd (#4) 32.5cm
upstream scintillator (assembly depth 1cm) 31.7cm
Cable lengths required are therefore
downstream dsssd (#1) 35.2 - 10.5 + 2.15 + 5 + 22.5 = 54.35cm 21.4"
(#2) 34.3 - 10.5 + 2.15 + 5 + 16.5 = 47.45cm 18.7"
(#3) 33.4 - 10.5 + 2.15 + 5 + 10.5 = 40.55cm 16.0"
upstream dsssd (#4) 32.5 - 10.5 + 2.15 + 5 + 4.5 = 33.65cm 13.2"
length of RH Kapton PCB coupler = 10.5cm (Carlo can you check please?)
turning the corner at the base of the AIDA snout support assembly 2.15+5=7.15cm
(see Peter's drawing)
distances to FEE64s 4.5, 10.5, 16.5 and 22.5cm
(see Peter's drawing)
f we use 3x DSSSDs then the cable lengths required will be as follows:
downstream dsssd (#1) 35.2 - 10.5 + 2.15 + 5 + 16.5 = 48.35cm 19.0"
(#2) 34.3 - 10.5 + 2.15 + 5 + 10.5 = 41.45cm 16.3"
upstream dsssd (#3) 33.4 - 10.5 + 2.15 + 5 + 4.5 = 34.55cm 13.6" |
Wed May 4 16:50:04 2022, NH, AIDA 4th May 
|
bPlas has been mostly cabled in now
Reseat LAN cable (AIDA Switch -> AIDA-3) and reachieve 1 gbit/s speed as needed.
CHeck DAQ with MBS, working well (16 MB/s). Performance seems better than before on MBS side too
Note implant counts in aida05, observable in rate spectra (fig 1)
Also a few early channels in aida10 seem empty
ASIC check-loads didn't fix
Try powercycle... also didn't fix
Tomorrow will inspect adapter board and ribbon cable, perhaps it is loose.
Absolute rates otherwise seem OK (fig 2) aida01 and aida07 are both very hot.
aida01 is usually quieter but aida07 has always been observed running fast |
|
Wed Apr 27 15:47:46 2022, NH, HA, SA, CJ, AIDA 27 Apr 22 7x
|
Remount snout in S4 (photos to come)
Alignment with DEGAS OK but a bit short... DEGAS platform will be brought 31 mm closer to AIDA by rail adjustment
Can mount bPlas booster boards on old ITEM frames and also partly on AIDA snout (isolated)
-
Reconnect all adapter boards and grounds, bias AIDA...
HV OK. We still have a light leak at the snout endcap. "Fixed" with sheet for now
HV leakage currents figure 1
Start AIDA DAQ and check rates.. all seem OK. aida07 is very noisy other p+n good
Notice some issues in aida04, check and ribbon cable slightly loose on one side of adapter
No change to other FEEs.
Aida07 was inspected but nothing obviously wrong
Rates figure 2
Rate spectra figure 3
Wavesform figures 4 (p+n), 5 (n+n)
Pulser peak withs
p+n figure 6, width ~ 70 channels (50 keV)
n+n figure 7, width ~ 570 channels (400 keV)
n.b. double peak in aida14 is an artifact from an uncleared histogram
DSSDs debiased to allow work on bPlas cabling without tripping DSSDs |
|
Sat May 1 08:45:19 2021, TD, AIDA 24cm x 8cm DSSSD + triPlast assembly for S496 12x
|
Photos provided by Nic Hubbard and Helena Albers from the assembly of 2x triPlast
plus 2x MSL type BB18(DS)-1000 24cm x 8cm detector assembly - attachments 1-3.
Spacers installed between MSL type BB18(DS)-1000 24cm x 8cm PCBs - attachment 4.
AIDA support stand in 24cm x 8cm DSSSD mode ready for installation of snout assembly - attachments 5 & 6
AIDA 24cm x 8cm DSSSD plus 2x TriPlast installed in S4 - view downstream towards upstream window
of AIDA detctor assembly - attachment 7
AIDA 24cm x 8cm DSSSD plus 2x TriPlast snout assembly installed at S4 - attachment 8
Side view of AIDA + triPLast + FATIMA + DEGAS - attachment 9 |
|
Mon Dec 16 14:48:02 2019, NH TD, AIDA 16.12.19
|
DSSD Snout was reassembled to install the second bPlast (downstream)
(fig 1)
Ribbon cable for aida10 was replaced (DSSD3 upper)
All DSSDs bias again (see fig 2)
Alpha run in progress - channels OK with 2 exceptions (fig 3)
- aida10 still recording minimal to no counts
- channel 55 aida08 reading lots of data even with comparator 0x64 - corresponds to DSSD1 strip 0 (bottom strip)
possible ground/cable issue? will check when access possible (fig 4)
Unusual MIDAS error noticed (fig 5) - restart of HTTPd fixed it |
|
Tue Aug 13 09:17:02 2019, NH, AIDA 12/08/2019
|
The other 6 FEEs were powered on one at a time to check no issues. None found.
All 12 FEEs plugged back into the PSU.
Note: The FEE temperatures reported on Friday were before a Reset/Setup of the DAQ, which means the ASICs run very fast and the hence run hotter.
Reset/Setup shows more sustainable temps (attached)
These temps are similar to those observed in Sep 2018 but warmer by ~4 C than those observed Dec-Jan and April. |
|
Mon Jul 1 09:42:16 2019, NH, AIDA - 01/07/2019
|
Water is still off.
Checked dewpoint sensor still flashing red light (indicating an error). Pipes warmed. Not what I expected. Possible fault?
I checked S4's ambient temp/humidity with a home-brought sensor: Reports 27 C/50% RH (Dew Point = 16 C)
Water temperature is set to exactly 20 C (there's a second dial showing ~18 C but I am unsure if that's the same water supply, I will ask)
Checking the dewpoint PSU with a multimeter shows a voltage of 16 V.
The PSU itself is rated for 12 V output, and the humidity sensor expects 24 V.
Regardless of which voltage 16 V seems incorrect. Might be causing issues.
Humidity is a bit lower now (4pm) the sensor has been in all day, 28 C/35% RH (Dew Point = 11 C) which isn't an issue.
Water is still off until I am sure what to do, or until the dewpoint sensor stops complaining.
|
|
Tue Jun 11 20:20:36 2024, TD, ADC data rates from 09.00 11.6.24
|
Per NH
"From ELOG (assuming times accurateish)
10:00 Source in (AIDA removed from DESPEC DAQ)
11:30: Source out (?)
11:45: Background Runs
13:00: bPlast thresholds changed
15:45: Beam Returned"
-
ADC data item rate
EOF time file LEC HEC
09.02 R4_538 0.906M 1.7k
10.02 _551 0.927M 1.7k
11.00 _564 0.875M 1.7k
11.19 _568 0.872M 1.7k
12.18 R6_7 1.121M 1.9k
13.03 6_19 1.134M 1.9k
14.02 6_35 1.197M 2.0k
15.00 6_52 1.232M 2.0k
15.10 6_54 1.225M 2.0k
15.36 6_56 1.467M 2.0k
16.00 6_65 1.630M 3.0k
17.01 6_89 1.778M 3.7k
21.43 6_207 1.759M 2.7k
12.6.24
00.09 6_274 2.258M 3.8k
03.12 6_372 2.295M 4.4k
07.13 6_506 2.284M 4.5k
11.12 6_611 1.853M 3.6k
14.55 7_16 1.877M 4.7k
13.6.24
07.01 7_457 1.788M 4.3k
12.34 7_610 2.000M 4.7k
14.00 7_649 1.756M 2.5k
14.12 7_654 1.817M 2.5k
15.33 7_693 1.907M 3.5k
14.6.24
07.23 8_35 1.677M 3.7k
10.54 8_122 1.880M 4.7k
14.38 8_213 1.935M 3.6k
15.6.24
01.25 9_27 0.951k 1.6k (beam off from c. 18.15 14.6.24)
08.56 9_152 1.215M 2.1k gamma source in use?
15.27 9_275 1.520M 2.3k 370kBq 152Eu source in use
16.6.24
15.06 9_747 1.667M 2.3k
15.06 analysis data file R9_747 - attachment 18
max dead time 4.3% aida08, all other FEE64s < 2%
ADC data rate 1.667M, HEC data rate 2.3k |
|
Fri Mar 22 14:31:24 2019, NH, A few more setup notes I forgot
|
Extra FEE information:
HV braid (0 V) goes into FEE10 (beam left) with LK1 jumper
HV core (-160 V) goes into FEE09 (top)
Could be related to noise in FEE09?
Both these FEEs have higher rates too
Other setup information:
According to the switch in S4, the AIDA workstation is only connected via 100Mbps rather than 1Gbps. (Fig 1)
Plastic is inside the snout, but is not yet connected to the electronics yet
Figures 2-4 Show the DSSD & Plastic in the Snout. Plastic is far upstream solely to ensure plenty of ribbon cable |
|
Sat Dec 7 02:39:02 2019, DK, NH et al., 34Si Fragments PID
|
See attached
Tentative PID (DK) as attachment #3
DESPEC LISE++ file as attachment #4 |
|
Wed Jun 1 08:00:18 2022, TD, 2xDSSSDs spectra layouts
|
Spectra layouts for 2x DSSSDs can be found in directory ~/LayOut/2xDSSSDs |
|
Fri May 13 00:50:16 2022, ML, 2h-Shift Checks
|
All seems ok (apart for my virgin network connection usually quite reliable but not so tonight !?!).
Statistics seems ok.
Temperatures appears stable
Leakage current ok.
System Wide Checks:
Clock check: Passed= 14: - Failed=: 0
ADC calibration check:
FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES
White rabbit decode status: Passed=14 ; Failed=0
FPGA timestamp check: Passed=14 ; Failed=0
Memory information: Page loading failed with the error message below:
Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]" |
|
Fri May 13 02:56:36 2022, ML, 2h-Shift Checks
|
All seemnormal on AIDA side.
Stats as before
Temperatures stable
HV-LV ok
System wide check : all results as in the previous checks |
|
Fri May 13 04:45:23 2022, ML, 2h-Shift Checks
|
AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.
Stats seems ok
Temperatures stable
Leakage current: ok
A systwm wide check gives same result as for the fist check on this shift at ~1am |
|
Fri May 13 06:47:33 2022, ML, 2h-Shift Checks
|
It has been a quiet night. AIDA has been stable.
Stats appears stable appear
Temperature also stable
Leakage current slowly dropping still.
System wide checks done. Status quo here again. |
|
Fri May 13 23:37:59 2022, ML, 2h-Shift Checks
|
Took over remotely from CB about an hour ago
At 0h40 (CET):
Stats still ok - attach 1
Leakage still OK - Attach 2
Temps - attach 3
Grafana OK
UCESB rates ok - attach 5
System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR |