 Wed May 22 20:10:19 2024, NH, CC, HMA, MV, ZC, KZ, AIDA Snout Remounted 9x
|
AIDA and the Snout have been remounted in S4
I had a check of the AIDA connectors and all looked good, the kapton PCB connectors were sitting comfortably above the interstage and I saw no stressed/loose connectors.
Multimeter showed no continuity between snout or inter-stage and AIDA ground
We put a single layer of wide black tape inside the end-cap to improve the light seal and minimise reflections
After mounting the snout I managed to bias both DSSDs to 100 V with no issues. I had to raise the DSSD#1 (Ch0) limit to 40 uA because it rises quite high during ramp up (then settles).
It is very hot in S4 (28 C) and the roof is closed for high-radiation experiments, so I expect it to get hotter. DSSD leakage currents were similar to before (20 and 15 uA)
There is also the new Temp and Humidity monitor, which you can view on AnyDesk at 192.168.11.119 (I think), there is a firefox window open already |
|
Thu May 9 09:08:26 2024, TD, Current NewMerger install - aida-gsi
|
[npg@aidas-gsi MIDAS]$ ls -l
total 524
drwxrwxr-x. 2 npg npg 156 Mar 17 2021 Aida
drwxrwxr-x. 2 npg npg 4096 Jul 13 2015 bin_Linux64
drwxrwxr-x. 4 npg npg 38 Jul 3 2018 config
drwxr-xr-x. 3 npg npg 22 Jan 23 2019 config_ORIG
drwxr-xr-x. 3 npg npg 17 Jan 23 2019 Data_Acq
drwxrwxr-x. 25 npg npg 4096 Aug 19 2020 DataPackage
drwxrwxr-x. 3 npg npg 25 Aug 29 2019 DB
drwxr-xr-x. 3 npg npg 25 Jan 23 2019 DB_ORIG
lrwxrwxrwx. 1 npg npg 14 May 30 2019 Embedded -> /home/Embedded
drwxrwxr-x. 2 npg npg 4096 Mar 17 2021 ISOL_Aida
drwxrwxr-x. 2 npg npg 4096 Nov 29 2018 ISOL_Aida_PreApr9
drwxrwxr-x. 7 npg npg 87 Jul 8 2009 lib
drwxrwxr-x. 2 npg npg 4096 Sep 19 2012 lib_Linux64
drwxr-xr-x. 5 npg npg 47 Jan 23 2019 Linux
drwxrwxr-x. 6 npg npg 58 Feb 22 2018 linux-ppc_4xx
drwxr-xr-x. 6 npg npg 58 Jan 23 2019 linux-ppc_4xx_ORIG
drwxrwxr-x. 3 npg npg 73 Feb 16 2022 log
-rwxrw-r--. 1 npg npg 12862 Jul 9 2009 midasicon.ico
-rwxrw-r--. 1 npg npg 773 Jul 23 2012 midasicon.png
-rwxrwxr-x. 1 npg npg 772 Jan 20 2019 MIDASNewGen_BuildAIDA.txt
-rwxrwxr--. 1 npg npg 2532 Jan 19 2019 MIDASNewGen_BuildCore.txt
drwxrwxr-x. 14 npg npg 4096 Feb 22 2018 MIDASpackages
drwxrwxr-x. 6 npg npg 124 May 14 2022 NewMerger
-rw-rw-r--. 1 npg npg 118873 Oct 18 2019 NewMergerdiff.log
-rw-rw-r--. 1 npg npg 118971 Oct 18 2019 NewMergerdiff.log~
drwxrwxr-x. 5 npg npg 52 Jan 8 2019 NewMerger_PreApr9
drwxrwxr-x. 6 npg npg 124 Mar 12 2020 NewMerger_PreMar21
drwxrwxr-x. 7 npg npg 104 Oct 17 2019 NewMerger_PreOct19
-rw-r--r--. 1 npg npg 230517 Mar 12 2020 NewMerger.tgz
drwxr-xr-x. 14 npg npg 4096 Jan 23 2019 OLD
drwxr-xr-x. 3 npg npg 17 Jan 23 2019 SOAP
drwxrwxrwx. 4 npg npg 67 Aug 19 2021 SpecData
drwxrwxr-x. 3 npg npg 21 Nov 15 2019 TapeServer
drwxr-xr-x. 3 npg npg 21 Jan 23 2019 TapeServer_PreFeb21
drwxrwxr-x. 3 npg npg 21 Nov 15 2019 TapeServer_PreMar21
drwxr-xr-x. 2 npg npg 44 Feb 2 2021 tcl
drwxr-xr-x. 12 npg npg 152 Jan 23 2019 TclHttpd
drwxrwxr-x. 11 npg npg 155 Oct 26 2017 TclTkpackages
lrwxrwxrwx. 1 npg npg 26 May 30 2019 XilinxLinux -> /home/Embedded/XilinxLinux
[npg@aidas-gsi NewMerger]$ ls -lR
.:
total 32
-rw-rw-r--. 1 npg npg 15096 Apr 16 2019 Aida_ASICreadout.c
drwxrwxr-x. 3 npg npg 23 Oct 18 2019 Html
drwxrwxr-x. 2 npg npg 172 May 14 2022 include
drwxrwxr-x. 8 npg npg 256 May 14 2022 MergeServer
-rw-rw-r--. 1 npg npg 11048 Aug 29 2019 R3B_readout.c
-rw-rw-r--. 1 npg npg 152 Oct 17 2019 README
drwxrwxr-x. 2 npg npg 34 Apr 29 2021 tcl
./Html:
total 0
drwxrwxr-x. 6 npg npg 187 Oct 13 2020 NewMerger
./Html/NewMerger:
total 60
lrwxrwxrwx. 1 npg npg 21 Oct 18 2019 Html -> /MIDAS/NewMerger/Html
drwxrwxr-x. 2 npg npg 6 Aug 26 2019 LinkRates
-rw-rw-r--. 1 npg npg 1210 Aug 26 2019 NewMerger.js
-rw-rw-r--. 1 npg npg 9417 Aug 29 2019 NewMerger.tcl
-rw-rw-r--. 1 npg npg 9332 Aug 26 2019 NewMerger.tcl~
-rw-rw-r--. 1 npg npg 13052 Aug 27 2019 NewMerger.tml
-rw-rw-r--. 1 npg npg 13851 Aug 26 2019 NewMerger.tml~
drwxrwxr-x. 2 npg npg 82 Oct 13 2020 Options
drwxrwxr-x. 2 npg npg 102 Oct 13 2020 RunControl
drwxrwxr-x. 2 npg npg 192 Aug 29 2019 Stats
./Html/NewMerger/LinkRates:
total 0
./Html/NewMerger/Options:
total 24
-rwxrwxr-x. 1 npg npg 725 Aug 26 2019 Options.js
-rwxrwxr-x. 1 npg npg 3934 Aug 26 2019 Options.tcl
-rwxrwxr-x. 1 npg npg 7864 Aug 27 2019 Options.tml
-rw-rw-r--. 1 npg npg 7864 Aug 27 2019 Options.tml~
./Html/NewMerger/RunControl:
total 44
-rw-rw-r--. 1 npg npg 4791 Aug 26 2019 implementation.tcl
-rw-rw-r--. 1 npg npg 1643 Aug 26 2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg 539 Aug 26 2019 stats.tcl
-rw-rw-r--. 1 npg npg 12328 Aug 29 2019 sys.tcl
-rw-rw-r--. 1 npg npg 12231 Aug 26 2019 sys.tcl~
./Html/NewMerger/Stats:
total 52
-rw-rw-r--. 1 npg npg 2880 Aug 26 2019 MergeStatsALL.tcl
-rw-rw-r--. 1 npg npg 1013 Aug 26 2019 MergeStats.js
-rw-rw-r--. 1 npg npg 1534 Aug 26 2019 MergeStatsSINGLE.tcl
-rw-rw-r--. 1 npg npg 4259 Aug 26 2019 MergeStats.tcl
-rw-rw-r--. 1 npg npg 8467 Aug 27 2019 MergeStats.tml
-rw-rw-r--. 1 npg npg 8338 Aug 26 2019 MergeStats.tml~
-rw-rw-r--. 1 npg npg 1829 Aug 29 2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg 1943 Aug 29 2019 stats.defn.tcl~
./include:
total 52
-rw-rw-r--. 1 npg npg 1079 Apr 16 2019 buffer.h
-rw-rw-r--. 1 npg npg 4473 Apr 16 2019 common.h
-rw-rw-r--. 1 npg npg 1199 Jul 16 2019 merge.h
-rw-rw-r--. 1 npg npg 1273 Apr 16 2019 message.h
-rw-rw-r--. 1 npg npg 1273 Oct 17 2019 message.h~
-rw-rw-r--. 1 npg npg 5288 Aug 29 2019 netvar.h
-rw-rw-r--. 1 npg npg 5256 Apr 16 2019 netvar.h~
-rw-rw-r--. 1 npg npg 912 Apr 16 2019 services.h
-rw-rw-r--. 1 npg npg 886 Apr 18 2019 stats.h
-rw-rw-r--. 1 npg npg 2710 Apr 16 2019 transfer.h
./MergeServer:
total 60
-rw-rw-r--. 1 npg npg 2606 Apr 16 2019 alloc_data_area.c
drwxr-xr-x. 2 npg npg 69 Aug 27 2019 bin64
drwxrwxr-x. 2 npg npg 128 Apr 6 2021 bin64.BackUp
-rw-rw-r--. 1 npg npg 10510 Sep 19 2019 buffer.c
-rwxrwxr-x. 1 npg npg 305 Apr 16 2019 build
drwxrwxr-x. 3 npg npg 279 May 14 2022 link
-rw-rw-r--. 1 npg npg 494 Apr 16 2019 Makefile
drwxrwxr-x. 3 npg npg 160 Apr 6 2021 master
drwxrwxr-x. 3 npg npg 143 Apr 6 2021 merge
-rw-rw-r--. 1 npg npg 3225 Sep 19 2019 message.c
-rw-rw-r--. 1 npg npg 3199 Sep 19 2019 netvar.c
-rwxrwxr-x. 1 npg npg 397 Aug 26 2019 run
-rw-rw-r--. 1 npg npg 15309 Sep 19 2019 signal.c
drwxrwxr-x. 3 npg npg 101 Sep 19 2019 statrate
-rw-rw-r--. 1 npg npg 1548 Sep 19 2019 stats.c
-rw-rw-r--. 1 npg npg 593 Apr 16 2019 version.c
./MergeServer/bin64:
total 148
-rwxr-xr-x. 1 npg npg 51964 May 11 2021 link64
-rwxr-xr-x. 1 npg npg 33312 May 11 2021 master64
-rwxr-xr-x. 1 npg npg 36027 May 11 2021 merge64
-rwxr-xr-x. 1 npg npg 21370 May 11 2021 statrate64
./MergeServer/bin64.BackUp:
total 256
-rwxr-xr-x. 1 npg npg 51948 Mar 16 2021 link64
-rwxrwxr-x. 1 npg npg 33184 Apr 6 2021 master64
-rwxr-xr-x. 1 npg npg 33312 Mar 16 2021 master64.new
-rwxr-xr-x. 1 npg npg 36027 Mar 16 2021 merge64
-rwxr-xr-x. 1 npg npg 35368 Mar 9 2021 merge64.new
-rwxrwxr-x. 1 npg npg 35910 Mar 8 2021 merge64.orig
-rwxr-xr-x. 1 npg npg 21370 Mar 16 2021 statrate64
./MergeServer/link:
total 164
-rw-rw-r--. 1 npg npg 10216 Oct 18 2019 handle_buffer.c
-rw-rw-r--. 1 npg npg 24623 Sep 19 2019 link_server.c
-rw-rw-r--. 1 npg npg 24732 Oct 18 2019 link_server.c~
drwxrwxr-x. 2 npg npg 217 Oct 24 2019 Linux64
-rw-rw-r--. 1 npg npg 7051 Sep 19 2019 main.c
-rw-rw-r--. 1 npg npg 7029 Oct 18 2019 main.c~
-rw-rw-r--. 1 npg npg 14464 Sep 19 2019 processAIDADATAbuffer.c
-rw-rw-r--. 1 npg npg 14463 Oct 18 2019 processAIDADATAbuffer.c~
-rw-rw-r--. 1 npg npg 14950 Sep 19 2019 processR3BDATAbuffer.c
-rw-rw-r--. 1 npg npg 14925 Oct 18 2019 processR3BDATAbuffer.c~
-rw-rw-r--. 1 npg npg 15077 Aug 22 2019 processR3BDATAbuffer.c_PreAug22
./MergeServer/link/Linux64:
total 176
-rw-rw-r--. 1 npg npg 3088 Oct 24 2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg 6216 Oct 24 2019 buffer.o
-rw-rw-r--. 1 npg npg 29840 Oct 24 2019 handle_buffer.o
-rwxrwxr-x. 1 npg npg 50660 Oct 24 2019 link64
-rw-rw-r--. 1 npg npg 24464 Oct 24 2019 link_server.o
-rw-rw-r--. 1 npg npg 14864 Oct 24 2019 main.o
-rw-rw-r--. 1 npg npg 795 Aug 22 2019 Makefile
-rw-rw-r--. 1 npg npg 776 Apr 16 2019 Makefile~
-rw-rw-r--. 1 npg npg 5264 Oct 24 2019 message.o
-rw-rw-r--. 1 npg npg 12848 Oct 24 2019 signal.o
-rw-rw-r--. 1 npg npg 3480 Oct 24 2019 stats.o
-rw-rw-r--. 1 npg npg 1864 Oct 24 2019 version.o
./MergeServer/master:
total 40
-rw-rw-r--. 1 npg npg 7850 Sep 18 2019 handle_signals.c
drwxrwxr-x. 2 npg npg 223 Oct 24 2019 Linux64
-rw-rw-r--. 1 npg npg 6952 Sep 19 2019 main.c
-rw-rw-r--. 1 npg npg 6973 Sep 19 2019 main.c~
-rw-rw-r--. 1 npg npg 1722 Aug 29 2019 start_link.c
-rw-rw-r--. 1 npg npg 1885 Aug 22 2019 start_link.c~
-rw-rw-r--. 1 npg npg 1372 Apr 16 2019 start_merge.c
-rw-rw-r--. 1 npg npg 1345 Apr 16 2019 start_statrate.c
./MergeServer/master/Linux64:
total 120
-rw-rw-r--. 1 npg npg 3408 Oct 24 2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg 4168 Oct 24 2019 buffer.o
-rw-rw-r--. 1 npg npg 22840 Oct 24 2019 main.o
-rw-rw-r--. 1 npg npg 793 Jul 16 2019 Makefile
-rwxrwxr-x. 1 npg npg 33184 Oct 24 2019 master64
-rw-rw-r--. 1 npg npg 5264 Oct 24 2019 message.o
-rw-rw-r--. 1 npg npg 14616 Oct 24 2019 signal.o
-rw-rw-r--. 1 npg npg 3408 Oct 24 2019 start_link.o
-rw-rw-r--. 1 npg npg 2904 Oct 24 2019 start_merge.o
-rw-rw-r--. 1 npg npg 2808 Oct 24 2019 start_statrate.o
-rw-rw-r--. 1 npg npg 3480 Oct 24 2019 stats.o
-rw-rw-r--. 1 npg npg 1864 Oct 24 2019 version.o
./MergeServer/merge:
total 48
drwxrwxr-x. 2 npg npg 228 Oct 24 2019 Linux64
-rw-rw-r--. 1 npg npg 5130 Sep 19 2019 main.c
-rw-rw-r--. 1 npg npg 4579 Oct 24 2019 mergeAIDA.c
-rw-rw-r--. 1 npg npg 4119 Oct 17 2019 #merge.c#
-rw-rw-r--. 1 npg npg 4148 Oct 21 2019 merge.c
-rw-rw-r--. 1 npg npg 3703 Sep 19 2019 mergeR3B.c
-rw-rw-r--. 1 npg npg 8060 Sep 17 2019 merge_server.c
-rw-rw-r--. 1 npg npg 2394 Sep 17 2019 storage.c
./MergeServer/merge/Linux64:
total 120
-rw-rw-r--. 1 npg npg 3088 Oct 24 2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg 3872 Oct 24 2019 buffer.o
-rw-rw-r--. 1 npg npg 10944 Oct 24 2019 main.o
-rw-rw-r--. 1 npg npg 789 Jul 16 2019 Makefile
-rw-rw-r--. 1 npg npg 776 Apr 16 2019 Makefile~
-rwxrwxr-x. 1 npg npg 35910 Oct 24 2019 merge64
-rw-rw-r--. 1 npg npg 9064 Oct 24 2019 merge.o
-rw-rw-r--. 1 npg npg 8160 Oct 24 2019 merge_server.o
-rw-rw-r--. 1 npg npg 5264 Oct 24 2019 message.o
-rw-rw-r--. 1 npg npg 12744 Oct 24 2019 signal.o
-rw-rw-r--. 1 npg npg 3480 Oct 24 2019 stats.o
-rw-rw-r--. 1 npg npg 4048 Oct 24 2019 storage.o
-rw-rw-r--. 1 npg npg 1864 Oct 24 2019 version.o
./MergeServer/statrate:
total 16
drwxrwxr-x. 2 npg npg 161 Oct 24 2019 Linux64
-rw-rw-r--. 1 npg npg 1876 Sep 19 2019 main.c
-rw-rw-r--. 1 npg npg 1717 Sep 19 2019 main.c~
-rw-rw-r--. 1 npg npg 2645 Apr 16 2019 statrate_server.c
-rw-rw-r--. 1 npg npg 2645 Apr 16 2019 statrate_server.c~
./MergeServer/statrate/Linux64:
total 80
-rw-rw-r--. 1 npg npg 8880 Oct 24 2019 main.o
-rw-rw-r--. 1 npg npg 725 Apr 16 2019 Makefile
-rw-rw-r--. 1 npg npg 728 Apr 16 2019 Makefile~
-rw-rw-r--. 1 npg npg 5376 Oct 24 2019 message.o
-rw-rw-r--. 1 npg npg 10144 Oct 24 2019 signal.o
-rwxrwxr-x. 1 npg npg 21370 Oct 24 2019 statrate64
-rw-rw-r--. 1 npg npg 4664 Oct 24 2019 statrate_server.o
-rw-rw-r--. 1 npg npg 3480 Oct 24 2019 stats.o
-rw-rw-r--. 1 npg npg 1872 Oct 24 2019 version.o
./tcl:
total 4
-rw-rw-r--. 1 npg npg 1775 Aug 26 2019 NewMergerControl.tcl |
|
Tue May 7 13:25:00 2024, JB, CC, NH, HMA, Tuesday 6 May - dismount snout
|
Status before dismount
14:26 DSSSD#1 119.99 V @ 17.968 uA
DSSSD#2 119.95 V @ 11.212 uA
Leakage currents slightly elevated from Monday.
S4 temperature at 26 °C.
14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.
Plan:
- Dismount snout.
- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.
|
|
Mon May 6 07:10:26 2024, TD, Monday 6 May
|
08.11 DSSSD#1 bias -120V leakage current -17.0uA
DSSSD#2 bias -120V leakage current -10.5uA
S4 temperature 25 deg C
cf. 04.00 30.4.24 S4 temperature 25 deg C
DSSSD#1 -21.9uA
DSSSD#2 -13.0uA
Leakage currents currently c. 80% of 30.4.24 values at same temperature |
|
Sun May 5 10:22:24 2024, TD, Sunday 5 May
|
11.22 DSSSD#1 bias -120V leakage current -17.1uA
DSSSD#2 bias -120V leakage current -10.6uA
S4 temperature 25 deg C
cf. 07.00 29.4.24 S4 temperature 25 deg C
DSSSD#1 -21.8uA
DSSSD#2 -12.8uA
Leakage currents currently c. 80% of 29.4.24 values at same temperature |
|
Sat May 4 07:36:07 2024, TD, Saturday 4 May 11x
|
08.32 DSSSD bias & leakage currents - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC, correlation scalers data item stats OK attachments 4-5
11/16 <20k max aida16 174k
per FEE64 Rate spectra - attachmnt 6
09.44 per p+n FEE64 1.8.L spectra - attachment 7
aida09 pulser peak width 58 ch FWHM
per FEE64 1.8.W spectra - attachments 8-10
09.47 bias cycle - off and then on
no significant change in leakage current
09.57 FEE64 power OFF
DSSSD bias ON |
|
Thu May 2 23:08:43 2024, TD, Friday 3 May
|
00.05 FEE64 temperatures OK
ADC data item stats OK
09.31 FEE64 temperatures OK
ADC data item stats OK
13.15 FEE64 temperatures OK
ADC data item stats OK
9/16 <20k max aida16 164k
20.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida03 no data - DAQ reset
9/16 <20k max aida16 158k
23.15 FEE64 temperatures OK
ADC data item stats
aida03 no data - DAQ reset
10/16 <20k max aida16 153k |
|
Thu May 2 00:16:22 2024, TD, NH, Thursday 2 May
|
01.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
6/16 < 20k
10.25 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 restarted
7/16 < 20k, max 220k aida16
all histograms zero'd
13.20 per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 59 ch FWHM
FEE64 temperatures OK
ADC data item stats
aida04 no data
5/16 < 20k, max 332k aida08
14.37
Moved Time Machine delayed scaler from aida04 to aida07
aida04 no data
Reset merger etc
aida05 no data
Reset merger etc
All FEEs with data again
S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C
17.26 FEE64 temperatures OK
ADC data item stats OK
19.28 FEE64 temperatures OK
ADC data item stats
aida02 no data - DAQ reset
|
|
Wed May 1 08:59:15 2024, TD, Wednesday 1 May
|
04.55 FEE64 temperatures OK
ADC data item stats OK *except* aida04 no data
TapeServer no storage mode
09.55 FEE64 temperatures OK
ADC data item stats OK *except* aida02 & aida04 no data
DSSSD bias and leakage current - Grafana - attachment 1
19.28 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
noted increase in rates for aid08 and aida16
5/16 < 20k |
|
Tue Apr 30 09:06:45 2024, TD, NH, Tuesday 30 April 7x
|
10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
9/16 FEE64s < 20k
per FEE64 Rate spectra - attachment 5
Merger etc - attachments 6-7
aida02, aida04 inactive
11.20 DAQ restarted
aida02 and aida04 now producing data
11.30 analysis data file R22_313
all deadtimes <1%
no timewarps
HEC data item rate 560Hz (all from one channel of aida08 asic #3)
aida03 & aida04 MBS rates c. 24kHz
14.32 FEE64 temperatures OK
ADC data item stats OK
all FEE64s producing data
14.43 all histograms & stats zero'd
[NH checks here]
21.26 aida04 stopped forwarding to merger
the "quick fix" (stop merger, rerun) worked
MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability
System Wide Checks
clocks PASSED
ADC IGNORED
WR:
Base Current Difference
aida02 fault 0xe4d8 : 0x65c0 : -32536
aida03 fault 0x2371 : 0x2380 : 15
aida04 fault 0xa210 : 0xaf58 : 3400
aida06 fault 0x3624 : 0x3626 : 2
White Rabbit error counter test result: Passed 12, Failed 4
FPGA:
Base Current Difference
aida02 fault 0xaa3b : 0xc455 : 6682
aida04 fault 0x5a4 : 0x6ba : 278
aida06 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1
Baselines reset
22.50 FEE64 temps ok
merger ok
all links active
ADC data item stats ok |
|
Mon Apr 29 08:11:16 2024, TD, Monday 29 April 41x
|
08.52 Beam off
09.11 New data file R22
Monitor accumulated implanted activity in bPlas and DSSSDs
ASIC settings 2024Apr26-01.29.53
DSSSD bias & leakage current OK - attachments 1-3
FEE64 temperatures OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
Merger etc - attachment 7
analysis data file R22_19 - attachment 8
all deadtimes 0.3%, or less
no timewarps
HEC data item rate 390Hz
10.00 all histograms & stats zero'd
13.24 DSSSD bias & leakage current OK - attachments 9-10
FEE64 temperatures OK - attachment 11
ADC data item stats - attachment 12
per FEE64 Rate spectra - attachment 13
per p+n FEE64 1.8.L spectra - attachments 14-16
common x/y axes - peak height proxy for width
aida09 pulser peak width 58 ch FWHM (~41keV FWHM) cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
electronic noise today very similar to electronic noise Monday 22 April
per FEE64 1.8.W spectra - 20us FSR - attachments 17-19
Merger etc - attachment 20
analysis data file R22_60 - attachment 21
max deadtime 1.7% (aida11), all others <1%
no timewarps
HEC data item rate 710Hz
14.13 ucesb - attachment 22
14.15 aida04 not producing data
14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
15.27 *all* FEE64s stop producing data - reason unknown
16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
16.29 DSSSD bias & leakage current OK - attachments 23-24
FEE64 temperatures OK - attachment 25
ADC data item stats - attachment 26
per FEE64 Rate spectra - attachment 27
Merger etc - attachment 28
20.45 DSSSD bias & leakage current OK - attachments 29-30
FEE64 temperatures OK - attachment 32
ADC data item stats - attachment 31
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
20.54 analysis data file R22_145 - attachment 35
max deadtime 1.5% (aida11), all others <1%
no timewarps
HEC data item rate 775Hz
03.45 DSSSD bias & leakage current OK - attachments 36-37
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41 |
|
Sun Apr 28 22:52:06 2024, Marc, 0:00-08:00 Monday 29 April 7x
|
Just another manic magic Monday, ....
Taking over Betool's AIDA shift. All is good.
This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.
01:15: AIDA04 is down again. DAQ shift crew informed.
01:27: AIDA04 is back and shift crew has been informed.
01:30 Checks:
Bias and leakage current - Attachment 1-2
Temperatures - Attachment 3
ADC Data item - Attachment 4
Rate Spectra - Attachment 5
UCESB - Attachment 6
Timestamp checks - Attachement 7
06:00 It has run very smoothly.
|
|
Sun Apr 28 20:35:57 2024, Betool Alayed, 16:0-00:00 28 April 2024 10x
|
21:30 screenshots
and
23:24 screenshots |
|
Sun Apr 28 16:26:47 2024, Betool Alayed, 16:00-00:00 Sunday 28 April 10x
|
19:24 pm screenshots
and
17:18 pm screenshots |
|
Sun Apr 28 16:07:59 2024, TD, [How To] Recover DAQ when one FEE64 stops sending data - short(er) version
|
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
6) Start DAQ
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
8) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
9) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
10) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance. |
|
Sun Apr 28 15:49:58 2024, TD, DAQ restart short(er) cut
|
16.45 aida04 stopped producing data
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
aida04 did not stop, all other FEE64s stopped
restart Merger (workspace 3, display 2, select 'NewMerger' icon)
SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
GO Merger (workspace 3, display 1, browser 'NewMerger' tab)
aida04 now stops
disable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
enable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
|
|
Sun Apr 28 15:09:34 2024, Betool Alayed, 16:0-00:00 28 April 2024
|
16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
|
Sun Apr 28 06:57:59 2024, Magda Satrazani, 08:00-16:00 Sunday 28 April 36x
|
Took over the shift from Marc.
10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19
Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD
10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).
Grafana (AIDA) is now reporting data correctly too (attachment 21).
Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )
Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.
Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.
14.45 analysis data file R21_500 - attachment 36
max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%
no timewarps
HEC data item rate 2.7kHz
|
|
Sat Apr 27 23:04:42 2024, Marc, 00:00-08:00 Sunday 28 April 12x
|
Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C
0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.
After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.
I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.
I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.
We are now back in business with AIDA back in main DAQ at 01:40
01:45 full checks:
Bias: AIDA-Graphana - Attachment 1
Leakage current ramping up again ! S4 temperaturenot really going down either: 25C
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 5
ucesb - attachment 6
03:00 full checks
No change to be highlighted. All pretty much the same as during previous full checks.
05:00 full checks
Expt still running very smoothly.
Bias: AIDA-Graphana - Attachment 7-8
Leakage current going down slightly
FEE64 temperatures OK - attachment 9
ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
per FEE64 Rate spectra - attachment 11
Merger etc - attachment 12
ucesb - attachment 13
07:15 all running smoothly.
07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.
07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.
|
|
Sat Apr 27 15:13:06 2024, Norah, 16:00 - 00:00 Saturday 27 April 15x
|
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |