 Tue Nov 5 15:19:35 2024, TD, S505 offline analysis R5_780 - R5_814 24x
|
Offline analysis of data files S505/R5_780 - R5_814 (corresponding to MBS data files 73-74)
Can find an analysis of alpha background runs at RIBF, RIKEN for comparison at https://elog.ph.ed.ac.uk/AIDA/816
Beam off - background runs without sources
MBS 73 27.6.22 09:33-11:33 CEST https://elog.gsi.de/despec/S505/204
MBS 74 27.6.22 11:34-13:34 CEST https://elog.gsi.de/despec/S505/206
MBS 75 27.6.22 13:35-16:29 CEST https://elog.gsi.de/despec/S505/209
ADC offsets per https://elog.ph.ed.ac.uk/DESPEC/556
FEE64 configuration
FEE64 a b c d
DSSSD#1 3 4 1 2
DSSSD#2 7 8 5 6
p+n junction FE64s odd numbered
Data analysis assumes
- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
- no clustering
- no p+n junction side - n+n Ohmic side correlation time gates
- valid LEC events
0 < p+n junction side multiplicity < 8
and
0 < n+n Ohmic side multiplicity < 8
Attachments 1-2 - per DSSSD p+n versus n+n multiplicity
Attachment 3 - per DSSSD x versus y
Attachments 4-5 - per DSSSD p+n versus n+n energy (20keV/channel nominal)
all combinations of per DSSSD p+n junction and n+n Ohmic energies
with projection of data within window onto x and y axes
too many events for natural (U decay series) background
off leading diagonal correlations anomalous
transverse width of leading diagonal correlation wider than expected - ADC offsets OK?
Attachment 6 - per FEE64 WR timestamp (32.768us/channel)
FEE64 sync test using pulser data - looks OK
Attachments 7-14 - per FEE64 ADC spectra (5.6keV/channel nominal)
note common x/y scale - pulser peak height proxy for peak width
per FEE64 1.8.L pulser peak widths (ch FWHM)
1 11.43
2 16.58
3 12.52
4 17.54
5 9.10
6 12.22
7 17.03
8 16.09
3 of 4 p+n junction FEE64 good (<70keV), 1 of 4 n+n Ohmic FEE64 good - all others < 100keV
Pulser peak indicates noise/gain/offset stable throughout background runs
Attachments 15-22 - per FEE64 ADC spectra (5.6kleV/channel nominal)
observe broad peak-like structures (c. 2.8MeV) in first channel of most ASICs?
Attachment 23 per DSSSD p+n versus n+n energy (20keV/channel nominal)
valid LEC events
p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1
Attachment 24 p+n FEE64 #0 energy versus each of n+n FEE64s (#1, #3, #5 and #7) energies (20keV/channel nominal)
valid LEC events
p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1
Attachments 23 & 24 do not unambiguously identify which FEE64s are attached to which DSSSD
|
|
Tue Apr 2 18:37:21 2024, TD, S505 ADC offsets
|
S505 ADC offsets using pulser walkthrough data from data file R1
ch = channel + ( module * 64 ) + ( range * 2048 )
adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 ) |
|
Tue Apr 13 14:24:48 2021, TD, S460 information
|
aida-gsi Anydesk 897170655
DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/
S460 shift plan - https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1097596683
Mattermost - https://mattermost.gsi.de/despec/channels/aida
DESPEC Zoom meeting
Zoom Meeting https://gsi-fair.zoom.us/j/94404952361
Passcode: s460 |
|
Fri Mar 5 23:45:41 2021, TD, S452 information
|
aida-gsi Anydesk 897170655
DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/
S452 shift checks - attachment 1
S452 shift plan - PDF attachment 2 - Google doc https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1745844766
Mattermost - https://mattermost.gsi.de/despec/channels/experiment-s452-nearline-sort
DESPEC Zoom meeting
Join Zoom Meeting
https://gsi-fair.zoom.us/j/98980822569
Meeting ID: 989 8082 2569
Passcode: S452
FRS Zoom meeting
Meeting-ID: 973 4809 5114
password: FRS-BT2021
Kenncode: 456657 |
|
Thu Mar 4 11:03:04 2021, NH, S452 Operating Notes
|
Please note the following "quirks" currently present in S452 that are important
a) Strips/Thresholds
Thresholds are 0xA, 0xA, 0x10 [Maybe changed depending on rate, etc]
Some strips are very noisy - damage?
b) Powering FEE64s
Due to a misbehaving USB hub(?) or cable only 7 FEE TTYs are monitored
aida04 and aida05 can take minutes to start, unsure why.
aida05 is monitored on nnrpi1 and hence when aida05 is ready it is safe to assume all 12 are ready
Watch the logs as normal
c) Slow control & nnrpi2
For the Grafana script to read the DSSD HV, it must be ran in a screen session
If nnrpi2 is restarted please run the HV as a screen by Putty SSHing (localhost 22)
and runningscreen -S caenhv /dev/ttyACM0 if screen was just closed you can do screen -x caenhv to reconnect
The pulser ideally should be started the samescreen -S pulser /dev/ttyUSB0 To allow remote changing by NH/GSI
The pulser is sent to all subsystems for correlation and should be kept on at 2 Hz for the experiment
IN the pulser putty you can check the settings by typingdisplay settings And general commands: help
The NIM crate can be monitored /dev/ttyUSB2 but you should be careful
It is very easy to turn off the NIM crate if you do
I run it in a tmux session with (tmux attach -r ) to have it "readonly"
There is a small script on nnrpi2 ./check_usbs.sh to see which ttyUSB is which device
As they can change if the pi is rebooted
d) Scaler Readout
If powercycled FEEs, enable the scaler in Local Controls page by setting BuTiS interface control to 0x3
To enable scaler readout after GOing the run please open the Correlation Control page
(Act on all FEE64s) and set control to 0x1
This has to be done every time the FEE64s are set to GOing
e) New Tape Server
For waveform readout in future experiments, the tape server is in "Expert" mode which is harder to use
Please read the "How to use" instructions by clicking the button on the bottom right
Or in a terminal cat /MIDAS/TclHttpd/Html/ETape/1.txt
f) GSI/MBS
If GSI team ask to restart AIDA MBS you may need to restart the MBS relay so it can reconnect
To start MBS in a terminalssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup
It should report it connected to AIDA-3 - if not check the MBS relay is running or restart the relay
Please check the MBS rate commandrate -nst -rst -nev -rev -ndata -rdata -st That the "Streams" column does not frequently (ever) have 0 empty streams
If so AIDA data may be getting dropped and should be looked into
If another DESPEC DAQ has failed this will eventually happen and is not a problem
But if it occurs in normal operation it is concerning
Do NOT stop the AIDA run without talking to GSI team - if no AIDA data the whole MBS DAQ stops
and blocks all other subsystems. It is best to keep AIDA ticking over constantly and let
MBS start/stop files when relevant.
Of course if things have to stop it is OK but inform GSI shifters so they can take action |
|
Tue Jun 14 12:15:33 2022, NH, S450 Archived
|
All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived |
|
Sun Oct 1 11:47:07 2023, TD, S4 cooling water
|
The photographs show the cooling water controls and temperature/pressure gauges outside S4 and the connections used by AIDA within S4. |
|
Mon Aug 5 12:57:48 2019, NH, S4 Conditions
|
Temp: 26 C
Humidity: 50%
Dew Point: ~15 C
There are two temperature dials outside S4, one shows exactly 20 C and one shows nearer 16 C.
Trying to clarify this discrepancy as 16 C is too cold at the moment.
In contact with developer of cooling system to confirm apparent 16 C after the heat exchanger
Today's conditions:
Temp: 26.9 C
Humidity: 48.7%
Dewpoint 15.2 C |
|
Sun Jul 14 16:52:28 2024, TD, S181 R7_38-44 7x
|
|
|
Sun Jul 14 14:37:53 2024, TD, S181 R6_375-380 6x
|
|
|
Thu Sep 8 12:31:25 2022, NH, Retrying AIDA DataAcq v10
|
Startup AIDA with ribbon cable connected to aida03 and aida07 for noise
Setup and run with waveforms enabled. Discriminators ADC power etc as default
Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to
aida03 - 320k
aida07 - 254k
Startup merger and observe rates
aida03 - 224k
aida07 - 213k
Rate drop observed as before.
Now update aidacommon to point to AidaExecV10 and powercycle FEEs
Rates again with 0x3
aida03 - 315k
aida07 - 252k
Restart with data transfer ON
aida03 - 317k
aida07 - 262k
No errors in merger terminal or "Merge time errors" statistic
Will keep running |
|
Fri Mar 22 13:47:32 2019, NH, Resolution Checks
|
Pulser widths of AIDA modules, resolution is fairly typical. AIDA09 still noisier than other DSSD channels.
Figure 1: Picture of pulser peaks
Figure 2: Raw waves (AIDA01 did not calibrate, so AIDA02 used instead as "non DSSD module")
Figure 3: Good Events
Fast comparator threshold (LEC/MEC) changed to 0xFF
AIDA09 seems noiser than before(?) but is known to be noisy.
AIDA01: 24.22
AIDA09: 205.21
AIDA10: 88.60
AIDA11: 63.33
AIDA12: 102.69 |
|
Mon Nov 4 13:17:12 2019, NH, Report: aida10 database corruption
|
Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.
Attached is corrupted database and correct database |
 Tue Nov 5 10:02:45 2019, NH PJCS, Report: aida10 database corruption
|
This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?
A quick look at this file and it looks normal , not all 0xad ?
Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program. That could well cause you problems with readout as it won't understand some of the new data.
Hope this helps.
| Quote: |
|
Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.
Attached is corrupted database and correct database
|
|
|
Tue Nov 5 10:32:21 2019, NH PJCS, Report: aida10 database corruption
|
Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.
The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.
| Quote: |
|
This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?
A quick look at this file and it looks normal , not all 0xad ?
Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program. That could welll cause you problems with readout as it won't understand some of the new data.
| Quote: |
|
Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.
Attached is corrupted database and correct database
|
|
|
|
Tue Nov 5 11:15:41 2019, NH PJCS, Report: aida10 database corruption
|
If ypou now check the actual values in the aida10 Options screen are the correct ?
It can happen that corruptions once loaded are propagated when Options are generally saved.
Looking at the dates the files were modified does that make sense to you. It doesn't look as if the corrupted database was changed except when they all were ? Can you check ? Have you checked the other Options files ? I seem to recall that RIKEN AIDA had a program to to this ?
| Quote: |
|
Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.
The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.
| Quote: |
|
This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?
A quick look at this file and it looks normal , not all 0xad ?
Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program. That could welll cause you problems with readout as it won't understand some of the new data.
| Quote: |
|
Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.
Attached is corrupted database and correct database
|
|
|
|
|
Mon Feb 17 09:48:34 2020, NH, Report: aida09 Kernel Panic & Lost WR
|
aida09 crashed over the weekend and automatically rebooted.
After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
Reset/Setup did not fix issue
Sync ASICs did not fix issue
GSI White Rabbit control page shows a correct WR timestamp
Attach 1: ttyUSB12 (aida09 log with kernel panic)
Attach 2: GSI White Rabbit control page
Attach 3: "Collect All WR Timestamps"
Attach 4: RAW Display for aida09
WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
c.f. "GSI page" timestamp starting 0x15F427F5
Attach 5: Timestamp shown by merger |
|
Wed Feb 19 10:58:29 2020, NH, PJCS, Report: aida09 Kernel Panic & Lost WR
|
> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
>
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
>
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
>
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5
>
> Attach 5: Timestamp shown by merger
Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a
logical "rollover" of the lower 14 bits of this 10nS timestamp.
Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?
It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.
The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.
If this occurs again then the system wide check results would be interesting. |
|
Mon Apr 22 09:03:54 2024, TD, Report: aida04 stops producing data
|
09.14 aida04 stopped producing data - system console output - attachment 1
It is not clear (to me at least) whether the cause is radiation-induced single event upsets or
the start/stop of LN2 fills.
Nor is it clear why the FEE64 stops
- WR timestamp error (data link to Merger blocks)
- AIDAExecV10 fails
- ?
rebooted aida04
DAQ RESET/SETUP etc cycle for all FEE64s
10.03 successful restart
current data file R15 |
|
Mon Mar 9 12:47:18 2020, NH, Report: Unusual AIDA Timestamp Error
|
Noted using my AIDA event builder code, a timewarp was reported and I noticed the ADC event had a different timestamp LSB to the "WR" markers preceding it, which I didn't think was possible in the merger as it stands
aida event 875015fa 0f68098c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68098c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68098c
aida event c1f6851b 0f68098c | ADC 08:54 L 851b
event time: 15fa4b164f68098c
aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68115c
aida event c1ef8238 0f68115c | ADC 08:47 L 8238
event time: 15fa4b164f68115c
aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68115c
aida event c1f884b0 0f68115c | ADC 08:56 L 84b0
event time: 15fa4b164f68115c
aida event 8a5015fa 0f681382 | INFO 5 SYNC6348 11 015fa
aida event 8a44b164 0f681382 | INFO 4 SYNC4828 11 4b164
event time: 15fa4b164f681382
aida event c2817dc8 0f681382 | ADC 11:01 L 7dc8
event time: 15fa4b164f681382
aida event 875015fa 0f68192c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68192c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68192c
aida event c1f9849c 0f68192c | ADC 08:57 L 849c
event time: 15fa4b164f68192c
aida event 815015fa 0f681f94 | INFO 5 SYNC6348 02 015fa
aida event 8144b164 0f681f94 | INFO 4 SYNC4828 02 4b164
event time: 15fa4b164f681f94
aida event c0568084 04b6854a | ADC 02:22 L 8084
event time: 15fa4b1644b6854a
AIDA Timewarp (15fa4b1644b6854a before 15fa4b164f68192c)
Event reported errors, skipping this file...
Events: 506459240 506459240 (0 errors)
Unsure if the cause, easily handled by ignoring such events but should be "impossible" |