 Mon Mar 9 12:47:18 2020, NH, Report: Unusual AIDA Timestamp Error
|
Noted using my AIDA event builder code, a timewarp was reported and I noticed the ADC event had a different timestamp LSB to the "WR" markers preceding it, which I didn't think was possible in the merger as it stands
aida event 875015fa 0f68098c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68098c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68098c
aida event c1f6851b 0f68098c | ADC 08:54 L 851b
event time: 15fa4b164f68098c
aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68115c
aida event c1ef8238 0f68115c | ADC 08:47 L 8238
event time: 15fa4b164f68115c
aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68115c
aida event c1f884b0 0f68115c | ADC 08:56 L 84b0
event time: 15fa4b164f68115c
aida event 8a5015fa 0f681382 | INFO 5 SYNC6348 11 015fa
aida event 8a44b164 0f681382 | INFO 4 SYNC4828 11 4b164
event time: 15fa4b164f681382
aida event c2817dc8 0f681382 | ADC 11:01 L 7dc8
event time: 15fa4b164f681382
aida event 875015fa 0f68192c | INFO 5 SYNC6348 08 015fa
aida event 8744b164 0f68192c | INFO 4 SYNC4828 08 4b164
event time: 15fa4b164f68192c
aida event c1f9849c 0f68192c | ADC 08:57 L 849c
event time: 15fa4b164f68192c
aida event 815015fa 0f681f94 | INFO 5 SYNC6348 02 015fa
aida event 8144b164 0f681f94 | INFO 4 SYNC4828 02 4b164
event time: 15fa4b164f681f94
aida event c0568084 04b6854a | ADC 02:22 L 8084
event time: 15fa4b1644b6854a
AIDA Timewarp (15fa4b1644b6854a before 15fa4b164f68192c)
Event reported errors, skipping this file...
Events: 506459240 506459240 (0 errors)
Unsure if the cause, easily handled by ignoring such events but should be "impossible" |
|
Mon Mar 16 07:25:24 2020, NH, End of Experiment
|
08:25 Beam Stopped, end of S480
Tape Server stopped at R12_2097
Started again in no storage mode
Will leave compressing runs |
|
Tue Jul 7 08:04:57 2020, NH, Power failure in Messhuette
|
Last week there was a power failure in Messhuette (FRS control room) - AIDA-3 and MBS were affected.
AIDA itself and S4 equipment is on UPS and was not shut down.
AIDA-3 has been restarted and MIDAS restarted. All seems OK
S4 conditions: 24.2 C / 33.5% RH / Td = 7.1 C |
|
Tue Jul 7 08:50:27 2020, NH, nnrpi2 does not boot
|
MIDAS was not working properly on nnrpi2 - power cycled. Now system does not boot properly, it gets stuck trying to login claiming "PAM critical error" - probable file system corruption?
Will investigate if SD card can be checked and repaired |
|
Tue Oct 6 15:48:06 2020, NH, nnrpi kernel update
|
FYI:
The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were *huge* and filled with the same error many times a second
Oct 4 15:01:22 raspberrypi kernel: [1030048.603846]
Oct 4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.607742]
Oct 4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.667755]
Oct 4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.667848]
Oct 4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.731753]
A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version.
now the message doesn't seem to appear so hopefully the pis won't break so rapidly. |
Tue Nov 24 13:10:42 2020, NH, Waveforms with frame grounded 
|
AIDA Frame is now grounded to platform (will add photo later)
Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now) |
|
Mon Feb 1 15:19:07 2021, NH, AIDA Status Feb 2021
|
01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.
For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
The system has been powered up and biased successfully. To be monitored over the next few days.
A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently |
 Sat Feb 6 10:45:55 2021, NH, AIDA Status Feb 2021
|
> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
>
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
>
> The system has been powered up and biased successfully. To be monitored over the next few days.
>
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
>
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
>
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
>
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently
05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.
The hose has arrived and the water line will be prepared in my office mid next week.
All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)
A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.
The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.
There will be a DESPEC Dry Run next week Mon-Tue. |
|
Fri Feb 12 17:31:04 2021, NH, Background run
|
A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21
Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64
S4 access will be limited.
I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal. |
|
Mon Feb 22 18:18:47 2021, NH, AIDA Rates/Situation 8x
|
The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.
(All data below taken with thresholds 0xA for all)
Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"
Leakage currents high but S4 very warm at the moment as roof is still closed.
Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.
Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done
Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)
aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3 |
|
Sat Feb 27 12:47:57 2021, NH, AIDA Rates/Situation 10x
|
Updated figures as of 27 Feb 2021 - S4 is now closed for experiments but may open on Wednesday.
Snout has been grounded and all FEEs are seemingly grounded properly.
Figure 1: Good Events
Figure 2: Rate Histograms (Log-y)
Figure 3: Waveforms (Odd)
Figure 4: Waveforms (Even)
Figure 5: Waveforms @ 5MS/s (Odd)
Figure 6: Waveforms @ 5MS/s (Even)
Figure 7: Waveforms with Pulser (Odd)
Figure 8: Waveforms with Pulser (Even)*
Figure 9: Waveforms with Pulser @ 5MS/s (Odd)
Figure 10: Waveforms with Pulser @ 5MS/s (Even)*
* y-Scale enlarged from 0-10000 to 0-15000 to capture top of pulse |
|
Thu Mar 4 11:03:04 2021, NH, S452 Operating Notes
|
Please note the following "quirks" currently present in S452 that are important
a) Strips/Thresholds
Thresholds are 0xA, 0xA, 0x10 [Maybe changed depending on rate, etc]
Some strips are very noisy - damage?
b) Powering FEE64s
Due to a misbehaving USB hub(?) or cable only 7 FEE TTYs are monitored
aida04 and aida05 can take minutes to start, unsure why.
aida05 is monitored on nnrpi1 and hence when aida05 is ready it is safe to assume all 12 are ready
Watch the logs as normal
c) Slow control & nnrpi2
For the Grafana script to read the DSSD HV, it must be ran in a screen session
If nnrpi2 is restarted please run the HV as a screen by Putty SSHing (localhost 22)
and runningscreen -S caenhv /dev/ttyACM0 if screen was just closed you can do screen -x caenhv to reconnect
The pulser ideally should be started the samescreen -S pulser /dev/ttyUSB0 To allow remote changing by NH/GSI
The pulser is sent to all subsystems for correlation and should be kept on at 2 Hz for the experiment
IN the pulser putty you can check the settings by typingdisplay settings And general commands: help
The NIM crate can be monitored /dev/ttyUSB2 but you should be careful
It is very easy to turn off the NIM crate if you do
I run it in a tmux session with (tmux attach -r ) to have it "readonly"
There is a small script on nnrpi2 ./check_usbs.sh to see which ttyUSB is which device
As they can change if the pi is rebooted
d) Scaler Readout
If powercycled FEEs, enable the scaler in Local Controls page by setting BuTiS interface control to 0x3
To enable scaler readout after GOing the run please open the Correlation Control page
(Act on all FEE64s) and set control to 0x1
This has to be done every time the FEE64s are set to GOing
e) New Tape Server
For waveform readout in future experiments, the tape server is in "Expert" mode which is harder to use
Please read the "How to use" instructions by clicking the button on the bottom right
Or in a terminal cat /MIDAS/TclHttpd/Html/ETape/1.txt
f) GSI/MBS
If GSI team ask to restart AIDA MBS you may need to restart the MBS relay so it can reconnect
To start MBS in a terminalssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup
It should report it connected to AIDA-3 - if not check the MBS relay is running or restart the relay
Please check the MBS rate commandrate -nst -rst -nev -rev -ndata -rdata -st That the "Streams" column does not frequently (ever) have 0 empty streams
If so AIDA data may be getting dropped and should be looked into
If another DESPEC DAQ has failed this will eventually happen and is not a problem
But if it occurs in normal operation it is concerning
Do NOT stop the AIDA run without talking to GSI team - if no AIDA data the whole MBS DAQ stops
and blocks all other subsystems. It is best to keep AIDA ticking over constantly and let
MBS start/stop files when relevant.
Of course if things have to stop it is OK but inform GSI shifters so they can take action |
|
Tue Mar 30 13:21:50 2021, NH, nnrpi1 (tty logs)
|
We had issues with all TTYs reading out on nnrpi1, I have moved them from an apparently faulty USB hub to a new one.
Trying to reboot the pi to detect the new USB ports and it didn't recover from boot
Trying to repair SD card but I get a lot of errors on my laptop, indications are the SD card is completely dead
A new SD card will be ordered and nnrpi1 restored from the original images
A new USB hub should also be organised as it seems one is faulty (pi hangs if 3 or more USB devices connected to it) |
|
Fri Apr 9 16:40:36 2021, NH, nnrpi1 Update
|
nnrpi1 is now back in S4 with a new SD card and new raspberry pi.
The freezing originally continued (it was the USB system failing) but was fixed with a change to /boot/cmdline.txt coherent_pool=4M
It seems the kernel update to fix one bug introduced another...
Now it is running again with all TTYs connected. I think one TTY may not be working but 11 should be |
|
Mon Apr 12 18:36:31 2021, NH, [HowTo] AIDA & MBS
|
Some instructions on AIDA and MBS and troubleshooting are available here
https://sf.gsi.de/lib/c9ba0b4c-26bc-43d0-9b49-213fc594610f/file/AIDA%20and%20MBS%20Tips%20and%20Troubleshooting.pdf |
|
Thu Apr 15 12:23:22 2021, NH, Fast Discriminator Masks
|
It is possible to get good rates with the fast discriminators with a few hot channels mask in aida03 and aida10
Threshold is 0x20 = 3.2 MeV
They can be tested in beam to see performance, especially note the first setting will involve alpha emitters to check in AIDA
All FEEs:
AS01 01, AS3 07 10 13 are masked
aida03:
AS0 12 and 14 are masked in addition
aida10:
AS1 10, 11 and AS2 05, 06 are masked in addiction |
|
Fri Apr 16 02:15:12 2021, NH, Implants/Punchthrough
|
Rate with beam < 1000 Hz still
Not Uranium but some fragments possibly produced in the thick 9g degrader
S4 slits are +/- 20mm but these punchthrough and cover the +/- 40mm of AIDA |
|
Sat Apr 24 10:21:06 2021, NH, Saturday 24.04.2021
|
I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again
Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)
System Wide Checks
Clocks: OK
FADC: OK
WR:
Base Current Difference
aida05 fault 0x31a2 : 0x31a2 : 0
aida05 : WR status 0x10
aida06 fault 0x22e7 : 0x22e8 : 1
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK
aida05 ASIC maybe didn't complete setup properly (WR 0x10?)
Timestamps: OK (1)
Temps: OK
Will start gzipping S460 runs for migration to lustre & tape |
|
Wed Apr 28 10:06:11 2021, NH, Wednesday 28 April
|
11.05 All system wide checks OK *except*
WR
Base Current Difference
aida01 fault 0xc960 : 0xc971 : 17
aida02 fault 0x8bbf : 0x8bd0 : 17
aida03 fault 0xae9e : 0xaeaf : 17
aida04 fault 0x6e3 : 0x6f4 : 17
aida05 fault 0x31a4 : 0x31bd : 25
aida05 : WR status 0x10
aida06 fault 0x22ea : 0x2303 : 25
aida07 fault 0x438f : 0x43a8 : 25
aida08 fault 0x61ed : 0x6206 : 25
aida09 fault 0x1c32 : 0x1c40 : 14
aida10 fault 0x9332 : 0x933f : 13
aida11 fault 0x227b : 0x2288 : 13
aida12 fault 0xb2fc : 0xb309 : 13
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA
Base Current Difference
aida01 fault 0x0 : 0x1 : 1
aida02 fault 0x0 : 0x3 : 3
aida03 fault 0x0 : 0x1 : 1
aida04 fault 0x0 : 0x2 : 2
aida06 fault 0x0 : 0x2 : 2
aida08 fault 0x0 : 0x1 : 1
aida11 fault 0x0 : 0x3 : 3
aida12 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 4, Failed 8
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Looks like a big WR glitch happened at some point (master in issue?)
Memory
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 20 26 15 3 1 3 2 2 3 4 : 27848
aida02 : 25 7 1 4 1 4 2 2 3 3 6 : 36204
aida03 : 30 5 3 3 3 3 1 3 3 3 6 : 36464
aida04 : 5 10 0 5 2 3 1 3 3 3 6 : 36356
aida05 : 34 40 16 13 4 3 3 2 3 3 4 : 29160
aida06 : 16 8 4 2 1 4 1 3 3 3 6 : 36416
aida07 : 28 23 23 10 4 3 2 4 3 4 3 : 27736
aida08 : 26 9 3 3 1 3 1 3 3 3 6 : 36352
aida09 : 54 42 10 17 3 4 3 4 3 3 4 : 30376
aida10 : 17 5 7 4 3 3 3 3 2 3 6 : 35996
aida11 : 19 10 11 2 1 1 2 4 2 3 6 : 35916
aida12 : 82 42 12 7 4 2 3 2 2 3 4 : 27960
Options unchanged from ELOG 284
DAQ continues OK but New Merger window has disappeared... clearly still running in background!
DSSSD leakage currents decreasing as expected
-
Plan later today:
Prepare AIDA Kapton Cables for triple
install 4 new FEE64s into S4 (13,14,15,16)
Proposed layout of FEE64s (perspective with beam) in fig 5
Cards 9-12 will be moved on Friday when triple is installed
When ready MAC addressed will be given to OH/TD who will do the PC configuration including:
- Network IP allocation
- MIDAS registration
- Firmware updating
- Checks
17.20:
The four new FEEs have been installed and connected to Ethernet, Power, HDMI and Water
No front-end adapter is connected yet
The MAC addresss are as follows
aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed |
|
Tue May 4 14:05:48 2021, NH, MACB Cables
|
During the installation of the 4 new HDMI cables for aidas13-16 the HDMI cables between the MACB Root and the MACB leaves were looped through the cable guide at the top to support them more
This may improve their connection and reduce the chance of WR errors seen in S460/S452
The connectors were all reseated a few times to clean the contacts too |