 Tue Jul 7 08:04:57 2020, NH, Power failure in Messhuette
|
Last week there was a power failure in Messhuette (FRS control room) - AIDA-3 and MBS were affected.
AIDA itself and S4 equipment is on UPS and was not shut down.
AIDA-3 has been restarted and MIDAS restarted. All seems OK
S4 conditions: 24.2 C / 33.5% RH / Td = 7.1 C |
|
Tue Jul 7 08:50:27 2020, NH, nnrpi2 does not boot
|
MIDAS was not working properly on nnrpi2 - power cycled. Now system does not boot properly, it gets stuck trying to login claiming "PAM critical error" - probable file system corruption?
Will investigate if SD card can be checked and repaired |
|
Tue Oct 6 15:48:06 2020, NH, nnrpi kernel update
|
FYI:
The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were *huge* and filled with the same error many times a second
Oct 4 15:01:22 raspberrypi kernel: [1030048.603846]
Oct 4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.607742]
Oct 4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.667755]
Oct 4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.667848]
Oct 4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode
Oct 4 15:01:22 raspberrypi kernel: [1030048.731753]
A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version.
now the message doesn't seem to appear so hopefully the pis won't break so rapidly. |
Tue Nov 24 13:10:42 2020, NH, Waveforms with frame grounded 
|
AIDA Frame is now grounded to platform (will add photo later)
Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now) |
|
Mon Feb 1 15:19:07 2021, NH, AIDA Status Feb 2021
|
01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.
For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
The system has been powered up and biased successfully. To be monitored over the next few days.
A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently |
 Sat Feb 6 10:45:55 2021, NH, AIDA Status Feb 2021
|
> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
>
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
>
> The system has been powered up and biased successfully. To be monitored over the next few days.
>
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
>
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
>
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
>
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently
05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.
The hose has arrived and the water line will be prepared in my office mid next week.
All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)
A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.
The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.
There will be a DESPEC Dry Run next week Mon-Tue. |
|
Fri Feb 12 17:31:04 2021, NH, Background run
|
A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21
Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64
S4 access will be limited.
I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal. |
|
Mon Feb 22 18:18:47 2021, NH, AIDA Rates/Situation 8x
|
The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.
(All data below taken with thresholds 0xA for all)
Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"
Leakage currents high but S4 very warm at the moment as roof is still closed.
Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.
Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done
Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)
aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3 |
|
Sat Feb 27 12:47:57 2021, NH, AIDA Rates/Situation 10x
|
Updated figures as of 27 Feb 2021 - S4 is now closed for experiments but may open on Wednesday.
Snout has been grounded and all FEEs are seemingly grounded properly.
Figure 1: Good Events
Figure 2: Rate Histograms (Log-y)
Figure 3: Waveforms (Odd)
Figure 4: Waveforms (Even)
Figure 5: Waveforms @ 5MS/s (Odd)
Figure 6: Waveforms @ 5MS/s (Even)
Figure 7: Waveforms with Pulser (Odd)
Figure 8: Waveforms with Pulser (Even)*
Figure 9: Waveforms with Pulser @ 5MS/s (Odd)
Figure 10: Waveforms with Pulser @ 5MS/s (Even)*
* y-Scale enlarged from 0-10000 to 0-15000 to capture top of pulse |
|
Thu Mar 4 11:03:04 2021, NH, S452 Operating Notes
|
Please note the following "quirks" currently present in S452 that are important
a) Strips/Thresholds
Thresholds are 0xA, 0xA, 0x10 [Maybe changed depending on rate, etc]
Some strips are very noisy - damage?
b) Powering FEE64s
Due to a misbehaving USB hub(?) or cable only 7 FEE TTYs are monitored
aida04 and aida05 can take minutes to start, unsure why.
aida05 is monitored on nnrpi1 and hence when aida05 is ready it is safe to assume all 12 are ready
Watch the logs as normal
c) Slow control & nnrpi2
For the Grafana script to read the DSSD HV, it must be ran in a screen session
If nnrpi2 is restarted please run the HV as a screen by Putty SSHing (localhost 22)
and runningscreen -S caenhv /dev/ttyACM0 if screen was just closed you can do screen -x caenhv to reconnect
The pulser ideally should be started the samescreen -S pulser /dev/ttyUSB0 To allow remote changing by NH/GSI
The pulser is sent to all subsystems for correlation and should be kept on at 2 Hz for the experiment
IN the pulser putty you can check the settings by typingdisplay settings And general commands: help
The NIM crate can be monitored /dev/ttyUSB2 but you should be careful
It is very easy to turn off the NIM crate if you do
I run it in a tmux session with (tmux attach -r ) to have it "readonly"
There is a small script on nnrpi2 ./check_usbs.sh to see which ttyUSB is which device
As they can change if the pi is rebooted
d) Scaler Readout
If powercycled FEEs, enable the scaler in Local Controls page by setting BuTiS interface control to 0x3
To enable scaler readout after GOing the run please open the Correlation Control page
(Act on all FEE64s) and set control to 0x1
This has to be done every time the FEE64s are set to GOing
e) New Tape Server
For waveform readout in future experiments, the tape server is in "Expert" mode which is harder to use
Please read the "How to use" instructions by clicking the button on the bottom right
Or in a terminal cat /MIDAS/TclHttpd/Html/ETape/1.txt
f) GSI/MBS
If GSI team ask to restart AIDA MBS you may need to restart the MBS relay so it can reconnect
To start MBS in a terminalssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup
It should report it connected to AIDA-3 - if not check the MBS relay is running or restart the relay
Please check the MBS rate commandrate -nst -rst -nev -rev -ndata -rdata -st That the "Streams" column does not frequently (ever) have 0 empty streams
If so AIDA data may be getting dropped and should be looked into
If another DESPEC DAQ has failed this will eventually happen and is not a problem
But if it occurs in normal operation it is concerning
Do NOT stop the AIDA run without talking to GSI team - if no AIDA data the whole MBS DAQ stops
and blocks all other subsystems. It is best to keep AIDA ticking over constantly and let
MBS start/stop files when relevant.
Of course if things have to stop it is OK but inform GSI shifters so they can take action |
|
Tue Mar 30 13:21:50 2021, NH, nnrpi1 (tty logs)
|
We had issues with all TTYs reading out on nnrpi1, I have moved them from an apparently faulty USB hub to a new one.
Trying to reboot the pi to detect the new USB ports and it didn't recover from boot
Trying to repair SD card but I get a lot of errors on my laptop, indications are the SD card is completely dead
A new SD card will be ordered and nnrpi1 restored from the original images
A new USB hub should also be organised as it seems one is faulty (pi hangs if 3 or more USB devices connected to it) |
|
Fri Apr 9 16:40:36 2021, NH, nnrpi1 Update
|
nnrpi1 is now back in S4 with a new SD card and new raspberry pi.
The freezing originally continued (it was the USB system failing) but was fixed with a change to /boot/cmdline.txt coherent_pool=4M
It seems the kernel update to fix one bug introduced another...
Now it is running again with all TTYs connected. I think one TTY may not be working but 11 should be |
|
Mon Apr 12 18:36:31 2021, NH, [HowTo] AIDA & MBS
|
Some instructions on AIDA and MBS and troubleshooting are available here
https://sf.gsi.de/lib/c9ba0b4c-26bc-43d0-9b49-213fc594610f/file/AIDA%20and%20MBS%20Tips%20and%20Troubleshooting.pdf |
|
Thu Apr 15 12:23:22 2021, NH, Fast Discriminator Masks
|
It is possible to get good rates with the fast discriminators with a few hot channels mask in aida03 and aida10
Threshold is 0x20 = 3.2 MeV
They can be tested in beam to see performance, especially note the first setting will involve alpha emitters to check in AIDA
All FEEs:
AS01 01, AS3 07 10 13 are masked
aida03:
AS0 12 and 14 are masked in addition
aida10:
AS1 10, 11 and AS2 05, 06 are masked in addiction |
|
Fri Apr 16 02:15:12 2021, NH, Implants/Punchthrough
|
Rate with beam < 1000 Hz still
Not Uranium but some fragments possibly produced in the thick 9g degrader
S4 slits are +/- 20mm but these punchthrough and cover the +/- 40mm of AIDA |
|
Sat Apr 24 10:21:06 2021, NH, Saturday 24.04.2021
|
I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again
Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)
System Wide Checks
Clocks: OK
FADC: OK
WR:
Base Current Difference
aida05 fault 0x31a2 : 0x31a2 : 0
aida05 : WR status 0x10
aida06 fault 0x22e7 : 0x22e8 : 1
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK
aida05 ASIC maybe didn't complete setup properly (WR 0x10?)
Timestamps: OK (1)
Temps: OK
Will start gzipping S460 runs for migration to lustre & tape |
|
Wed Apr 28 10:06:11 2021, NH, Wednesday 28 April
|
11.05 All system wide checks OK *except*
WR
Base Current Difference
aida01 fault 0xc960 : 0xc971 : 17
aida02 fault 0x8bbf : 0x8bd0 : 17
aida03 fault 0xae9e : 0xaeaf : 17
aida04 fault 0x6e3 : 0x6f4 : 17
aida05 fault 0x31a4 : 0x31bd : 25
aida05 : WR status 0x10
aida06 fault 0x22ea : 0x2303 : 25
aida07 fault 0x438f : 0x43a8 : 25
aida08 fault 0x61ed : 0x6206 : 25
aida09 fault 0x1c32 : 0x1c40 : 14
aida10 fault 0x9332 : 0x933f : 13
aida11 fault 0x227b : 0x2288 : 13
aida12 fault 0xb2fc : 0xb309 : 13
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA
Base Current Difference
aida01 fault 0x0 : 0x1 : 1
aida02 fault 0x0 : 0x3 : 3
aida03 fault 0x0 : 0x1 : 1
aida04 fault 0x0 : 0x2 : 2
aida06 fault 0x0 : 0x2 : 2
aida08 fault 0x0 : 0x1 : 1
aida11 fault 0x0 : 0x3 : 3
aida12 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 4, Failed 8
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Looks like a big WR glitch happened at some point (master in issue?)
Memory
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 20 26 15 3 1 3 2 2 3 4 : 27848
aida02 : 25 7 1 4 1 4 2 2 3 3 6 : 36204
aida03 : 30 5 3 3 3 3 1 3 3 3 6 : 36464
aida04 : 5 10 0 5 2 3 1 3 3 3 6 : 36356
aida05 : 34 40 16 13 4 3 3 2 3 3 4 : 29160
aida06 : 16 8 4 2 1 4 1 3 3 3 6 : 36416
aida07 : 28 23 23 10 4 3 2 4 3 4 3 : 27736
aida08 : 26 9 3 3 1 3 1 3 3 3 6 : 36352
aida09 : 54 42 10 17 3 4 3 4 3 3 4 : 30376
aida10 : 17 5 7 4 3 3 3 3 2 3 6 : 35996
aida11 : 19 10 11 2 1 1 2 4 2 3 6 : 35916
aida12 : 82 42 12 7 4 2 3 2 2 3 4 : 27960
Options unchanged from ELOG 284
DAQ continues OK but New Merger window has disappeared... clearly still running in background!
DSSSD leakage currents decreasing as expected
-
Plan later today:
Prepare AIDA Kapton Cables for triple
install 4 new FEE64s into S4 (13,14,15,16)
Proposed layout of FEE64s (perspective with beam) in fig 5
Cards 9-12 will be moved on Friday when triple is installed
When ready MAC addressed will be given to OH/TD who will do the PC configuration including:
- Network IP allocation
- MIDAS registration
- Firmware updating
- Checks
17.20:
The four new FEEs have been installed and connected to Ethernet, Power, HDMI and Water
No front-end adapter is connected yet
The MAC addresss are as follows
aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed |
|
Tue May 4 14:05:48 2021, NH, MACB Cables
|
During the installation of the 4 new HDMI cables for aidas13-16 the HDMI cables between the MACB Root and the MACB leaves were looped through the cable guide at the top to support them more
This may improve their connection and reduce the chance of WR errors seen in S460/S452
The connectors were all reseated a few times to clean the contacts too |
|
Tue May 4 14:26:50 2021, NH, FEE64 Adapter wiring diagram and layout
|
A figure showing the FEE64 numbers (a previous diagram had 2/6 and 4/8 on the wrong side of the DSSD) and the wiring of pulser/HV
Also the jumpers connected to the boards
As always diagram is looking with direction of beam
Edit: Figure as signals (HV/Pulser) come from the beam right not beam left
Otherwise OK |
|
Fri May 7 13:11:22 2021, NH, Friday 7th May
|
14:11 - Alpha has been running most of morning
Just saw rates in tape spike to 6 MB/s...
Stop output to tape and look:
ASIC check & load... all rates except aida04 back to 0
Start output to tape
Back to ~300 KB/s
R6_49 will be affected by this.
Others seem OK
System wide check failures:
Base Current Difference
aida01 fault 0xb405 : 0xb406 : 1
aida02 fault 0xefc7 : 0xefc8 : 1
aida03 fault 0xdaab : 0xdaac : 1
aida04 fault 0x8f7c : 0x8f7d : 1
aida05 fault 0xb5bd : 0xb5be : 1
aida06 fault 0xeff1 : 0xeff2 : 1
aida07 fault 0x8f57 : 0x8f58 : 1
aida08 fault 0xbef7 : 0xbef8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x200 : 512
aida13 fault 0x0 : 0x61 : 97
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page)
Temps & Rates OK
Will now stop Alpha run and check noise situation
Run stopped at R6_49
Switching to nostorage (NOTAPE/R2)
Set ASICs to threshold 0xa, Rates OK
Rate to Tape/MBS ca 9 MB/s
No change to situation since yesterday... good sign
During MBS test aida13 rebooted... crash log from ttyUSB5 attached
It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again...
cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :)
A power cycle of all FEEs was performed after aida13's reboot to make sure things are good.
All systems check OK |