ID |
Date |
Author |
Subject |
307
|
Wed May 12 06:59:55 2021 |
OH, RDP | Wed 12 May 08:00-16:00 | 08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.
They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID
08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2
09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
Data rate 16.3MB/s
ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.
09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage
09:47 Beam back turned off no storage. On file R1_10
11:04 Now that we have implants on the detector we get a better look at the strips
From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
This adaptor board corresponds to FEE11
12:17 Leakage currents screenshot 512.png
13:30 Beam off. Expected back at 15:00...
14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
Rates are as before. No better, no worse.
15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising... |
306
|
Wed May 12 02:07:59 2021 |
MA, BA | Wed 12 May 00:00-08:00 | 03:12 System Check
attachment 1 : Current
attachment 2 : Temperature
attachment 3 : Rates
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 16, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x36ca : 0x36cb : 1
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida13 fault 0xa : 0xf : 5
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 5 7 2 1 2 2 3 3 3 6 : 36464
aida02 : 9 8 3 3 1 4 1 2 4 3 6 : 36916
aida03 : 5 2 6 2 0 3 2 3 3 3 6 : 36420
aida04 : 6 5 2 3 4 4 2 3 3 3 6 : 36800
aida05 : 17 6 6 2 2 4 1 3 2 4 6 : 37524
aida06 : 7 12 3 4 3 3 1 3 3 3 6 : 36460
aida07 : 17 11 5 1 3 3 3 2 3 3 6 : 36428
aida08 : 3 5 0 1 4 2 2 4 2 3 6 : 35924
aida09 : 27 6 4 2 0 2 2 2 3 3 6 : 35868
aida10 : 16 11 8 0 2 2 1 3 2 4 6 : 37272
aida11 : 15 2 2 3 1 4 3 4 2 3 6 : 36364
aida12 : 1 6 4 3 1 3 2 4 2 3 6 : 35988
aida13 : 22 14 10 2 4 2 2 4 2 3 6 : 36264
aida14 : 26 10 4 3 2 1 1 3 3 3 6 : 36184
aida15 : 14 2 3 2 2 4 1 2 3 3 6 : 35896
aida16 : 7 5 4 0 2 4 2 3 2 3 6 : 35588
06:10 DSSD1 rate high!
Attached 4
Called OH and he woke up to fix it :)
It was a problem with one of the ASIC according to what he says:
One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35
07:03 System check
attachment 5 Spectrum rate
attachment 6 Voltages
attachment 7 Rates
attachment 8 Temperature
System wide clock are all ok except
White rabbit
Base Current Difference
aida05 fault 0x36ca : 0x36cb : 1
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA
Base Current Difference
aida13 fault 0xa : 0x14 : 10
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
|
309
|
Wed May 12 18:53:38 2021 |
JS | Wed 12 20:00-00:00 | 19:51 Shift change
Stats ok
ucesb ok, Max implants ~1500 Hz
20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)
20:51
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.
20:59
Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
Base Current Difference
aida05 fault 0x36ca : 0x36ce : 4
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
Base Current Difference
aida13 fault 0xa : 0xb6 : 172
FPGA Timestamp error counter test result: Passed 15, Failed 1
--- both errors above same module as earlier shift
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 3 2 0 1 3 3 2 3 3 6 : 36092
aida02 : 6 4 4 2 2 3 1 3 3 3 6 : 36280
aida03 : 6 3 2 0 0 3 3 2 3 3 6 : 36048
aida04 : 10 3 1 1 3 3 1 3 3 3 6 : 36272
aida05 : 19 8 2 0 2 3 1 3 3 3 6 : 36268
aida06 : 2 2 0 2 3 4 2 2 3 3 6 : 36120
aida07 : 6 7 3 0 3 4 3 2 3 3 6 : 36416
aida08 : 7 5 4 0 3 2 2 3 3 3 6 : 36420
aida09 : 21 6 3 2 2 4 2 3 2 3 6 : 35700
aida10 : 13 3 1 1 3 3 1 4 2 3 6 : 35772
aida11 : 25 6 4 2 1 4 2 4 2 3 6 : 36180
aida12 : 0 4 0 3 2 3 3 3 2 3 6 : 35712
aida13 : 18 5 2 3 2 3 2 4 2 3 6 : 36080
aida14 : 2 1 1 1 2 3 2 4 2 3 6 : 35904
aida15 : 0 1 1 3 0 4 2 2 3 3 6 : 35960
aida16 : 4 5 3 2 3 4 1 3 2 3 6 : 35432
ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4
21:36
ucesb ok
Stats ok
22:01
ucesb ok 1500 Hz peak implant
Stats ok
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.
22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler
23:40 FRS magnet issue, possibly power, no beam
00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah |
308
|
Wed May 12 15:03:10 2021 |
CB, OH | Wed 12 16:00-20:00 | 16:00 Shift change
Statistics OK (attach 1)
Temps OK (attach 2)
Bias OK (attach 3)
System wide checks
Clock, ADC - OK
WR
Base Current Difference
aida05 fault 0x36ca : 0x36cb : 1
White Rabbit error counter test result: Passed 15, Failed 1
FPGA
Base Current Difference
aida13 fault 0xa : 0x4b : 65
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 5 6 1 1 2 2 3 3 3 6 : 36388
aida02 : 21 10 2 2 2 3 2 2 3 3 6 : 36100
aida03 : 25 5 3 1 0 3 3 2 3 3 6 : 36188
aida04 : 5 2 0 3 2 3 2 3 3 3 6 : 36484
aida05 : 16 6 3 1 2 4 1 3 2 4 6 : 37440
aida06 : 2 2 2 2 2 3 1 3 3 3 6 : 36216
aida07 : 16 9 3 0 3 4 3 2 3 3 6 : 36472
aida08 : 0 7 1 1 2 2 3 3 2 3 6 : 35560
aida09 : 15 7 4 2 2 4 2 3 2 3 6 : 35700
aida10 : 3 7 3 4 1 3 1 4 1 4 6 : 36788
aida11 : 5 1 3 1 1 4 3 3 2 3 6 : 35756
aida12 : 5 5 0 3 2 3 2 4 2 3 6 : 35996
aida13 : 2 5 1 2 2 4 3 3 2 3 6 : 35840
aida14 : 19 5 3 1 1 2 1 3 3 3 6 : 36100
aida15 : 20 6 6 2 1 3 2 2 3 3 6 : 36064
aida16 : 7 9 4 1 3 4 2 3 2 3 6 : 35716
File size change
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Mon May 10 16:47:51 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
16:20 Beam is back. Currently on File R1_200
16:43 Beam is being optimised.
17:00 UCESB OK
Stats OK (attach 4)
Temps OK
Bias OK (attach 5)
System-wide checks as before
17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK
17:52 Started compressing R1_1*
18:00 UCESB ok (attach 5)
Stats OK (attach 6)
Temps OK
Bias OK (attach 7)
System wide checks as before
19:00 UCESB ok (as before)
Stats OK (attach 8)
Temps OK
Bias OK (attach 9)
Merger OK (attach 10)
19:32 UCESB ok
Stats OK (attach 11)
Temps OK (attach 12)
Bias OK (attach 13)
System wide checks as before
19:51 Shift handed over to James S |
162
|
Tue Nov 24 13:10:42 2020 |
NH | Waveforms with frame grounded | AIDA Frame is now grounded to platform (will add photo later)
Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now) |
100
|
Fri Nov 22 08:55:31 2019 |
NH | Waveforms | AIDA Waveforms immediately after beam at 21.11.19 8am or so. |
101
|
Mon Nov 25 13:25:51 2019 |
NH | Waveforms | > AIDA Waveforms immediately after beam at 21.11.19 8am or so.
Update 25.11.19 with all FEE64 waveforms added
Things to note:
aida01, aida05, aida09 are all on TOP - show VHF noise, esp. aida01?
TOP is where HV is connected (-ve core)
Everything noisy in general though |
578
|
Thu Apr 18 13:59:55 2024 |
TD, JB | WR timestamp OK, readout time not OK |
WR timestamps appear to be OK but readout times fort aida03, aida04 and aida09 appear to be out of sequence. Attachment 1-3.
Reseated HDMI cable of aida09 and performed complete powercycle. Check the MACB HDMI port (appears to be loose) - still no change aida09 not collecting WR timestamps. Attachment 4-5. |
699
|
Thu Feb 20 17:06:42 2025 |
JB, CC, TD,MP | WR error for aida 13,14,15,16 | WR timestamp errors resolved after reseating the HDMI cables to the MACB |
74
|
Thu Oct 31 14:27:06 2019 |
NH | WR Timestamps | All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
HDMI cables in aida09 checked and good. |
78
|
Thu Oct 31 19:02:42 2019 |
Patrick | WR Timestamps | > All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
>
> HDMI cables in aida09 checked and good.
The problem would be the cable , one end or the other.
I think ( if I recall ) a setup would restart the WR decoder.
I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ? |
26
|
Sun Feb 3 16:01:49 2019 |
TD | Updated analyses GSIJan19/R1_0 - R10_0 |
Summary statistics for WR test data files R1_0, R2_0, R3_0, R4_0, R7_0 and R10_0 (attachments 1-6)
Latest version of WR TDR data analyser (attachment 7) |
551
|
Tue Mar 26 18:56:03 2024 |
TD | USB-controlled ac mains relay interlock box - wiring |
Sensor ( 4 pins )
Red +24V
Blue 0V
Yellow } contact
Green } closure in ( short these two wires together for logic 1 )
N.B. yellow and green wires of unused inputs must be connected together
Output ( 3 pins ) to USB-controlled ac mains relay
Red NC ( normally closed - with respect to COM with no power applied, single pole double throw relay out )
Blue NO ( normally open - with respect to COM with no power applied, single pole double throw relay out )
Green COM |
182
|
Mon Mar 8 22:51:47 2021 |
OH | Tuesday 9th March | 18:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)
at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message
01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event
Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.
02:00 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 4
FEE Temperatures - attachment 5
Bias and leakage currents ok - attachment 6
03:10 Problem with accelerator will be no beam for ~30 minutes
03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
There are now also errors in all FEEs for WR events
Base Current Difference
aida01 fault 0xc45f : 0xc46a : 11
aida02 fault 0xd250 : 0xd25b : 11
aida03 fault 0x27b2 : 0x27bd : 11
aida04 fault 0x714b : 0x7156 : 11
aida05 fault 0x74f5 : 0x7500 : 11
aida06 fault 0x558a : 0x5595 : 11
aida07 fault 0x8d20 : 0x8d2b : 11
aida08 fault 0x8131 : 0x813c : 11
aida09 fault 0xf412 : 0xf41d : 11
aida10 fault 0x82c2 : 0x82cd : 11
aida11 fault 0x2845 : 0x2850 : 11
aida12 fault 0x1bf6 : 0x1c01 : 11
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Also errors in FPGA
Base Current Difference
aida03 fault 0x0 : 0x1 : 1
aida04 fault 0x0 : 0x1 : 1
aida10 fault 0x0 : 0x2 : 2
aida11 fault 0x0 : 0x1 : 1
aida12 fault 0x1 : 0x2 : 1
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000
aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000
aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000
aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000
aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000
aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000
aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000
aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000
aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000
aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000
aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000
aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000
System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results
AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).
03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.
All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.
05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
The problem was with the accelerator operators system. There was no beam.
06:20 Error in WR system wide checks. Rest all ok
This time there is no bad merge items recorded in the merger terminal.
There are also no timewarp events seen in ucesb which were observed with the issues at 3am.
Base Current Difference
aida01 fault 0xdbe9 : 0xdbea : 1
aida02 fault 0xa93d : 0xa93e : 1
aida03 fault 0x8fc4 : 0x8fc5 : 1
aida04 fault 0x3e56 : 0x3e57 : 1
aida05 fault 0xb04c : 0xb04d : 1
aida06 fault 0x5576 : 0x5577 : 1
aida07 fault 0x99ff : 0x9a00 : 1
aida08 fault 0x8412 : 0x8413 : 1
aida09 fault 0x2457 : 0x2458 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000
aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000
aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000
aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000
aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000
aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000
aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000
aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000
aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000
aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000
aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000
aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000
Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s)
Cannot check file as was not writing to file.
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
06:40 There are occasional seconds of beam. According to operators it comes and then it goes again
06:45 Beam fairly stable so we are running again.
AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19
07:00 Beam stopped again
07:02 Beam returned
There are apparently issues with MUSIC chamber one and its resolution
07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. |
185
|
Tue Mar 9 20:04:20 2021 |
TD | Tuesday 9 March | 21.04 DAQ continues OK - file R20_379
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
FEE64 Temperatures OK - attachment 1
Good event statistics OK - attachment 2
Detector bias & leakage currents OK - attachment 3
Merger OK - 4.2M data items/s
TapeServer OK - 45 Mb/s
21.20 no merger server error/warning messages since 16.30UTC
21.35 p+n junction strip HEC & LEC spectra, Rate spectra common y-scale 0-30000 & 0-10
all spectra zero'd
22.05 no beam for 2-3h whilst accelerator generator is fixed |
567
|
Tue Apr 9 08:37:47 2024 |
TD | Tuesday 9 April | 09.36 Cooling water temperature and flow OK
FEE64 power ON
DSSSD bias & leakage current OK - attachment 1
FEE64 temps OK - attachment 2
*except* aida02 ASIC temp which is known to be u/s
All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3
WR timestamps OK - attachment 4
info code 4, correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 5-10
per FEE64 Rate spectra - attachments 11
per 1.8.W spectra - 20us FSR - attachments 12-13
12.25 DSSSD bias & leakage current OK - attachment 16
FEE64 temps OK - attachment 15
*except* aida02 ASIC temp which is known to be u/s
ADC data item stats - attachments 14
system stable |
405
|
Tue Mar 8 10:26:56 2022 |
TD NH | Tuesday 8 March 2022 |
11.25 System wide checks OK except aida06 clock fail status 6 - attachments 1-13
ADC data item stats - attachment 14
FEE64 temperatures OK - attachment 15
DSSSD bias & leakage currents - attachment 16
DSSSD#1 OK. DSSSD#2 leakage current has increased from c.5uA to c. 11uA overnight.
11.43 per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - attachments 17-24
per FEE64 1.8.L spectra - attachments 25-26
aida12 1.8.L pulser peak width 143 ch FWHM
aida02 1.8.L pulser peak width 439 ch FWHM
Per FEE64 rate spectra - attachment 27
12.00 Cycled DSSSD#2 bias from -100V to 0 to -100V leakage current -7.7uA but ramping slowly (c. 5nA/10s ... ?)
13.26 DSSSD#2 bias -100V leakage current -8.54uA
15.46 - Noticed that DSSSD#1 braid no longer has filter, DSSSD#1 core still does. DSSSD#2 is unfiltered on both.
- Swap FEE64 power connectors #3 with #4, and #6 with #7, this means both n+n FEE64s are on the same paired channel
No significant change to stats
- Notice braid was only connected to FEEs 4 and 8, which meant beam-left bias guard ring 0v was floating. Use T to bring bias braid over to FEES 2 and 6 as well.
Middle bias guard ring 0v is still floating
No significant change to stats, amplitude of 100 kHz waveform in n+n waves seems attenuated though
aida12 1.8.L pulser peak width 76 ch FWHM
aida02 1.8.L pulser peak width 438 ch FWHM
Attachment 28 - HV leakage current
Attachment 29 - FEE ADC stats
Attachment 30 - p+n side waveforms
Attachment 31 - n+n side waveforms
Attachment 32 - p+n pulser peak widths
Attachment 33 - n+n pulser peak widths
15.55 - Reattach filter to DSSSD#1 braid
No major change, possible improvement in waveforms
Attachment 34 - Stats
Attachment 35,36 - waveforms
16.46 - Reattach LK1 jumper to FEES 2, 4, 6, 8
Waveforms look OK, p+n rate slightly worse?
Attachment 37 - Stats
Attachment 38, 39 - waveforms
Attachment 40 - Per FEE Rates |
360
|
Tue Jun 8 10:51:55 2021 |
TD | Tuesday 8 June | 11.52 DAQ continues file S496/R32_340
High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning
11.53 ASIC check
System wide checks OK *except*
Base Current Difference
aida01 fault 0x8944 : 0x8947 : 3
aida02 fault 0x9b36 : 0x9b39 : 3
aida03 fault 0xf26b : 0xf26e : 3
aida04 fault 0x57a0 : 0x57a3 : 3
aida05 fault 0x4d3c : 0x4d3d : 1
aida05 : WR status 0x10
aida06 fault 0x640c : 0x640d : 1
aida07 fault 0x255e : 0x255f : 1
aida08 fault 0xffbd : 0xffbe : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 9 3 12 8 3 3 2 2 1 4 7 : 40508
aida02 : 5 5 10 6 1 3 3 2 1 4 7 : 40540
aida03 : 25 7 12 5 1 2 2 3 2 3 7 : 39740
aida04 : 19 6 11 5 1 3 2 3 2 3 7 : 39820
aida05 : 9 6 7 5 0 2 2 3 1 4 7 : 40548
aida06 : 10 8 7 4 3 3 1 3 1 4 7 : 40600
aida07 : 9 3 4 8 1 3 3 2 1 4 7 : 40508
aida08 : 12 5 8 5 3 3 3 2 1 4 7 : 40632
aida09 : 15 4 8 5 1 3 3 2 1 4 7 : 40508
aida10 : 23 5 12 5 1 2 2 2 1 4 7 : 40228
aida11 : 15 9 11 6 2 4 3 3 2 3 7 : 40308
aida12 : 14 9 4 5 3 3 2 2 1 4 7 : 40352
aida13 : 15 9 6 3 4 4 3 3 2 3 7 : 40260
aida14 : 11 5 7 6 4 4 3 3 2 3 7 : 40324
aida15 : 16 8 7 4 1 2 2 4 1 4 7 : 41136
aida16 : 15 8 7 4 3 4 3 3 2 3 7 : 40236
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - 20us FSR - attachments 3 & 4
ADC data items - attachment 5
FEE64 temperatures OK - attachment 6
DSSSD bias & leakage currents OK - attachment 7
Merger/Tape Server/Merger statistics - attachment 8
no merger errors reported since previous restart
16.28 analysis of file S496/R32_339 - attachment 9
max deadtime 0.24% aida04 |
630
|
Tue May 7 13:25:00 2024 |
JB, CC, NH, HMA | Tuesday 6 May - dismount snout | Status before dismount
14:26 DSSSD#1 119.99 V @ 17.968 uA
DSSSD#2 119.95 V @ 11.212 uA
Leakage currents slightly elevated from Monday.
S4 temperature at 26 °C.
14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.
Plan:
- Dismount snout.
- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.
|
296
|
Wed May 5 16:57:48 2021 |
TD | Tuesday 5 May |
17.55 Grafana DSSSD bias & leakage current - attachment 1
Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37
DSSSD bias & leakage currents - attachment 2 |
|