| ID |
Date |
Author |
Subject |
|
515
|
Thu Oct 6 16:51:49 2022 |
NH | Oscilloscope analysis | Investigating AIDA noise with a TA041 differential probe and oscilloscope
AC Mains (DESPEC platform AC, L-N)
Probe attentuation = 1:100
Fig 1: Main AC waveform [X: 5ms/div, Y: 100 V/div]
Fig 2: Zoomed in at peak (20 V FSR, any less and the waveform clipped) [X: 10us/div, Y: 20V/div]
Fig 3: Longer time base and FFT of 0-5 MHz. No significant frequency harmonics noticed [X: 5ms/div, Y:20 V, FFT X: 500 kHz/div, Y: 10 dBm/div]
No significant noise or distortion present, fully within any AC specification.
Note that at the moment there is almost no load on AC
Equipment on on DESPEC rack: AIDA NIM, AIDA Raspberry Pis, bPlas PC (+ WR) + 2x DESPEC NIM crates
No autofill, VME crate or detectors
All big machines at GSI (SIS, FRS) off (suspect pumps are on)
Ion catcher not on (I think under repair)
-
FEE PSU studies
Probe connected to 5V exposed power pin on FEE64 (+v) and to grounding crimp on FEE64 (-v)
No adapter board connected
Attentuation = 1:10
Fig 4: FFT when FEEs are *off* - essentially probe+scope noise [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 5: FFT when FEEs are *on* - notice 1.4 MHz peak in FFT, also seen on ADC waveform readout before (fig 6) [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 7: 500 ns/div 5V output on FEE, single FEE on the PSU [X: 500 ns/div, Y: 100 mV/div]
Note average max voltage is 5.31 V (power on) and ~ 70 mV "peak to peak" -might be from probe/scope?
Also see voltage changes with FEE power draw:
Power on : 5.45 V (different scale to above)
SETUP ran : 5.51 V
FADCs off : 5.86 V
ACQ Go: : Unchanged; ASIC threshold 0xa: Unchanged
Also check situation on a fully loaded PSU (8 fees connected and powered on)
Power on: 5.29 V (fig 8)
SETUP ran: 5.36 V (fig 9)
FADCs off: 5.64 V (fig 10)
All X: 500 ns/div, Y: 100 mV/div
Both cases observe voltage rises as current draw drops (as expected for voltage drop along a cable)
Noise on 'scope seems to get slightly worse with reduced current (and higher voltage)
No sign of strong 100 kHz noise as seen in ADC traces beforehand
Todo:
- Check -6V and 7V rails
- Check 5V and noise when front-end card is added and pulser/HV connected
- Check between two FEE64 grounds
- Check direct out of PSU vs ground to see if 1.4 MHz appears on PSU side or FEE64 side
-
11.10.22 Updates
Attachement 11 - 5V PSU on upper PSU with no FEEs attached whatsoever. No 1.4 MHz (on FFT) but clear low frequency beats from switching - presumably low/no load behaviour
Attachement 12 - 5V PSU on aida12 with 8 FEEs on PSU. Longer time base to allow lower frequencies in FFT. 1.4 MHz switching spikes visible but nothing around 100 kHz region
Attachments 13-16: 5V PSU on aida12 at 20 mV/div vertical and 1, 0.5, 2, 5 us/div horizontal respectively
12.10.22 Updates
Attachment 17: -6V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: -6.21 V
Attachment 19: 7V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: 7.46V
Measurement between AIDA12 ground and Reference ground/copper bar
+ve (red probe) attached to copper bar at ground point (not strong connection at present)
-ve (black probe) attached to ground crimp on aida12 (connected to cooling plate)
aida12 no adapter board connected: connections are PSU, Ethernet, HDMI and TTY only
Attachment 21: 5 us/div 100 mv/div waveform, big oscillations present. Not seen before FEEs turned on (8 FEES, 1-7+12)
Attachment 22: 10 ms/div for FFT, sharp peak at exactly 100 kHz observed...
Attachment 23: Between 5V PSU (+ve) and 19" rack (-ve) with no FEEs connected to PSU
See strong 100 kHz oscillations too, note that voltage isn't 5V as PSU is floating w.r.t. ground
Looks to be common mode noise (on both 5V and Return of PSU)
Attachment 24: Same as 21 but using thick crocodile clips on probe to ground and aida12. Noise is attenuated but still present |
|
516
|
Fri Dec 16 14:02:12 2022 |
NH | AIDA System off for christmas break | The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible |
|
519
|
Mon Aug 28 12:47:56 2023 |
NH | Power Failure 24.08.2023 | There was a power failure in the morning of 24.08.2023 in the Rhein-Main area affecting GSI
The Aida workstation (aida-3) has been restarted, it is unknown if the Pis in S4 rebooted as well (there is a UPS)
29.8.23 TD Both RPi systems rebooted four days ago. |
|
520
|
Thu Aug 31 15:24:56 2023 |
NH | New AIDA MBS PC | The AIDA MBS FDR will be x86l-119 from now on, not x86l-94
the MBS relay and startup scripts will be changed for this |
|
532
|
Sun Mar 10 17:08:12 2024 |
NH | AIDA FEE Layout + Cabling Plan for S100 | Proposed FEE numbering and wiring plan for upcoming experiment S100 (2x Wide DSSSDs)
Image designed in draw.io, source attached
FEE numbering is as S450, minimises cable movement from S505/Narrow AIDA
But means merger is not working with 1 DSSD (until all FEEs installed)
Wiring of adapter boards as from noise tests and what should work for DSSD bias
LK3 on middle bottom adapter to ground DSSD
LK1 on one n+n adapter to ground n+n side bias
p+n has -ve voltage (w.r.t. ground) bias applied via lower adapter boards
ground loop grounds all adapter boards, except 2 p+n adapter boards which are grounded by the bias lemo shield instead
MACB layout also included, with expected NIM logic signals for the aida scalers:
1: Pulser/Sync clock (send to all subsystems, "trigger 3")0
3/4: Time Machine
5/6: SC41L/R
All other FEEs have their scaler available
(Scaler should be in left LEMO on MACB, right is output (AIDA->NIM/unused), bottom 4 are triggers from AIDA (unused)
Test circuit will not be used in experiment due to noise, but can be temporarily set up for pulser walkthrough
Revision 2 correct as of 27 March 2023 |
|
539
|
Fri Mar 15 16:29:57 2024 |
NH | Leakage currents | The behaviour of the DSSSD leakage current at low voltages and during biases is unusual and varies depending on how the adapter boards are connected
To summarise the behaviour I have observed
Minimum bias configuration:
4 adapter boards, one n+n (LK1), three p+n (-ve bias), ground from n+n to one p+n
Voltage (and leakage current) unstable at low voltages, seems to settle at around -60 V
Drops can include 0 leakage current
Full adapter configuration:
8 adapter boards, ground ring complete
Same as minimum, but the drops seem to be much smaller (and not to 0 leakage current)
-60V again seems to be the turnover to a stable leakage current
In both cases the leakage current during ramping appears basically the same as when settled
Full into FEEs
8 adapter boards, fully connected to 8 FEEs
The leakage current is *much* higher during ramp,up to 17 uA near the end. No fluctuations
Once ramping has finished the current quickly drops back down and settles at the nominal leakage current
This has been observed in October/December too, it is not new (https://elog.ph.ed.ac.uk/AIDA/910)
During power up of the FEEs the current sometimes drops briefly (when the ASICs get programmed, I believe)
I think it is related to the ground (more or less current flowing through the HV supply instead of alternate paths?)
It should be kept in mind when testing new detectors to not worry about the detector at low voltages |
|
540
|
Mon Mar 18 18:04:43 2024 |
NH | Preparation for pre-s100 dry run (and test beam???) | In preparation for the dry run the following *temporary* changes to the FEE numbering have been prepared
These should be reverted after the dry run to ensure cable->fee agreement again
AIDA09 => AIDA06
AIDA11 => AIDA07
AIDA12 => AIDA08
This will allow the merger to run with 8 FEEs for 1 DSSD
dhcpd.conf updated
The 2023Oct19-13.46.30 should work with this numbering (check tomorrow)
As should layout GSI_triple_test_renumber
AFTER dry run:
Revert DHCP and prepare for full 16 FEEs
Make new ASIC settings key for 16 FEEs and prepare the aidaXX folders
Prepare a new Layout.mlf set |
|
541
|
Tue Mar 19 10:17:30 2024 |
NH | Dry Run 2024 - 19th March 24 | AIDA has 8 FEEs and 1 DSSSD
Aida08 (HDMI#12) had no WR again, I moved it to a different MACB and now it gets WR
The MACB (currently with jsut HDMI#10) seems issues, check/replace the upstream HDMI and thent eh MACB (after dry run!)
DSSSD#1 biased to -120 V, leakage current 5.6 uA (fig 1)
(also on Grafana)
Temps OK fig2
System Checks (fig3-5)
Clocks OK
aida07 fails calibration, others OK
WR OK (aida04 0x10, seems OK)
FPGA OK
Did "Synchronise ASIC clocks" to align ASIC clocks
(Notes for scalers: SC41L HDMI 5, SCI41R HDMI 9)
18:12 FRS is taking beam, AIDA is powered off and unbiased |
|
542
|
Wed Mar 20 12:22:27 2024 |
NH | Wed Mar 20 | Turn on AIDA for Dry Run demonstrations and so on
All system wide checks, temp, bias OK
Noise situation is dreadful (but has not been optimised). Deterioriation since first mounted, suspect cabling issues with bPlast and BB7.
Note thresholds at 0x32 (!!!) to not brutalise the DAQs during testing
aida08 seems OK
Server running to MBS totally fine
18:00
Carole grounded some of the Bplast and this reduced the rates in AIDA, although they are a bit fluctuatey. Due to position constraints she couldn't ground it all
Also AIDA ribbon cables are not grounded yet
The indication is these fixes should make a lot of difference to the situation
AIDA is now powered off for the end of day |
|
543
|
Wed Mar 20 17:02:53 2024 |
NH | /dev/sdd | The following messages are in the system log very often:
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk) |
|
557
|
Wed Apr 3 12:09:47 2024 |
NH | Report - aida06 frequently fails to boot first time (PHY error) | When booting up AIDA aida06 usually crashes the first time, it fails to get IP from DHCP
After 180 seconds it reboots and seems to connect fine
Log file attached, key part (to me) is this:
27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16| |
|
558
|
Wed Apr 3 12:42:57 2024 |
NH | Report aida02 WR errors | The WR error counter for aida02 seems to consantly rise
Tried reseating cable on both ends, no change
However clock status passed, aida02 has a correct WR timestamp and no FIFO/PLL errors seen
Edit to add: aida02 has the faulty ASIC temperature readout as well, related or coincidence? |
|
559
|
Wed Apr 3 13:02:19 2024 |
NH | Merger for 16 FEEs | Changed /MIDAS/Linux/startup/NewMerger
Change parameters -i and -l in master64 to 16 for 16 FEEs
Update NewMerger Options LinksAvailable to 16, LinksInUse to 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Fix NetVar RunOptions 1 (was 0)
Restart Merger HTTPd, Tape, Merger, MBS Spy
Reset/Setup/Go
16 Links green and status going, all good?
Bias DSSSDs and turn data transfer on
Merger connected, shows rate and updates... no rate in Tape Server?
.. Oops forget to turn on Output to data storage in merger!
Rate in tape server and to MBS: 7 MB/s
Merger, Tape and MBS working with 16 FEEs |
|
638
|
Tue Jun 4 15:40:32 2024 |
NH | Tues 4 Jun | DEGAS Array over Snout again
Situation similar to yesterday, most p+n rates good |
|
312
|
Thu May 13 17:05:42 2021 |
Muneerah, MArc | Thu 13 May 04:00-00:00 | 19:08 System Check
Rates from ucesb below 1 kHz
Atachment 1 Stats
Atachment 2 Temperatures
Atachment 3 Voltages
System wide checks are all ok except
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dba : 2
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA all passed
Memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 6 2 2 1 2 4 3 4 2 3 7 : 40424
aida02 : 3 4 2 2 1 2 1 3 3 3 7 : 40140
aida03 : 10 10 0 3 2 4 2 4 2 3 7 : 40280
aida04 : 3 2 0 2 2 3 2 3 1 4 7 : 40540
aida05 : 4 9 2 3 2 4 3 3 1 4 7 : 41048
aida06 : 8 6 1 1 2 3 2 2 3 3 7 : 40064
aida07 : 1 4 0 1 1 3 1 3 3 3 7 : 40196
aida08 : 1 4 2 2 2 4 3 4 2 3 7 : 40452
aida09 : 8 5 2 2 1 2 1 3 3 3 7 : 40168
aida10 : 22 5 4 2 1 3 1 4 2 3 7 : 39872
aida11 : 2 1 2 1 2 2 3 3 2 3 7 : 39632
aida12 : 8 5 2 2 1 3 1 4 2 3 7 : 39784
aida13 : 18 8 2 1 0 2 3 3 2 3 7 : 39624
aida14 : 23 12 2 2 1 2 2 4 2 3 7 : 40028
aida15 : 3 5 3 2 2 3 1 4 2 3 7 : 39844
aida16 : 0 2 1 2 1 4 2 4 2 3 7 : 40096
|
|
330
|
Wed May 19 09:12:52 2021 |
Muneerah, Betool, LS | 17th May 00:00-12:00 | Auther: Muneerah and Betool
02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok
Statistics ok - 210517_0202_Stats
Temperatures ok - 210517_0202_Temp
Bias and leakage ok - 210517_0200_Bias
02:05 System wide checks:
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1552 : 0x155d : 11
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 5 8 3 3 3 5 2 2 2 3 7 : 39460
aida02 : 22 4 5 0 3 2 2 3 2 3 7 : 39560
aida03 : 21 10 2 1 1 3 2 4 2 3 7 : 40100
aida04 : 18 7 2 0 3 3 3 2 2 3 7 : 39392
aida05 : 16 6 5 1 2 4 3 2 2 3 7 : 39520
aida06 : 18 6 6 2 2 2 2 3 2 3 7 : 39576
aida07 : 12 8 6 1 2 3 1 3 2 3 7 : 39408
aida08 : 13 11 4 0 1 3 3 2 2 3 7 : 39308
aida09 : 10 8 4 2 3 4 2 3 2 3 7 : 39848
aida10 : 16 8 2 1 2 4 2 3 1 4 7 : 40768
aida11 : 26 10 2 2 1 2 2 3 2 3 7 : 39512
aida12 : 16 9 2 1 2 4 2 2 2 3 7 : 39240
aida13 : 15 9 3 2 2 4 2 2 2 3 7 : 39284
aida14 : 25 10 2 3 3 3 2 3 2 3 7 : 39796
aida15 : 4 2 2 1 1 3 2 2 2 3 7 : 38944
aida16 : 16 7 3 3 1 3 3 2 2 3 7 : 39368
04:00 Statistics ok - 210517_0403_Stats
Temperatures ok - 210517_0403_Temp
Bias and leakage ok - 210517_0400_Bias
04:05 System wide checks:
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1552 : 0x1560 : 14
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 7 5 2 3 5 2 2 2 3 7 : 39492
aida02 : 23 6 7 2 1 3 2 3 2 3 7 : 39676
aida03 : 17 9 3 2 0 3 2 3 2 3 7 : 39548
aida04 : 16 4 1 2 1 3 3 3 2 3 7 : 39792
aida05 : 22 7 3 3 1 4 3 3 2 3 7 : 40032
aida06 : 18 6 7 3 1 3 2 3 2 3 7 : 39688
aida07 : 18 7 4 1 2 4 1 4 2 3 7 : 40032
aida08 : 19 5 4 1 1 3 3 3 2 3 7 : 39828
aida09 : 10 6 7 1 3 4 2 2 2 3 7 : 39336
aida10 : 12 10 2 3 1 4 2 3 1 4 7 : 40768
aida11 : 10 9 4 3 2 1 2 3 2 3 7 : 39440
aida12 : 8 7 6 1 2 4 2 3 2 3 7 : 39768
aida13 : 11 2 6 1 2 3 2 3 2 3 7 : 39612
aida14 : 27 7 4 2 3 3 2 3 2 3 7 : 39780
aida15 : 4 2 3 1 1 3 2 2 2 3 7 : 38960
aida16 : 18 8 4 2 1 4 2 2 2 3 7 : 39240
06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA
06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again.
07:01 it is working fine.
07:40 Screen shots after the restart
Statistics ok - 210517_0742_Stats
Temperatures ok - 210517_0743_Temp
Bias and leakage ok - 210517_0740_Bias
System wide checks
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x61da : 0x61dd : 3
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 6 3 1 1 3 1 3 3 3 7 : 40336
aida02 : 11 6 0 1 1 5 2 4 2 3 7 : 40252
aida03 : 0 5 2 3 3 3 3 4 2 3 7 : 40424
aida04 : 2 4 2 1 2 4 3 3 2 3 7 : 39912
aida05 : 11 7 5 1 2 2 4 2 3 3 7 : 40532
aida06 : 20 6 0 2 3 2 1 3 3 3 7 : 40320
aida07 : 15 5 1 2 2 3 2 2 3 3 7 : 40116
aida08 : 2 3 4 1 3 3 2 2 3 3 7 : 40128
aida09 : 18 5 1 3 2 4 2 2 3 3 7 : 40288
aida10 : 11 6 3 4 2 3 3 2 2 4 7 : 41484
aida11 : 4 4 5 2 2 2 3 3 2 3 7 : 39744
aida12 : 18 7 2 3 2 2 4 3 2 3 7 : 40064
aida13 : 2 2 1 3 0 3 2 4 2 3 7 : 39944
aida14 : 16 4 5 3 3 3 3 2 2 4 7 : 41552
aida15 : 9 3 1 1 2 3 2 3 3 3 7 : 40556
aida16 : 16 5 1 2 1 2 3 4 2 3 7 : 40184
Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200
***** 08.30 *****
Checked R12_34 through analyser, dead time seems okay:
*** Timestamp elapsed time: 232.154 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.021 0.000
1 2.781 0.000
2 0.849 0.000
3 5.686 0.000
4 0.000 25.816
5 0.017 0.000
6 0.177 0.000
7 1.349 0.000
8 1.791 0.000
9 0.000 91.660
10 6.194 0.000
11 0.320 0.000
12 1.425 0.000
13 0.000 72.999
14 0.052 0.000
15 1.729 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s
***** 09.00 *****
Checked R12_48 through analyser, dead time seems okay:
*** Timestamp elapsed time: 170.459 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.196 0.000
1 6.294 0.000
2 0.992 0.000
3 8.944 0.000
4 0.024 0.000
5 0.001 0.000
6 0.605 0.000
7 2.096 0.000
8 1.654 0.000
9 0.000 31.395
10 6.175 0.000
11 2.524 0.000
12 2.076 0.000
13 0.000 10.337
14 0.103 0.000
15 2.203 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0902_Stats.png)
***** 09.30 *****
AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20
Ran analyser for R13_16
*** Timestamp elapsed time: 168.059 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.047 0.000
1 2.282 0.000
2 0.080 0.000
3 0.806 0.000
4 0.000 0.000
5 0.260 0.000
6 0.030 0.000
7 1.471 0.000
8 67.467 0.000
9 80.552 0.000
10 0.057 0.000
11 0.000 11.671
12 1.191 0.000
13 0.080 0.000
14 0.091 0.000
15 0.202 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision
*** Timestamp elapsed time: 149.082 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 3.580 0.000
2 0.000 0.000
3 2.800 0.000
4 0.000 0.000
5 0.514 0.000
6 0.140 0.000
7 1.372 0.000
8 55.544 0.000
9 66.588 0.000
10 1.065 0.000
11 0.000 10.010
12 0.681 0.000
13 0.044 0.000
14 0.070 0.000
15 0.400 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)
Merger rate ~5M items/s
Tape server ~17 MB/s
Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision
*** Timestamp elapsed time: 122.147 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.027 0.000
1 19.945 0.000
2 0.601 0.000
3 21.555 0.000
4 0.042 0.000
5 3.226 0.000
6 1.211 0.000
7 9.674 0.000
8 52.817 0.000
9 54.972 0.000
10 1.848 0.000
11 0.008 0.000
12 2.034 0.000
13 0.278 0.000
14 0.018 0.000
15 0.634 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
high deatime now also seen in aida02 and aida04
Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check
*** Timestamp elapsed time: 205.006 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 40.176
1 0.438 0.000
2 0.199 0.000
3 2.927 0.000
4 0.000 140.347
5 1.282 0.000
6 0.150 0.000
7 2.097 0.000
8 1.673 0.000
9 0.000 87.749
10 1.150 0.000
11 0.489 0.000
12 0.374 0.000
13 0.048 20.995
14 0.251 0.000
15 2.441 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs
Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200
***** 08.30 *****
Checked R12_34 through analyser, dead time seems okay:
*** Timestamp elapsed time: 232.154 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.021 0.000
1 2.781 0.000
2 0.849 0.000
3 5.686 0.000
4 0.000 25.816
5 0.017 0.000
6 0.177 0.000
7 1.349 0.000
8 1.791 0.000
9 0.000 91.660
10 6.194 0.000
11 0.320 0.000
12 1.425 0.000
13 0.000 72.999
14 0.052 0.000
15 1.729 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s
***** 09.00 *****
Checked R12_48 through analyser, dead time seems okay:
*** Timestamp elapsed time: 170.459 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.196 0.000
1 6.294 0.000
2 0.992 0.000
3 8.944 0.000
4 0.024 0.000
5 0.001 0.000
6 0.605 0.000
7 2.096 0.000
8 1.654 0.000
9 0.000 31.395
10 6.175 0.000
11 2.524 0.000
12 2.076 0.000
13 0.000 10.337
14 0.103 0.000
15 2.203 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0902_Stats.png)
***** 09.30 *****
AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20
Ran analyser for R13_16
*** Timestamp elapsed time: 168.059 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.047 0.000
1 2.282 0.000
2 0.080 0.000
3 0.806 0.000
4 0.000 0.000
5 0.260 0.000
6 0.030 0.000
7 1.471 0.000
8 67.467 0.000
9 80.552 0.000
10 0.057 0.000
11 0.000 11.671
12 1.191 0.000
13 0.080 0.000
14 0.091 0.000
15 0.202 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision
*** Timestamp elapsed time: 149.082 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 3.580 0.000
2 0.000 0.000
3 2.800 0.000
4 0.000 0.000
5 0.514 0.000
6 0.140 0.000
7 1.372 0.000
8 55.544 0.000
9 66.588 0.000
10 1.065 0.000
11 0.000 10.010
12 0.681 0.000
13 0.044 0.000
14 0.070 0.000
15 0.400 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)
Merger rate ~5M items/s
Tape server ~17 MB/s
Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision
*** Timestamp elapsed time: 122.147 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.027 0.000
1 19.945 0.000
2 0.601 0.000
3 21.555 0.000
4 0.042 0.000
5 3.226 0.000
6 1.211 0.000
7 9.674 0.000
8 52.817 0.000
9 54.972 0.000
10 1.848 0.000
11 0.008 0.000
12 2.034 0.000
13 0.278 0.000
14 0.018 0.000
15 0.634 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
high deatime now also seen in aida02 and aida04
Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check
*** Timestamp elapsed time: 205.006 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 40.176
1 0.438 0.000
2 0.199 0.000
3 2.927 0.000
4 0.000 140.347
5 1.282 0.000
6 0.150 0.000
7 2.097 0.000
8 1.673 0.000
9 0.000 87.749
10 1.150 0.000
11 0.489 0.000
12 0.374 0.000
13 0.048 20.995
14 0.251 0.000
15 2.441 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs
12:52 Check FPGA Timestamp errors gives following error message:
Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"
Tcl Call Trace
6: DocSubstSystemFile sock8 error 500 can't\ read\ "\;TS_Base_Collected"\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n"\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}"\;\n\ \ \ \ (procedure\ "\;do_TS_errors"\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n"\;do_TS_errors\ "\;\n\ \ \ \ ("\;TSERRORS"\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ..."\;\n\ \ \ \ (procedure\ "\;do_click"\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n"\;do_click"\;\n\ \ \ \ invoked\ from\ within\n"\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ "\;"\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ "\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form..."\;\n\ \ \ \ (file\ "\;Check.tcl"\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n"\;source\ Check.tcl"\;\n\ \ \ \ (in\ namespace\ eval\ "\;::Check"\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n"\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB..."\;\n\ \ \ \ invoked\ from\ within\n"\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n<\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>\;..."\;\n\ \ \ \ ("\;uplevel"\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;uplevel\ #0\ [list\ subst\ \$script\]"\;\n\ \ \ \ (procedure\ "\;SubstFile"\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n"\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}"\;\n\ \ \ \ ("\;uplevel"\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]"\;\n\ \ \ \ (procedure\ "\;Subst_File"\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n"\;Subst_File\ \$template\ \$interp"\;\n\ \ \ \ invoked\ from\ within\n"\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)"\;\n\ \ \ \ (procedure\ "\;Doc_application/x-tcl-template"\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n"\;\$cmd\ \$path\ \$suffix\ \$sock"\;\n\ \ \ \ (procedure\ "\;Doc_Handle"\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n"\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock"\;\n\ \ \ \ (procedure\ "\;DocDomain"\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n"\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml"\;\n\ \ \ \ ("\;eval"\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]"\;
5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\"
4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE
3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {}
2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml}
1: HttpdReadPostGlobal sock8 Httpds
Dead time check:
[npg@aidas-gsi S496]$ ~/analyser/analyser R14_24
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 252995858 ( 451517.9 Hz)
Other data format: 8924143 ( 15926.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 85 ( 0.2 Hz)
RESUME: 85 ( 0.2 Hz)
SYNC100: 34087 ( 60.8 Hz)
WR48-63: 34087 ( 60.8 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 8855799 ( 15804.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 0 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 560.323 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 349.960
1 0.002 0.000
2 0.000 5.974
3 1.105 0.000
4 0.000 447.454
5 0.000 2.809
6 0.000 16.727
7 0.137 0.000
8 0.090 0.000
9 0.000 429.897
10 0.461 0.000
11 0.018 0.000
12 0.000 8.985
13 0.000 466.311
14 0.000 15.095
15 0.346 0.000
13:24 Beam is off at the moment...
13:43 Changing spill structure from 1.5s/3.5s to 1s.
13:50
[npg@aidas-gsi S496]$ ~/analyser/analyser R14_29
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 251763810 ( 930342.1 Hz)
Other data format: 10156190 ( 37530.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 205 ( 0.8 Hz)
RESUME: 205 ( 0.8 Hz)
SYNC100: 33009 ( 122.0 Hz)
WR48-63: 33009 ( 122.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 10089762 ( 37284.7 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 527077 ( 1947.7 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 270.614 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.034 6.803
1 8.461 0.000
2 0.569 0.000
3 11.664 0.000
4 0.024 26.416
5 0.854 0.000
6 0.069 0.000
7 0.741 0.000
8 1.336 0.000
9 0.000 75.926
10 0.353 0.000
11 0.105 0.000
12 0.070 0.000
13 0.000 93.585
14 0.017 0.000
15 0.121 0.000
15:06 Spill structure back to 1.5 on 2s off. |
|
310
|
Thu May 13 00:51:40 2021 |
Muneerah and OH | Thu 13 May 00:00-08:00 | 01:50 Beam on
The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing.
Tempetratures OK
Rates OK
Attachment 1 Voltages
Atachment 2 Rates
Atachment 3 Temperatures
Attachment 4 ucesb rates less than 1 kHz
System wide check are ok except:
Base Current Difference
aida05 fault 0x36ca : 0x36ce : 4
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida13 fault 0xa : 0xe9 : 223
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760
aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644
aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188
aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404
aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104
aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128
aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500
aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672
aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588
aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460
aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348
aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000
aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840
aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996
aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280
aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716
<strong>05:00 OH Took over at 04:00</strong>
System wide checks ok except
Base Current Difference
aida05 fault 0x36ca : 0x36d0 : 6
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x1 : 1
aida13 fault 0xa : 0xe9 : 223
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
05:41 AIDA 05 has dropped out. Will restart the system
05:58 DAQ recovered from powercycle
p+n strips at 0xa
n+n strips at 0x20
ADC control register at 0xFF
All waveforms turned off in LED
Statistics - attachment 5
Correlation scaler in FEE 3 and 4 are mostly tracking 1:1
06:34 Local users have been unable to recover MBS
And have been unable to contact local experts
07:22 Bias and leakage currents ok - attachment 6
Stats attachment -7 |
|
335
|
Thu May 20 01:00:30 2021 |
Muneerah | Thu May 20 00:00-08:00 | 02:00 System check
The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!
The rates on ucesb got back to work by around 01:12
Deat Time check ok R14_870, attachment 1
Statistics are ok attachment 2
Temperatures are ok attacment 3
Voltages are ok attachment 4
system wide check
All ok except
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbite
1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 8 8 5 0 6 1 2 2 2 7 : 37296
aida02 : 19 8 11 2 3 4 2 2 3 3 6 : 36412
aida03 : 14 6 3 4 1 3 1 3 3 3 6 : 36312
aida04 : 25 12 12 4 3 4 1 4 3 3 6 : 37316
aida05 : 4 8 3 4 1 2 2 4 3 3 6 : 36928
aida06 : 42 12 8 7 2 3 3 4 3 3 6 : 37736
aida07 : 24 6 7 3 3 3 1 3 2 2 7 : 37536
aida08 : 10 15 7 2 1 4 2 3 3 3 6 : 36752
aida09 : 29 13 6 6 4 3 3 3 3 3 6 : 37244
aida10 : 18 12 7 2 2 3 3 2 2 4 6 : 37464
aida11 : 20 8 0 5 5 4 2 3 3 3 6 : 36976
aida12 : 7 2 4 3 2 3 2 2 4 3 6 : 37068
aida13 : 2 5 5 4 2 3 2 3 3 3 6 : 36608
aida14 : 9 13 8 2 2 2 3 3 2 4 6 : 37836
aida15 : 24 12 12 1 0 3 2 3 3 3 6 : 36640
aida16 : 12 9 9 2 1 3 2 4 3 3 6 : 37128
6:45 System check
Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9
System wide check are all ok except:
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit 1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 11 3 5 0 6 1 2 3 2 7 : 38264
aida02 : 14 9 9 2 3 3 3 2 3 3 6 : 36496
aida03 : 13 7 3 5 2 3 2 3 3 3 6 : 36668
aida04 : 30 15 13 4 3 4 1 3 4 3 6 : 37888
aida05 : 17 9 3 3 1 2 2 4 3 3 6 : 36956
aida06 : 40 16 5 6 2 3 2 3 2 4 6 : 37936
aida07 : 3 2 3 2 3 4 2 2 2 2 7 : 37196
aida08 : 17 10 6 2 1 3 3 3 3 3 6 : 36852
aida09 : 29 11 7 6 3 3 2 4 3 3 6 : 37436
aida10 : 1 10 5 1 1 3 3 2 2 4 6 : 37252
aida11 : 30 9 0 5 4 4 2 3 3 3 6 : 36960
aida12 : 19 10 5 3 2 3 2 2 4 3 6 : 37196
aida13 : 22 9 6 3 2 3 3 3 3 3 6 : 36960
aida14 : 19 6 9 2 2 3 3 2 2 4 6 : 37452
aida15 : 6 8 11 2 1 3 2 3 3 3 6 : 36616
aida16 : 19 14 5 3 1 3 1 4 3 3 6 : 36908
07:33 system check
Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13
System wide check
all ok except
ADC all failed
White rabbit
1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 9 4 6 0 5 1 2 2 2 7 : 37152
aida02 : 19 6 13 1 3 3 4 2 3 3 6 : 36780
aida03 : 23 5 6 4 2 3 1 3 3 3 6 : 36452
aida04 : 24 11 14 3 3 4 2 3 3 3 6 : 37048
aida05 : 17 3 6 3 1 2 2 3 3 3 6 : 36444
aida06 : 36 15 4 9 2 4 2 4 3 3 6 : 37608
aida07 : 20 8 5 3 3 4 2 2 2 2 7 : 37376
aida08 : 21 11 10 2 1 4 2 2 2 4 6 : 37324
aida09 : 27 14 7 6 3 3 2 4 3 3 6 : 37452
aida10 : 25 9 5 2 2 3 3 2 2 4 6 : 37436
aida11 : 30 14 2 4 5 3 2 3 3 3 6 : 36936
aida12 : 22 3 5 3 2 3 2 3 3 3 6 : 36640
aida13 : 10 7 1 4 2 3 3 2 3 3 6 : 36336
aida14 : 13 9 12 2 3 3 2 3 2 4 6 : 37820
aida15 : 20 8 11 3 0 3 2 3 3 3 6 : 36640
aida16 : 23 6 6 2 0 2 1 3 4 3 6 : 37164
|
|
338
|
Thu May 20 23:26:11 2021 |
Marc Labiche (ML) | system wide checks | System Wide check at begining of the night shift:
-----
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
-----
ADC calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
-----
White rabbit status:
Base Current Difference
aida05 fault 0x500 : 0x56d : 109
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
-----
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
-----
Memory info from FEE cards
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 7 7 6 1 5 2 2 3 3 6 : 36472
aida02 : 19 5 4 4 4 3 3 3 2 3 6 : 36020
aida03 : 22 10 2 2 5 2 2 3 3 3 6 : 36680
aida04 : 12 7 1 0 4 3 1 4 3 3 6 : 36856
aida05 : 16 7 5 3 1 3 2 3 3 3 6 : 36584
aida06 : 20 7 5 6 2 4 4 2 3 3 6 : 36888
aida07 : 13 7 5 2 3 2 1 3 2 2 7 : 37308
aida08 : 18 7 5 3 0 2 3 3 3 3 6 : 36656
aida09 : 21 6 4 4 1 4 2 3 3 3 6 : 36740
aida10 : 21 10 2 3 2 3 2 2 2 4 6 : 37156
aida11 : 23 2 4 4 3 2 2 3 2 4 6 : 37612
aida12 : 24 1 4 3 2 2 1 2 4 3 6 : 36744
aida13 : 26 9 4 2 1 4 3 2 3 3 6 : 36464
aida14 : 19 9 9 2 3 4 1 2 2 4 6 : 37156
aida15 : 10 7 10 0 1 3 1 3 3 3 6 : 36288
aida16 : 28 7 3 2 1 3 3 3 3 3 6 : 36824 |
|
497
|
Fri Jun 24 15:04:06 2022 |
Marc | Friday 24th June - evening shift | 16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.
Next wide check will be in about an hour.
17:00
Stats -ok - attachment 1
Temperatures ok - attachement 2
Leakage current ok but - attachment 3
ucesb screen-shot - attachment 4
Wide check completed. Nothing different.
WR status decoder status:
Base Current Difference
aida07 fault 0xc53d : 0xc547 : 10
aida08 fault 0xf1be : 0xf1e9 : 43
White Rabbit error counter test result: Passed 6, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp::
Base Current Difference
aida07 fault 0x2a : 0x2c : 2
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.
19:20 -
Stats ok - attachment 6
Temp ok - attachment 7
Leakage current ok - attachement 8
Wide check completed and same output as above.
22:00
Stats ok - attachment 9
Temp ok - attachment 10
Leakage current ok - attachement 11
Wide check completed and same output as above.
|
|