| |
ID |
Date |
Author |
Subject |
|
|
331
|
Wed May 19 11:24:55 2021 |
RDP | Monday 19 April 12 noon shift |
System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed.
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
Base Current Difference
aida07 fault 0xd052 : 0xd056 : 4
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x16 : 22
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 2 6 2 0 5 5 1 2 3 3 6 : 36120
aida02 : 25 10 0 4 3 5 2 3 3 3 6 : 36980
aida03 : 22 12 5 2 1 3 2 3 3 3 6 : 36616
aida04 : 16 3 3 3 4 4 4 2 3 3 6 : 36840
aida05 : 14 5 6 2 3 4 4 2 3 3 6 : 36800
aida06 : 30 10 6 3 3 4 2 3 3 3 6 : 36936
aida07 : 27 11 1 0 0 3 2 3 3 3 6 : 36436
aida08 : 38 7 7 3 3 2 2 3 3 3 6 : 36704
aida09 : 4 6 4 2 4 4 1 4 3 3 6 : 37056
aida10 : 21 10 3 3 1 5 1 3 3 3 6 : 36596
aida11 : 24 8 2 1 4 3 2 4 2 3 6 : 36192
aida12 : 31 12 4 3 5 5 2 2 3 3 6 : 36668
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Mon Apr 19 10:32:22 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
There should be a separate elog note about changing one of the bias voltages.
All histograms zeroed at 16:13.
16.20 DAQ continues file NULL/R33_156
aida09 HEC fast comparator 0x2->0x1
DSSSD#3 bias -100->-120V |
|
|
610
|
Fri Apr 26 15:34:00 2024 |
RDP | 16:00 - 00:00 Friday 26 April |
This appeared in the system log:
Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
These messages seem to appear sporadically.
18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?
18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.
aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters
Thanks Nic!
22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)
22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.
|
|
|
313
|
Thu May 13 19:52:36 2021 |
Philippos and Marc | Connection problem |
At 20:15, We lost connection with AnyDesk.
Liliana restarted Anydesk on the AIDA DAQ PC. All seemed fine on AIDA DAQ but there was a more generic issue on the main DAQ, possibly the timesorter.
Helena restarted all after ~30mn and it's all working again now.
|
|
|
314
|
Thu May 13 21:21:25 2021 |
Philippos and Marc | system wide checks |
Wide checks at Time 22:22 CET
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dbc : 4
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 35 11 3 5 5 4 1 3 2 4 6 : 37876
aida02 : 20 5 4 3 2 5 2 4 3 3 6 : 37400
aida03 : 20 3 4 3 4 4 1 3 2 4 6 : 37640
aida04 : 25 6 2 3 4 4 3 3 3 3 6 : 37140
aida05 : 30 10 5 7 2 6 2 4 2 4 6 : 38776
aida06 : 21 11 4 4 3 5 2 4 3 3 6 : 37548
aida07 : 23 8 4 4 2 6 1 3 2 4 6 : 37852
aida08 : 27 7 2 3 4 3 2 4 3 3 6 : 37284
aida09 : 31 6 5 6 3 3 2 4 3 3 6 : 37372
aida10 : 22 5 3 4 3 6 2 2 2 4 6 : 37616
aida11 : 23 5 4 2 5 2 1 3 2 4 6 : 37444
aida12 : 20 7 4 7 3 5 2 3 3 3 6 : 37096
aida13 : 28 5 3 3 1 5 2 3 3 3 6 : 36840
aida14 : 38 12 2 4 2 4 1 4 3 3 6 : 37144
aida15 : 27 9 5 4 2 5 3 2 3 3 6 : 36740
aida16 : 21 6 4 4 1 4 2 3 3 3 6 : 36740
|
|
|
494
|
Thu Jun 23 15:10:17 2022 |
Philippos Papadakis | Thursday 23 June 16:00-24:00 |
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc525 : 39
aida08 fault 0xf0e9 : 0xf166 : 125
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x1c : 11
aida08 fault 0x1a : 0x23 : 9
Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)
18:05
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc527 : 41
aida08 fault 0xf0e9 : 0xf177 : 142
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x1c : 11
aida08 fault 0x1a : 0x24 : 10
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)
20:22
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc529 : 43
aida08 fault 0xf0e9 : 0xf17f : 150
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x20 : 15
aida08 fault 0x1a : 0x25 : 11
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)
21:56
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc529 : 43
aida08 fault 0xf0e9 : 0xf186 : 157
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x21 : 16
aida08 fault 0x1a : 0x26 : 12
Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)
23:50
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc52e : 48
aida08 fault 0xf0e9 : 0xf18c : 163
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x21 : 16
aida08 fault 0x1a : 0x27 : 13
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15) |
|
|
78
|
Thu Oct 31 19:02:42 2019 |
Patrick | WR Timestamps |
> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
>
> HDMI cables in aida09 checked and good.
The problem would be the cable , one end or the other.
I think ( if I recall ) a setup would restart the WR decoder.
I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ? |
|
|
651
|
Tue Jun 11 15:01:07 2024 |
PP TD | 16:00-00:00 shift Tuesday 11/06/24 |
All seems OK
Screenshots attached.
17.33 per FEE64 1.8.W spectra - attachments 8-9
per p+n FEE64 1.8.L spectra - attachment 10
aida09 pulser peak width 66 ch FWHM
cf
c. 11:00 today 57 ch FWHM https://elog.ph.ed.ac.uk/DESPEC/650
cf
c. 00:10 today 53 ch FWHM https://elog.ph.ed.ac.uk/DESPEC/649
17.48 ADC data item stats - attachment 11
All histograms zero'd
17.51 Analysis data file S181/R6_96 - attachment 12
max deadtime 5% (aida08)
HEC Data items 3.7kHz, LEC data items 1.8MHz
cf.
c. 12.29 today 1.1MHz https://elog.ph.ed.ac.uk/DESPEC/650
c. 00.09 today 1.1MHz https://elog.ph.ed.ac.uk/DESPEC/649 |
|
|
446
|
Sat May 14 03:14:00 2022 |
PP | Shift Checks |
4:13
AIDA stats OK
Leakage current OK
Temperatures OK
grafana OK
ucesb rates OK
System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Screenshot 1: statistics
Screenshot 2: temperatures
Screenshot 3: scalers
Screenshot 4: Bias and leakage current
6:36
AIDA stats OK
Leakage current OK
Temperatures OK
grafana OK
ucesb rates OK
System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Screenshot 5: statistics
Screenshot 6: temperatures
Screenshot 7: scalers
Screenshot 8: Bias and leakage current
|
|
|
590
|
Tue Apr 23 07:27:18 2024 |
PP | 08:00-16:00 Tuesday 24 April shift |
Checks 08:00-08:30
Screenshots are attached and everything looks OK. |
|
|
591
|
Tue Apr 23 09:21:01 2024 |
PP | 10:20 checks - aida02 is down |
Aida02 is down. Contacted Nick.
The rest looks stable, see screenshots.
10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14
Go > All 16 writing to merger, looks OK
aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again |
|
|
592
|
Tue Apr 23 10:10:31 2024 |
PP | aida02 is back |
Nick rebooted AIDA and aida02 came back at around 10:55.
AIDA was out of the DAQ from about 10:30 until 10:55 |
|
|
593
|
Tue Apr 23 10:52:39 2024 |
PP | Aida02 down and back |
We lost aida02 at about 11:25 again. Nick fixed it about 11:50.
It is now back in the DAQ. |
|
|
594
|
Tue Apr 23 11:09:00 2024 |
PP | 12:00 checks |
Everything is OK now.
Screenshots attached. |
|
|
597
|
Tue Apr 23 13:15:43 2024 |
PP | 14:00 checks |
All looks good.
Screenshots attached |
|
|
599
|
Tue Apr 23 19:23:42 2024 |
PP | 20:30 checks |
All looks good.
Screenshots attached. |
|
|
602
|
Wed Apr 24 15:37:59 2024 |
PP | 16:00-00:00 shift Wednesday 24 April |
All seems OK.
Screenshots attached. |
|
|
603
|
Wed Apr 24 18:44:05 2024 |
PP | 19:30 checks |
All seems smooth.
Screenshots attached. |
|
|
652
|
Tue Jun 11 18:45:37 2024 |
PP | Mid-shift checks, 19:45 |
All looks good.
Sceenshots attached. |
|
|
656
|
Wed Jun 12 11:06:55 2024 |
PP | Mid-shift checks, 12:00 |
All seems normal.
Screenshots attached. |
|
|
517
|
Wed Jan 18 13:40:33 2023 |
PJCS TD | MACB settings with either Emulator or VITAR |
When using the VETAR connected to the HDMI port of the root MACB the settings for all the MACB in the system 0x3.
This setting takes the Clock and Data line from the input HDMI and outputs it via teh HDMI output ports.
When using the Emulator connected via the SMA connectors on the back of the root MACB then the setting for the root MACB should be 0xD and all others should be 0x3.
Attached is the .jed file for programming the MACB and the .vhd source file to help with understanding of the settings. |