 Tue May 7 13:25:00 2024, JB, CC, NH, HMA, Tuesday 6 May - dismount snout 
|
Status before dismount
14:26 DSSSD#1 119.99 V @ 17.968 uA
DSSSD#2 119.95 V @ 11.212 uA
Leakage currents slightly elevated from Monday.
S4 temperature at 26 °C.
14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.
Plan:
- Dismount snout.
- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.
|
|
Wed May 5 16:57:48 2021, TD, Tuesday 5 May
|
17.55 Grafana DSSSD bias & leakage current - attachment 1
Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37
DSSSD bias & leakage currents - attachment 2 |
|
Tue Mar 5 09:10:01 2024, MA, JB, TD, Tuesday 5 March
|
Old ribbon cable assemblies - attachments 1 2
Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors
p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm
Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm
Manufacture of new ribbon cable assemblies - attachment 3
8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables
17.52 from Helena
"These are the detector positions within the snout:
Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame
Upstream AIDA @ 513 mm
Downstream AIDA @ 523 mm
This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by
10mm.
The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position." |
|
Tue Apr 5 08:52:16 2022, TD, Tuesday 5 April 6x
|
DSSSD bias OFF. DAQ STOPped
Attachment 1 FEE64 temps OK
Attachment 2 system wide checks OK *except* WR decoder (attachment 2) and FPGA ts errors no longer works
Attachment 6 System wide check - 'Synchronise ASIC clocks'
Attachment 5 System wide check - check ADC calibration - all 16x FEE64s fail
Attachment 4 System wide check - WR decoder status - aida09 WR status 0x10 clears
Repeat Synchronsie ASIC clocks
Attachment 3 System wide check - WR decoder status - aida05 WR status 0x10 clears
|
|
Thu Sep 6 15:11:17 2018, OH, TD, Tuesday 4 September - Thursday 6 September 19x
|
Attachment 1 - FEE64 temperatures illustrating (we assume) a faulty PSU temperature sensor for aida07
Attachment 2 - FEE64 temperatures following replacement of aida07
Attachment 3-5 - 1.*.W waveforms for aida01 - aida03 - LED settings for aida02 incorrect, aida01 adaptor PCB
incorrectly aligned
Attachment 6-7 - 1.*.W waveforms for aida02 and aida04
Attachment 8-9 - aida01 - aida04 1.8.L spectra full range and expanded scale
pulser peak width c. 15-20 channels FWHM
Attachment 10-12 - good events stats for all FEE64s and all statistics for aida01 and aida05
Note aida01 - aida04 show rates expected, all other FEE64s show very rates due to noise
aida01 - aida04 adaptor PCB are well grounded to FEE64 copper mezzanine/cooling plate, the other
FEE64s are not
Attachment 13-14 - aida01 - aida04 rate spectra before and after correcting alignment of aida01 adaptor PCB
Attachment 15 - aida01 - aida04 1.8.L spectra, expanded scale
Attachment 16 - illustrates high frequency noise (period 5 samples = 100ns) for aida01 which is not observed for
aida02 - aida03
subsequently determined that this is not due to a possible adaptor PCB fault by changing the adaptor
PCB
Note that with bias turned on this noise did appear when Bias was on but all the other groudning
connectors for aida05-aida12
were not in effect.
Attachment 17-19 - PuTTY connection to CAEN 1419ET 4ch high voltage supply illustrating PuTTY, N1419ET configuration
and leakage current with 100V applied to adaptor PCBs (no input cabling or DSSSD connected)
Note the CAEN N1419ET appears as a 'USB ACM' device and we connect using /dev/ttyACM<n>. |
|
Tue May 4 13:53:22 2021, OH, TD, Tuesday 4 May 7x
|
NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290
14.40 Rate spectra - attachment 1
adc data items - attachment 2
slow comparator 0xa
stats for p+n much improved, n+n remain poor
1.8.W spectra - attachments 3 & 4
1.8.L spectra - attachments 5 & 6
pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM
17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7 |
|
Tue May 31 09:05:00 2022, NH, TD, Tuesday 31 May
|
DSSD stack will be as follows
1 3208-7 1019 125 upstream
2 3208-13 1020 120 downstream
First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance
3208-7:
Top and bottom: 22 Ohm
Left and right: 22 Ohm
3208-13:
Top and bottom: 22 Ohm
Left and right: 22 Ohm
Conclusion: All bias bond wires are intact on both DSSDs
Visual inspection is both DSSDs look good with no damage or bond wire issues
Installed in snout: full stack
3 mm bPlas 1
17 mm Air
1 mm AIDA 1 (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2 |
|
Tue May 31 17:09:35 2022, TD, Tuesday 31 May 20x
|
2x MSL type BB18(DS)-1000 installed
Adaptor PCB LK config
all FEE64s LK2, LK4 fitted
n+n FEE64s LK1 fitted
bottom p+n FEE64s LK3 fitted
AIDA PSU config
top 1-3 2-4
bottom 5-7 6-8
DSSSD bias -> p+n FEE64 core, n+n FEE64 braid
Heavy duty grounds not connected to FEE64 adaptor PCBs
ASIC settings 2021Apr29-13-16-00
slow comparator 0xa
BNC PB-5 settings
amplitude 1.000V
attenuation x1
tau_d 1ms
frequency 25Hz
polarity +
Attachment 1 - DSSSD bias & Leakage current - OK
Attachment 2 - FEE64 temps OK
Attachment 3 - adc data item stats
Attachments 4-9 - system wide checks OK *except* aida07 fails ADC calibration
Attachments 10 & 11 - 1.8.W spectra 20us FSR
Attachments 12 & 13 - 1.8.L spectra - aida03 pulser peak width 155 ch FWHM
Attachment 14 - per FEE64 rate spectra
19.15 Attachments 15-18 - 2*W spectra for aida01, 02, 03 &04
Attachment 19 - per FEE64 rate spectra
Attachment 20 - adc data item stats |
|
Tue Apr 30 09:06:45 2024, TD, NH, Tuesday 30 April 7x
|
10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
9/16 FEE64s < 20k
per FEE64 Rate spectra - attachment 5
Merger etc - attachments 6-7
aida02, aida04 inactive
11.20 DAQ restarted
aida02 and aida04 now producing data
11.30 analysis data file R22_313
all deadtimes <1%
no timewarps
HEC data item rate 560Hz (all from one channel of aida08 asic #3)
aida03 & aida04 MBS rates c. 24kHz
14.32 FEE64 temperatures OK
ADC data item stats OK
all FEE64s producing data
14.43 all histograms & stats zero'd
[NH checks here]
21.26 aida04 stopped forwarding to merger
the "quick fix" (stop merger, rerun) worked
MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability
System Wide Checks
clocks PASSED
ADC IGNORED
WR:
Base Current Difference
aida02 fault 0xe4d8 : 0x65c0 : -32536
aida03 fault 0x2371 : 0x2380 : 15
aida04 fault 0xa210 : 0xaf58 : 3400
aida06 fault 0x3624 : 0x3626 : 2
White Rabbit error counter test result: Passed 12, Failed 4
FPGA:
Base Current Difference
aida02 fault 0xaa3b : 0xc455 : 6682
aida04 fault 0x5a4 : 0x6ba : 278
aida06 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1
Baselines reset
22.50 FEE64 temps ok
merger ok
all links active
ADC data item stats ok |
|
Tue Apr 2 07:39:44 2019, CB, TD, CA, Tuesday 2nd April 2019 36x
|
08:15 beam tuning in progress
08.30 System wide checks
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- 'collect memory information from FEE64 Linux' still returns scripting error -> will not check again until update
- WR decoder test passes
- good event statistics OK, except aida 09, 10, 11, 12 as yesterday morning (see attach 1)
- FEE temperature OK (see attach 2)
- Detector OK (see attach 3)
- NewMerger. MergerHTTPd, newMerger Terminal, data relay MBS screenshots attached (4,5,6,7)
10:37 Issues with magnets due to the new control system. Beam tuning continues
AIDA continues running OK
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- WR decoder test passes
- good event statistics OK, except aida 09, 10 (see attach 8). Aida 9 & 12 rates now lower
- FEE temperature OK
- Detector OK
- Merger OK
12:21 Beam start
AIDA continues running OK
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- WR decoder test passes
-ASIC check load performed
-Writing to file R11 in directory 290319
-histograms zeroed
-good event statistics OK, except aida 09, 10 (attachment 9)
-FEE temperatures ok (attachment 10)
-detector bias ok (attachment 11)
-thick degrader was placed in front of snout during beam operation
12:40 beam stop
TapeServer stopped, file R11_0 in 290319
TapeServer restarted, not writing to file
low energy spectra for aida09 (attachment 12)
high energy spectra for aida09 (attachment 13)
high energy spectra for aida11 (attachment 14)
1D hit patterns (stat) in aida09,10,11,12 (attachment 15)
ASIC control settings used during beam, including thresholds (attachment 16)
14.20 *NOTE degrader setting different from yesterday, silver beam still not observed, just unidentified light ions.
AIDA continues running OK
aida06 global clock failure (check clock status)
aida02 , aida04, aida06 fail calibration (ADC calibration)
SYNC error counter ok
SYNC pulse check ok
WR decoder test passes
14.37 good event statistics OK, except aida 10 (attachment 17)
detector bias ok (attachment 18)
FEE temperatures ok (attachment 19)
14.54 Degrader removed
16:13 AIDA continues running OK
aida06 global clock failure (check clock status)
aida02 , aida04, aida06 fail calibration (ADC calibration)
SYNC error counter ok
SYNC pulse check ok
WR decoder test passes
FEE64 temperatures OK
good event statistics OK, except aida 3, 9, 10 running hot (see attach 18)
17:00 Beam start - no degrader
AIDA running OK
Writing on file R13
Beam is better focused (see attach 19). Appears to be off-centre in the bottom right part of the detector
Rates OK (see attach 20)
DSSD bias/current OK (see attach 21)
17:21 Beam re-centred. Histograms cleared - see attach 22
AIDA continues OK
Run continues
High energy spectra in aida09 (attachment 25)
Beam rate approx 5000 ions/s
Beam being diffused
TapeServer start stop - AIDA writing to file R14 in 290319
17:48 1D hit patterns - beam in process of being diffused (attachment 26)
17:57 leakage current back at 1.465 uA (attachment 27)
18:08 beam stopped
TapeServer stopped and restarted - not writing to file
updated hit patterns from beam diffusing (attachment 28)
aida10, ADC3 high energy spectra (attachment 29)
18.11 calibrating tpc with masks
TapeServer stop/start R16, 290319
18.49 leakage current briefly fluctuating between ~1.2 uA and ~2 uA, but has returned to stability at ~ 1.4 uA
(attachment 30)
18.55 beam stopped for tuning - TapeServer stop/start. Not writing to file.
18.58 leakage current still stable after beam off (attachment 31)
19.10 beam start. TapeServer stop/start writing to file R18 in 290319
19.15 leakage current still ok after beam activation
19.34 fee12 high energy spectra (attachment 32)
20.35 beam stop, tapeserver stopped
histogramming link dropped on aida 12
20.38 beam start. TapeServer start R20
|
|
Mon Jun 27 23:11:47 2022, TD, Tuesday 28 June 00:00-08:00 30x
|
00:07 Zero stats & all histograms
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 -> 0xa
all waveform AD9252 ADCs disabled
all fast discs disabled
BNC PB-5 settings (to p+n FEE64s only)
amplitude 1.0V
attenuator x1
decay time 1ms
polarity +
frequency 22Hz
analysis of file S505/R5_896 - attachment 1
zero timewarps
deadtime all FEE64s << 1%
All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3
adc data item stats - attachment 4
FEE64 temps OK - attachment 5
DSSSD bias & leakage currents OK - attachment 6 & 7
00:15 Check ASIC control all FEE64s, all ASICs
02:03
analysis of file S505/R5_914 - attachment 8
zero timewarps
deadtime all FEE64s << 1%
per FEE64 1.8.H spectra - attachments 9 & 10
per p+n FEE64 1.8.L spectra - attachment 11
aida01 pulser peak width 94 ch FWHM
per FEE64 stat & rate spectra - attachments 12 & 13
All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15
adc data item stats - attachment 16
FEE64 temps OK - attachment 17
DSSSD bias & leakage currents OK - attachment 18
03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today
03:41
analysis of file S505/R5_926 - attachment 19
zero timewarps
deadtime all FEE64s << 1%
All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21
adc data item stats - attachment 22
FEE64 temps OK - attachment 23
DSSSD bias & leakage currents OK - attachment 24
07:00
analysis of file S505/R5_954 - attachment 25
zero timewarps
deadtime all FEE64s << 1%
All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27
adc data item stats - attachment 28
FEE64 temps OK - attachment 29
DSSSD bias & leakage currents OK - attachment 30 |
|
Tue Apr 27 08:26:01 2021, TD, Tuesday 27 April - aida09 system console reports 'received undefined information code: 1' and 'code: 9' 6x
|
09.25 All system wide checks OK *except*
Base Current Difference
aida01 fault 0xc960 : 0xc963 : 3
aida02 fault 0x8bbf : 0x8bc2 : 3
aida03 fault 0xae9e : 0xaea1 : 3
aida04 fault 0x6e3 : 0x6e6 : 3
aida05 fault 0x31a4 : 0x31ac : 8
aida05 : WR status 0x10
aida06 fault 0x22ea : 0x22f2 : 8
aida07 fault 0x438f : 0x4397 : 8
aida08 fault 0x61ed : 0x61f5 : 8
aida09 fault 0x1c32 : 0x1c33 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 18 24 16 3 1 3 2 2 3 4 : 27792
aida02 : 25 6 3 5 1 4 2 2 3 3 6 : 36260
aida03 : 19 10 3 6 3 3 1 3 3 3 6 : 36556
aida04 : 21 7 3 4 2 4 2 2 3 3 6 : 36284
aida05 : 40 37 16 13 4 3 2 2 3 3 4 : 28904
aida06 : 32 4 4 3 0 4 1 3 3 3 6 : 36416
aida07 : 26 22 18 13 3 3 2 4 3 4 3 : 27672
aida08 : 15 9 4 1 2 3 1 3 3 3 6 : 36324
aida09 : 29 41 11 19 3 4 3 4 3 3 4 : 30348
aida10 : 17 5 8 4 3 3 3 3 2 3 6 : 36012
aida11 : 25 9 3 5 1 1 2 4 2 3 6 : 35900
aida12 : 80 43 12 7 4 2 3 2 2 3 4 : 27960
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sat Apr 24 03:57:22 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
09.25 DSSSD bias |& leakage currents Ok - attachment 1
FEE64 temps OK - attachment 2
good event stats - attachment 3
aida09 zero rate
merger time seq errors - attachment 4
Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5
DAQ continues OK *except* aida09 - no data storage
10.00 from aida09 system console
24/03:04:39|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
24/03:04:39|Also in RDOGo_Operate: Enabled waveform readout
24/03:04:39|
24/03:04:39|get_WAVEBlk (C) : which = 0 , Status = 00001031 , Count = 0
24/03:04:39|get_WAVEBlk (A) which = 0 ---- status = 0540, State machines = 050A
24/03:04:39|WV:0:Buffer info. Length = 131072 : Offset = 0
24/03:04:40|WV:1:Buffer info. Length = 131070 : Offset = 2
24/03:04:40|WV:2:Buffer info. Length = 131068 : Offset = 4
24/03:04:40|WV:3:Buffer info. Length = 131066 : Offset = 6
24/03:04:40|WV:4:Buffer info. Length = 131064 : Offset = 8
24/03:04:40|WV:5:Buffer info. Length = 131062 : Offset = 10
24/03:04:40|WV:6:Buffer info. Length = 131060 : Offset = 12
24/03:04:40|WV:7:Buffer info. Length = 131058 : Offset = 14
24/03:04:40|WV:8:Buffer info. Length = 131056 : Offset = 16
24/03:04:40|WV:9:Buffer info. Length = 131054 : Offset = 18
24/03:04:40|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:40|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 804441A5 : 1 => 09CD0000
24/03:04:40|Last Data : Databuffer 274139128 : 0 => C01B7F88 : 1 => 0C05B135
24/03:04:40| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51
24/03:04:40| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A
24/03:04:40| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26
24/03:04:41| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E
24/03:04:41| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2
24/03:04:41| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B
24/03:04:41| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC
24/03:04:41| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E
24/03:04:41|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C03A7F13 : 1 => 0C05B455
24/03:04:41|Last Data : Databuffer 274139128 : 0 => C0127D9D : 1 => 0DB9DEED
24/03:04:41| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51
24/03:04:41| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A
24/03:04:41| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26
24/03:04:42| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E
24/03:04:42| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2
24/03:04:42| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B
24/03:04:42| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC
24/03:04:42| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E
24/03:04:42|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:42|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C02D7E26 : 1 => 0DB9DEED
24/03:04:42|Last Data : Databuffer 274139128 : 0 => C0217F73 : 1 => 0F816B3D
24/03:04:42| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81
24/03:04:42| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9
24/03:04:42| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94
24/03:04:43| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA
24/03:04:43| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003
24/03:04:43| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98
24/03:04:43| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F
24/03:04:43| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54
24/03:04:43|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C02D7EB4 : 1 => 0F816D95
24/03:04:43|Last Data : Databuffer 274139128 : 0 => C0167F53 : 1 => 0148F2DD
24/03:04:43| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81
24/03:04:43| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9
24/03:04:43| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA
24/03:04:44| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003
24/03:04:44| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98
24/03:04:44| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F
24/03:04:44| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54
24/03:04:44|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0207E17 : 1 => 0148F2DD
24/03:04:44|Last Data : Databuffer 274139128 : 0 => C0177F46 : 1 => 047FF0CD
24/03:04:44| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81
24/03:04:44| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9
24/03:04:44| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA
24/03:04:45| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003
24/03:04:45| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98
24/03:04:45| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F
24/03:04:45| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:47|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:56|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:05|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:05:15|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:24|197 spectra zeroed
25/10:21:10|do_GetState returned z=0 and 1
25/10:21:11|Data Acquisition Statistics counters now cleared
25/10:24:01|Clear Statistics (1)
25/10:24:01|bad event data (0x45000034 0x84e64000)
26/16:52:50|bad event data (0x40061e21 0xc0a80b63)
26/16:52:50|received undefined information code: 1
26/16:52:50|bad event data (0x0101080a 0xe9b68def)
26/16:52:50|bad event data (0x03527827 0x0101050a)
26/16:52:50|received undefined information code: 9
From system console
Apr 24 02:52:42 aida09 kernel: IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
Apr 24 02:52:42 aida09 kernel: IP-Config: Complete:
Apr 24 02:52:42 aida09 kernel: device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
Apr 24 02:52:42 aida09 kernel: host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
Apr 24 02:52:42 aida09 kernel: bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100003/2 on 192.168.11.99
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100005/1 on 192.168.11.99
Apr 24 02:52:43 aida09 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Apr 24 02:52:43 aida09 kernel: Freeing unused kernel memory: 168k init
Apr 24 02:52:43 aida09 kernel: xaida: device parameters: base=0x81000000 size=0x200000
Apr 24 02:52:44 aida09 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Apr 24 02:52:44 aida09 kernel: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Apr 24 02:52:44 aida09 kernel: xaida: driver assigned major number 254
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service telnet, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service ftp, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Apr 24 02:52:44 aida09 xinetd[273]: Started working: 2 available services
Apr 24 02:52:48 aida09 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: driver assigned major number 253
Apr 24 02:53:04 aida09 kernel: xaida: open:
Apr 24 02:53:04 aida09 kernel: AIDAMEM: aidamem_open:
Apr 24 02:57:33 aida09 kernel: xaida: open:
Apr 24 13:39:36 aida09 xinetd[273]: START: telnet pid=396 from=192.168.11.99
Apr 24 13:42:05 aida09 xinetd[273]: EXIT: telnet status=0 pid=396 duration=149(sec)
Apr 27 09:01:50 aida09 xinetd[273]: START: telnet pid=422 from=192.168.11.99 |
|
Tue May 25 11:16:54 2021, TD, Tuesday 25 May 7x
|
12.16 DAQ continues file S496/R18_324
alpha background
Tape server c. 2.5Mb/s
ASIC check
Tape server c. 300kb/s - presumably hot HEC channels
12.20 All system wide checks OK *except*
Base Current Difference
aida01 fault 0x82b0 : 0x82b1 : 1
aida02 fault 0xfe72 : 0xfe73 : 1
aida03 fault 0x97f4 : 0x97f5 : 1
aida04 fault 0x950 : 0x951 : 1
aida05 fault 0x8e93 : 0x8e94 : 1
aida06 fault 0x545a : 0x545b : 1
aida07 fault 0xc04d : 0xc04e : 1
aida08 fault 0xa0f7 : 0xa0f8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 2 6 6 2 1 3 2 2 2 4 6 : 37016
aida02 : 3 6 5 3 3 4 3 2 1 3 7 : 38572
aida03 : 21 10 8 6 3 3 2 3 2 4 6 : 37924
aida04 : 29 15 4 5 3 4 2 3 3 3 6 : 37004
aida05 : 17 5 2 2 2 2 2 2 2 4 6 : 36940
aida06 : 1 6 0 1 2 2 2 3 3 4 6 : 38356
aida07 : 11 6 7 4 3 3 3 3 2 4 6 : 38028
aida08 : 13 8 3 2 2 3 3 2 1 3 7 : 38372
aida09 : 7 9 2 1 3 3 3 3 2 4 6 : 37860
aida10 : 9 8 4 2 0 3 3 3 1 4 6 : 36708
aida11 : 16 6 3 2 2 3 3 3 1 4 6 : 36832
aida12 : 21 10 7 4 2 4 2 2 1 3 7 : 38420
aida13 : 8 3 3 1 3 3 3 3 2 4 6 : 37832
aida14 : 14 3 6 1 0 4 2 2 2 4 6 : 37072
aida15 : 2 4 2 2 1 3 3 4 2 4 6 : 38216
aida16 : 16 5 5 3 0 3 2 2 2 4 6 : 37016
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
Rate spectra - attachment 2
ADC data item stats - attachment 3
FEE64 temps OK - attachment 4
DSSSD bias & leakage current OK - attachment 5
Merger/Tape Server/Merger stats - attachment 6
no merger errors reported since previous restart
17.30 analysis of file S496/R18_340
max deadtime 0.36% aida04 |
|
Tue Apr 23 15:11:50 2024, TD, Tuesday 23 April 16:00-00:00 20x
|
16.12 Analysis of data file R17_101 - attachment 1
max deadtime c. 2% (aida02)
no timewarps observed
16.15 DSSSD bias & leakage current OK - attachment 2
FEE64 temperatures OK - attachment 7
ADC data item stats - attachment 3
per FEE64 rate spectra (on and off spill) - attachments 4-5
Merger etc - attachment 6
16.53 aida04 stops producing data
DAQ STOP fails for aida04
Restart Merger to enable aida04 to complete DAQ STOP
aida04 rebooted and DAQ restarted OK
data file S100_alpha/R18 |
|
Tue Jun 21 10:19:05 2022, OH ML, Tuesday 21st June 08:00-00:00 12x
|
11:19 Stats ok 0xa on all FEE - attachment 1
Current pulser settings - attachment 2
FEE Temperatures ok - attachment 3
Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
Layout 1 after recovering channels - attachment 4
New stats (Note increased rate on 3 and 7) attachment 5
Bias and leakage currents ok - attachment 6
S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.
14:05 Have had it confirmed that the flange beam dump is in position in S4
15:43 Stats ok -attachment 7
Temp ok - attachment 8
Bias and Leakage current ok - attachment 9
17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
During the experiment this will be changed to SC1 trigger to solve this.
19:49: Stats ok - attachment 10
temperatures ok - attachment 11
Bias - Leakage current ok - attachment 12 |
|
Tue Apr 20 19:00:00 2021, JS, Tuesday 20th 20:00-00:00 12x
|
20:00
Taking over from Liam, all systems good.
20:30
DB No faults.
Stats good
ucesb ok
20:58
DB No faults.
Stats good
Temps good
ucesb ok
21:31
DB No faults.
Stats good
Temps good
ucesb ok
22:02
DB No faults.
Stats good elog:254/3
Temps good elog:254/4
ucesb ok elog:254/1
Current ok elog:254/2
ADC Calibration check:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
22:06 ucesb - DAQ Error - Onsite team checking daq crash
Tape server icon stopped spinning
Stats Rates 0, Counter aida01 zeroed but others normal. elog:254/5
Called Tom. Tried resetting in midas, no luck. Full reboot of DAQ and midas required.
According to log at 22:07 aida01 reset
Data taking stopped at 22:04 file R45_193
22:50
After a full power cycle all appear to be up and running well. Starting full checks.
23:00
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
Check FPGA is having a page error
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k
4k
aida01 : 4 8 2 2 1 3 1 4 2 3 7 :
39792
aida02 : 13 9 2 4 2 3 2 3 2 3 7 :
39708
aida03 : 19 5 4 3 1 3 1 4 2 3 7 :
39892
aida04 : 14 5 2 2 0 5 2 3 2 3 7 :
39744
aida05 : 16 11 3 2 2 4 3 2 1 4 7 :
40584
aida06 : 15 9 5 2 2 2 1 4 2 3 7 :
39828
aida07 : 15 9 3 3 2 2 1 4 2 3 7 :
39828
aida08 : 21 4 3 4 2 2 1 4 2 3 7 :
39844
aida09 : 11 6 1 2 3 4 2 3 2 3 7 :
39788
aida10 : 0 6 4 2 0 3 2 3 2 3 7 :
39472
aida11 : 16 7 2 0 1 3 2 3 2 3 7 :
39512
aida12 : 3 7 4 1 3 2 2 3 2 3 7 :
39524
DB No faults.
Stats good elog:254/6
Temps good elog:254/7 aida01 had been high during reboot issue, >70, but now is back down
Current ok elog:254/8
ucesb ok
23:25
DB No faults.
Stats good
Temps good
ucesb ok
23:42 No Beam
23:48 Beam Back
23:57
DB No faults.
Stats good elog:254/11
Temps good elog:254/12
ucesb ok elog:254/9
Current ok elog:254/10
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
*FPGA check page error*
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 9 5 2 1 2 1 4 2 3 7 : 39812
aida02 : 13 9 5 2 2 2 1 4 2 3 7 : 39820
aida03 : 21 8 3 2 2 2 1 4 2 3 7 : 39812
aida04 : 13 4 3 1 1 5 2 3 2 3 7 : 39780
aida05 : 24 5 3 2 2 4 2 3 2 3 7 : 39800
aida06 : 16 5 5 2 2 2 1 4 2 3 7 : 39800
aida07 : 22 6 2 2 1 3 1 4 2 3 7 : 39848
aida08 : 21 8 2 3 2 2 1 4 2 3 7 : 39828
aida09 : 18 7 1 1 3 4 2 3 2 3 7 : 39792
aida10 : 19 3 3 1 0 3 2 3 2 3 7 : 39476
aida11 : 17 4 4 0 1 3 2 3 2 3 7 : 39524
aida12 : 13 4 3 2 3 1 2 3 2 3 7 : 39428
00:04 Handing over to Corrigan |
|
Tue Apr 20 14:56:38 2021, LPG, Tuesday 20th 16:00-20:00 15x
|
16:00
New shift, ready to go! Beam stopped just as I started... Waiting for update.
Beam back at 16:08, ucesb scalers shown in elog:253/1
Database ok!
Stats ok!
ucesb ok!
Temperatures ok! elog:253/2
Statistics ok! elog:253/3
Leakage currents ok! Steadily rising, day/night effect? - elog:253/4 and elog:253/5
16:21
Online histograms zeroed in MIDAS. Same done in MBS DAQ.
16:45
Database ok!
Stats ok!
ucesb ok!
System wide checks:
Clock ok!
ADC 1 failed:
FEE64 module aida06 failed
WR ok!
FPGA ok!
Memory ok!
17:15
Database ok!
Stats ok!
ucesb ok!
17:45
Database ok!
Stats ok!
ucesb ok!
18:15
Database ok!
Stats ok!
ucesb ok!
System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!
MIDAS Rates ok! elog:253/6
Temperatures ok! elog:253/7
Leakage currents ok, but still slowly rising! Topped out at ~20:30 yesterday elog:253/8
18:42
Message in the Merger terminal window (elog:253/9)
MERGE Actor (15671): Working with 0 from 12 data sources
19:05
Database ok!
Stats ok!
ucesb ok!
19:30
Some MIDAS spectra in elog:253/10 and elog:253/11
Database ok!
Stats ok!
ucesb ok!
19:50
Database ok!
Stats ok!
ucesb ok! elog:253/12
System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!
MIDAS Rates ok! elog:253/13
Temperatures ok! elog:253/14
Leakage currents ok, now plateaued! elog:253/15
19:55 - Handing over to James  |
|
Tue May 18 18:47:25 2021, LS, Tuesday 18th May 20:00 - 24:00 24x
|
19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x520 : 32
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~1.8M items/s
Tape server ~7MB/s
Analysed R14_399, deadtimes are okay (Fig 4)
20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)
21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)
21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)
22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x524 : 36
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~2.3M items/s
Tape server ~8MB/s
Analysed R14_430, deadtimes are okay (Fig 14)
22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%
23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)
23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)
24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x529 : 41
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The difference has increased over time (not sure if an issue)
Merger rate ~3M items/s
Tape server ~8MB/s
Analysed R14_462, deadtimes are okay (Fig 24) |
|
Tue May 18 14:57:43 2021, CB, Tuesday 18th May 16:00 - 20:00 11x
|
15:58 Took shift over from OH and PP
Stats OK - attach 1
Temps OK - attach 2
Bias OK - attach 3
System-wide checks
Clock - all pass
ADC - all fail (as before)
Base Current Difference
aida05 fault 0x500 : 0x510 : 16
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA - all pass
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 8 7 4 1 4 2 2 2 2 7 : 37320
aida02 : 12 9 4 2 4 2 3 3 3 3 6 : 36856
aida03 : 19 6 7 2 2 2 2 4 3 3 6 : 37036
aida04 : 18 9 10 6 1 4 1 4 3 3 6 : 37168
aida05 : 17 12 2 1 2 3 3 3 3 3 6 : 36836
aida06 : 19 8 7 7 2 4 2 4 3 3 6 : 37468
aida07 : 28 5 4 1 2 4 2 3 2 2 7 : 37752
aida08 : 17 5 4 2 1 2 2 3 3 3 6 : 36396
aida09 : 21 10 4 5 4 4 3 3 3 3 6 : 37252
aida10 : 15 13 4 2 2 2 3 2 2 4 6 : 37284
aida11 : 22 13 3 3 4 3 1 4 3 3 6 : 37072
aida12 : 2 8 1 3 1 3 2 3 3 3 6 : 36472
aida13 : 23 5 4 4 1 4 3 2 3 3 6 : 36484
aida14 : 19 7 4 3 1 3 2 4 2 4 6 : 38116
aida15 : 20 10 2 2 3 2 3 2 3 3 6 : 36288
aida16 : 29 6 1 2 0 3 1 4 3 3 6 : 36724
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
Analysed R14_342. Attach 4. Dead time <5%
17:00
Stats OK - attach 5
Temps OK - as before
Bias increasing quite quickly - attach 6
System-wide checks
System-wide checks
Clock - all pass
ADC - all fail (as before)
Base Current Difference
aida05 fault 0x500 : 0x512 : 18
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA - all pass
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 28 7 8 4 1 4 2 2 2 2 7 : 37352
aida02 : 24 11 6 4 3 2 3 2 3 3 6 : 36440
aida03 : 17 11 5 2 1 2 2 3 3 3 6 : 36460
aida04 : 25 13 10 6 2 4 2 3 3 3 6 : 37036
aida05 : 23 7 3 2 2 3 3 3 3 3 6 : 36868
aida06 : 1 12 3 8 2 4 3 3 3 3 6 : 37140
aida07 : 22 12 1 1 2 4 2 2 2 2 7 : 37224
aida08 : 21 8 7 2 1 3 3 2 3 3 6 : 36356
aida09 : 23 7 4 6 4 3 2 4 3 3 6 : 37396
aida10 : 27 7 2 2 2 2 3 3 2 4 6 : 37764
aida11 : 12 5 3 5 4 3 1 4 3 3 6 : 37032
aida12 : 23 9 4 1 2 3 2 3 3 3 6 : 36612
aida13 : 10 12 6 3 1 4 3 2 3 3 6 : 36488
aida14 : 23 7 3 3 1 3 2 3 2 4 6 : 37604
aida15 : 22 6 6 4 2 2 3 2 3 3 6 : 36328
aida16 : 27 7 3 1 0 3 1 4 3 3 6 : 36724
18:00
Stats OK - attach 7
Temps OK - as before
Bias - attach 8
System-wide checks - as before except
Base Current Difference
aida05 fault 0x500 : 0x517 : 23
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Analysed R14_371. Attach 9. Dead time below 5%.
18:35 MBS DAQ crashed
18:40
Stats OK - attach 9
Temps OK - as before
Bias - attach 10
System-wide checks - as before except
Base Current Difference
aida05 fault 0x500 : 0x51a : 26
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
18:42 MBS DAQ restarted - new file being opened.
19:36
Stats OK - attach 11
Temps OK - attach 12
Bias - attach 13
System-wide checks
Clock OK
ADC all fail
Base Current Difference
aida05 fault 0x500 : 0x51e : 30
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA OK
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 24 11 5 4 1 4 2 3 2 2 7 : 37832
aida02 : 0 5 6 3 4 2 3 2 3 3 6 : 36328
aida03 : 15 8 5 3 2 2 1 3 4 3 6 : 37292
aida04 : 14 8 7 5 3 4 2 3 3 3 6 : 36936
aida05 : 31 6 1 1 2 3 3 3 2 4 6 : 37852
aida06 : 9 8 3 8 2 4 3 3 3 3 6 : 37140
aida07 : 21 7 2 1 2 4 1 3 2 2 7 : 37452
aida08 : 8 4 5 3 1 3 2 3 3 3 6 : 36528
aida09 : 24 7 3 5 4 3 2 4 3 3 6 : 37352
aida10 : 19 9 7 1 2 2 3 2 2 4 6 : 37284
aida11 : 27 5 0 5 4 3 1 4 3 3 6 : 37044
aida12 : 31 9 2 1 2 3 2 3 3 3 6 : 36612
aida13 : 21 10 4 3 1 4 3 2 3 3 6 : 36484
aida14 : 22 6 4 2 1 3 2 4 2 4 6 : 38088
aida15 : 21 9 3 4 2 2 3 2 3 3 6 : 36300
aida16 : 23 11 4 2 1 4 2 3 3 3 6 : 36724
19:43 Shift handed over to LS |
|
Tue May 18 06:50:53 2021, OH, PP, Tuesday 18th May 08:00 - 16:00 6x
|
08:00 OH Takes over
Statistics still look good from yesterday. Rate is still down.
Dead time in FEEs 2 and 4 is 2.04%
System wide checks clock ok
Base Current Difference
aida05 fault 0x4da : 0x500 : 38
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
*Baseline reset*
FPGA check ok
Stats - attachment 1
Temp - attachment 2
Bias - attachment 3
08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips.
I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4
The results of the test appear to indicate that it is the network link that we are saturating causing dead time
*** Before After
FEE DT (s) Dead time % DT (s) Dead time %
0 0.013 0.004184451 0.026 0.010025681
1 6.352 2.04458693 4.097 1.57981599
2 0.295 0.09495484 0.292 0.112596112
3 6.349 2.043621288 8.255 3.183153771
4 0.021 0.006759497 0.03 0.011568094
5 0.285 0.091736032 3.388 1.306423377
6 0.02 0.006437616 0.149 0.057454865
7 1.219 0.392372712 2.133 0.822491459
8 1.137 0.365978485 0.457 0.176220627
9 0 0 0 0
10 1.633 0.525631369 1.545 0.595756823
11 0.061 0.01963473 0.166 0.064010118
12 0.059 0.018990968 0.085 0.032776265
13 0 0 0 0
14 0.016 0.005150093 0.023 0.008868872
15 0.282 0.09077039 0.168 0.064781324
Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased
09:00 They are stopping the run to decrease the bias on the bPlast detector
09:25 No noticable change in AIDA
They will now begin to change the spill structure.
09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change
10:21 Beam is back on AIDA
10:39 System wide checks.
- ADC calibration check failed (this is anticipated because waveforms are off).
- 1 Failed test in White Rabbit:
Base Current Difference
aida05 fault 0x500 : 0x506 : 6
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed
11:00
Stats - attachment 4
Temp - attachment 5
Bias - attachment 6
12:46 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off).
- 1 Failed test in White Rabbit:
Base Current Difference
aida05 fault 0x500 : 0x50b : 11
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed
14:36 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off).
- 1 Failed test in White Rabbit:
Base Current Difference
aida05 fault 0x500 : 0x50f : 15
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed |
|