|
|
|
AIDA
GELINA
BRIKEN
nToF
CRIB
ISOLDE
CIRCE
nTOFCapture
DESPEC
DTAS
EDI_PSA
179Ta
CARME
StellarModelling
DCF
K40
|
| DESPEC |
 |
|
|
|
Message ID: 250
Entry time: Mon Apr 19 23:01:02 2021
|
| Author: |
OH |
| Subject: |
20th April 00:00-08:00 |
|
|
23:52 Noticed AIDA07 has dropped out of the merger and is producing no statistics
Powercycling to recover
00:14 All FEEs recovered from powercycle
aida09 running again at HEC threshold of 0x2 (Was set to 0x1 earlier in the day while testing the HEC event disparity in DSSD3)
Now writing to R34
Following the powercycle the rates in the FEEs are much improved and back to where they were before the 08:30 crash on the 19th. - Attachment 1
Waveforms also look improved - attachmens 2 and 3
Current system parameters
ASIC settings Dec19 (All LEC comparator 0xa)
Pulser at 2V and 2Hz
DSSD 1 and 2 Bias 160V
DSSD 3 at 120V (Will lower to 100 again during the beam off period tomorrow)
00:28 In analysis of R34_13 the number of HEC items in DSSD3 appears to make sense again. The number in aida09 are consistent with the number in aida05 and the same for aida11 and aida07.
This appears to confirm that the earlier HEC issues we were observing were hardware related but we are unsure what/why/how these issues were caused.
Analysis - attachment 4
00:40 Issue with one of the FRS magnets so they stop recording data. Leaving AIDA running.
01:08 While beam was down have reset DSSD3 bias to 100V as was agreed with Tom and Nic in email
Leakage current decreased as expected - attachment 5
Once beam is back will check HEC still as expected.
01:14 Last magnet in FRS has an issue with the water flowing. Either sensor issue or water flow is to low.
Hakke have increased the water flow to hopefully keep the water running.
Beam is back.
From ucesb it seems that the HEC events in DSSD3 are still appearing as expected. Will check a file once we have a complete file.
Checked R34_36 and the HEC events in DSSD3 are still consistent - attachment 6
2:20 Statistics ok - attachment 7
Temp - attachment 8
Bias and leakage current ok - attachment 9
All system wide checks ok except for WR error check:
Base Current Difference
aida07 fault 0x9154 : 0x915b : 7
aida10 fault 0x8d3 : 0x8d3 : 0
aida10 : WR status 0x10
White Rabbit error counter test result: Passed 10, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Note aida07 reports two bad timestamps in the merger
MERGE Data Link (3762): bad timestamp 6 3 0xc1817ea7 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
MERGE Data Link (3762): bad timestamp 6 3 0xc1b77e25 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
02:25 Remembered during a pre-experiment meeting with Vic, Patrick and Carl they mentioned the merger would track timestamp sequence errors
Can confirm this is true. Can see the two in aida07 in the counter in the merger stats page - attachment 10
02:45 Now up to 7 timestamp errors on aida07 in the merger
02:49 Ran R34_45 through AIDASort - It reports a greater number of implant events are able to have a front back match performed
Ratio between aida07 and 09 is 0.71 -> Double what it was before
Energy response of the HEC channels in aida07 and aida09 looks equivalent (left group aida09 right group aida11) - attahchment 11
04:40 Statistics ok - attachment 12
Temperatures ok - attachment 13
Bias and leakage currents - attachment 14
Clock status, ADC calibration and memory infomration all ok
aida07 fails both WR and FPGA
Base Current Difference
aida07 fault 0x915b : 0x9161 : 6
aida10 fault 0x8d3 : 0x8d3 : 0
aida10 : WR status 0x10
White Rabbit error counter test result: Passed 10, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
TS errors in the merger still only at 7 with all in aida07
06:25 Statistics - attachment 15
Temps ok - attachment 16
Bias and leakage currents ok - attachment 17
System wide checks: clock status, adc calibration and memory information all ok
WR multiple fails
Base Current Difference
aida05 fault 0xc9ca : 0xc9cb : 1
aida06 fault 0xe900 : 0xe901 : 1
aida07 fault 0x915b : 0x9166 : 11
aida08 fault 0x641e : 0x641f : 1
aida10 fault 0x8d3 : 0x8d3 : 0
aida10 : WR status 0x10
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA aida07 fails with difference of 1
No new TS sequence errors in merger
07:10 Beam has dropped out.
07:14 Beam is back |
|
|
|
|
|
|
|
*** TDR format 3.3.0 analyser - TD - January 2019
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 255975543 ( 1356743.5 Hz)
Other data format: 5944457 ( 31507.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 32703 ( 173.3 Hz)
WR48-63: 32703 ( 173.3 Hz)
FEE64 disc: 1073243 ( 5688.5 Hz)
MBS info: 4805808 ( 25472.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 345437 ( 1830.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 188.669 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 46.534
2 0.000 63.789
3 0.000 54.847
4 0.000 174.519
5 0.000 123.945
6 0.000 113.898
7 0.000 135.923
8 0.000 59.774
9 0.000 27.296
10 0.000 12.263
11 0.000 65.598
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 14400003 100475 0 0 0 0 1767 1767 95813 1128 0 35602
1 23251868 63483 0 0 0 0 2988 2988 57507 0 0 21204
2 17192461 2025458 0 0 0 0 2415 2415 200396 1820232 0 34093
3 20019932 1908259 0 0 0 0 2815 2815 81941 1820688 0 32715
4 3059738 405674 0 0 0 0 403 403 100680 304188 0 35774
5 10595698 355900 0 0 0 0 1322 1322 58122 295134 0 21319
6 12037952 388178 0 0 0 0 1554 1554 105230 279840 0 34521
7 9390918 368608 0 0 0 0 1205 1205 81600 284598 0 31165
8 46854628 72459 0 0 0 0 5872 5872 60715 0 0 21758
9 43874737 60599 0 0 0 0 5426 5426 49747 0 0 17382
10 36184433 112868 0 0 0 0 4528 4528 103812 0 0 31745
11 19113175 82496 0 0 0 0 2408 2408 77680 0 0 28159
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 1641.051s ( 19.500 blocks/s, 1.219 Mb/s)
|
|
|
|
*** TDR format 3.3.0 analyser - TD - January 2019
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 255999603 ( 1309621.8 Hz)
Other data format: 5920397 ( 30287.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 32728 ( 167.4 Hz)
WR48-63: 32728 ( 167.4 Hz)
FEE64 disc: 1044504 ( 5343.4 Hz)
MBS info: 4810437 ( 24608.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 327887 ( 1677.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 195.476 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 51.325
2 0.000 68.599
3 0.000 55.115
4 0.000 180.174
5 0.000 140.295
6 0.000 123.427
7 0.000 138.446
8 0.000 55.510
9 0.000 13.463
10 0.000 24.357
11 0.000 74.336
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 14611253 94841 0 0 0 0 1936 1936 89802 1167 0 33511
1 23423897 61346 0 0 0 0 3006 3006 55334 0 0 20222
2 17311487 2050104 0 0 0 0 2477 2477 195362 1849788 0 32021
3 20533060 1944396 0 0 0 0 2862 2862 77493 1861179 0 30741
4 2942569 381933 0 0 0 0 417 417 93930 287169 0 33667
5 9776355 338395 0 0 0 0 1239 1239 56110 279807 0 20096
6 11887130 369619 0 0 0 0 1555 1555 103862 262647 0 32503
7 9239609 348065 0 0 0 0 1200 1200 76985 268680 0 29470
8 49507336 72711 0 0 0 0 6129 6129 60453 0 0 21190
9 44170393 65100 0 0 0 0 5430 5430 54240 0 0 18016
10 32977762 112783 0 0 0 0 4022 4022 104739 0 0 30280
11 19618752 81104 0 0 0 0 2455 2455 76194 0 0 26170
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 5995.524s ( 5.337 blocks/s, 0.334 Mb/s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|