| |
ID |
Date |
Author |
Subject |
|
|
456
|
Sun May 15 07:40:52 2022 |
OH | Sunday 15th May 08:00-16:00 |
08:40 FEE Temperatures ok - attachment 1
FEE statistics - Attachment 2
Bias and leakage currents ok - attachment 3
ASIC check ok
All system wide checks ok - Except aida09 fails clock check with bit 6
11:00 FEE statistics - attachment 4
FEE temperatures all ok - attachment 5
BIAS and leakage currents - attachment 6
ASIC check ok
All system wide checks as before
Currently on file R4_1030
13.1 MB/s to disk
4.8 million items per second
13:13 FEE statistics - attachment 7
FEE Temperatures all ok - attahcment 8
Bias and leakage currents ok - attachment 9
ASIC check ok
All system wide checks as before
Currently on file R4_1078
13.1 MB/s to disk
4.8 million items per second
15:35 Analysis of R4_1128 - attachment 10 |
| Attachment 1: Screenshot_from_2022-05-15_08-38-04.png
|
|
| Attachment 2: Screenshot_from_2022-05-15_08-38-34.png
|
|
| Attachment 3: Screenshot_from_2022-05-15_08-39-12.png
|
|
| Attachment 4: Screenshot_from_2022-05-15_11-00-32.png
|
|
| Attachment 5: Screenshot_from_2022-05-15_11-00-50.png
|
|
| Attachment 6: Screenshot_from_2022-05-15_11-01-02.png
|
|
| Attachment 7: Screenshot_from_2022-05-15_13-13-50.png
|
|
| Attachment 8: Screenshot_from_2022-05-15_13-14-12.png
|
|
| Attachment 9: Screenshot_from_2022-05-15_13-14-31.png
|
|
| Attachment 10: R4_1128
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 259195371 ( 1637875.8 Hz)
Other data format: 2724629 ( 17217.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 2455 ( 15.5 Hz)
RESUME: 2456 ( 15.5 Hz)
SYNC100: 32590 ( 205.9 Hz)
WR48-63: 32590 ( 205.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 2654538 ( 16774.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 38026 ( 240.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 158.251 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 4.735 0.000
2 0.000 0.000
3 16.674 0.000
4 0.000 0.000
5 77.034 0.000
6 0.000 0.000
7 33.677 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 12.872
11 17.247 0.000
12 0.000 0.000
13 0.000 8.200
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 7464928 2802 0 0 0 0 927 927 0 948 0 661
1 38695355 10162 0 0 254 254 4827 4827 0 0 0 32843
2 6890959 1397581 0 0 0 0 1028 1028 0 1395525 0 633
3 41303989 1261446 0 0 398 398 5227 5227 0 1250196 0 331
4 4066993 1070 0 0 0 0 535 535 0 0 0 242
5 37291979 10883 0 0 737 738 4704 4704 0 0 0 222
6 12390385 7719 0 0 0 0 1626 1626 0 4467 0 394
7 42901955 15088 0 0 626 626 5217 5217 0 3402 0 167
8 10872099 2752 0 0 0 0 1376 1376 0 0 0 592
9 4061050 1014 0 0 0 0 507 507 0 0 0 432
10 2113468 548 0 0 0 0 274 274 0 0 0 291
11 41561869 11172 0 0 440 440 5146 5146 0 0 0 468
12 6798748 1722 0 0 0 0 861 861 0 0 0 436
13 2781594 670 0 0 0 0 335 335 0 0 0 314
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 33.348s ( 959.588 blocks/s, 59.974 Mb/s)
|
|
|
464
|
Tue May 17 06:58:51 2022 |
OH | Tuesday 17th May |
08:00 Still have beam for the time being.
Currently on file R4_2049
FEE Stats - attachment 1
FEE Temps ok - attachment 2
Bias and leakage currents - attachment 3
System wide checks all ok *except aida09 fails clock check with value 6 which is ok*
Plan for the morning is once beam is lost is Ge group will enter and place source on the snout to perform efficiency measurement.
Once they have left the area and are taking data a pulser walkthrough will be performed.
Patrick can then have access to the system
08:16 Beam taken away
Data taken stopped on file R4_2055
Will perform a pulser walkthrough now
Pulser walkthrough started on R5
1V->0.1V in 0.1V steps
Pulser setting - attachment 4
p+n pulser peaks - attachment 5
n+n pulser "peaks" - attachment 6 |
| Attachment 1: Screenshot_from_2022-05-17_08-00-50.png
|
|
| Attachment 2: Screenshot_from_2022-05-17_08-01-13.png
|
|
| Attachment 3: Screenshot_from_2022-05-17_08-01-25.png
|
|
| Attachment 4: Screenshot_from_2022-05-17_08-49-02.png
|
|
| Attachment 5: Screenshot_from_2022-05-17_08-51-19.png
|
|
| Attachment 6: Screenshot_from_2022-05-17_08-51-52.png
|
|
|
|
484
|
Sun Jun 19 12:36:10 2022 |
OH | Sunday 19th June |
Yesterday the sum/inverter was found to be a considerable source of noise in the system.
Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace
Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow
Put inverter back in CAEN crate in S4 and confirm similar behavior.
Move CAEN crate to new mains port, same behaviour
Move CAEN crate to AC voltage stabiliser, same behaviour
Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears
Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.
MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd
CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?
MACBs and inverter checked on another crate - no signal
Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected
Tested adding bias filters - No change
15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
We get ok stats even with waveforms on at 0xa- attachment 5
Waveforms still have the 100kHz oscillation in them though attachments 6 and 7
Without waveforms we get similar performance to before the switch to the new mains - attachment 8
15:56 Move CAEN NIM crate across to this mains network
300+kHz on all FEEs with waveforms on. Can recover waveform off performance though
16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.
17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9 |
| Attachment 1: 1.jpg
|
|
| Attachment 2: 2.jpg
|
|
| Attachment 3: 3.jpg
|
|
| Attachment 4: 4.jpg
|
|
| Attachment 5: 220618_1753_stats_no_waveform.png
|
|
|
|
493
|
Thu Jun 23 06:56:02 2022 |
OH | Thursday 23 June 08:00-16:00 |
07:56 Taken over from AM
Current White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc51a : 28
aida08 fault 0xf0e9 : 0xf132 : 73
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
FPGA Timestamp error counter test result: Passed 6, Failed 2
Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link
Statistics ok - attachment 1
FEE temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
Analysis of file R3_93 - attachment 4
Still deadtime in aida02 of ~15%
Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
Currently 2.2 TB of space on /media/SecondDrive
Current data rate of 6.23MB/s
At current data rate this should last 4 days
08:27 Currently no beam
Problem with Alvarez 3 now
Tomorrow will have a 1 hour break due to changing of the source
09:36 Beam back
09:59 System wide checks
Base Current Difference
aida07 fault 0xc4fe : 0xc51b : 29
aida08 fault 0xf0e9 : 0xf138 : 79
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
Stats ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
11:14 FRS group is changing from Hg setting back to Pt
11:36 Back on Pt R3_129
14:14 Stats ok - attachment 8
Temperature ok - attachment 9
Bias and leakage currents ok - attachment 10
System wide checks:
WR:
Base Current Difference
aida07 fault 0xc4fe : 0xc523 : 37
aida08 fault 0xf0e9 : 0xf156 : 109
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x11 : 0x1b : 10
aida08 fault 0x1a : 0x20 : 6
Analysis of R3_156 - attachment 11 |
| Attachment 1: 220623_0758_Stats.png
|
|
| Attachment 2: 220623_0759_Temp.png
|
|
| Attachment 3: 220623_0759_Bias.png
|
|
| Attachment 4: R3_95_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260243782 ( 817307.6 Hz)
Other data format: 1676218 ( 5264.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1230 ( 3.9 Hz)
RESUME: 1231 ( 3.9 Hz)
SYNC100: 33186 ( 104.2 Hz)
WR48-63: 33186 ( 104.2 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1607385 ( 5048.1 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 440709 ( 1384.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 318.416 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.258 0.000
1 47.390 0.000
2 1.040 0.000
3 6.094 0.000
4 0.003 59.113
5 0.094 0.000
6 0.304 0.000
7 0.013 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 20778195 27751 0 0 46 46 2749 2749 0 22161 0 72140
1 62611625 17407 0 0 705 706 7998 7998 0 0 0 66280
2 31916579 410233 0 0 83 83 4122 4122 0 401823 0 94767
3 52868439 408659 0 0 312 312 6874 6874 0 394287 0 28791
4 5535248 1334 0 0 1 1 666 666 0 0 0 39988
5 19704728 4912 0 0 25 25 2431 2431 0 0 0 52543
6 35689179 392725 0 0 47 47 4446 4446 0 383739 0 67737
7 31139789 413197 0 0 11 11 3900 3900 0 405375 0 18463
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.666s ( 923.094 blocks/s, 57.693 Mb/s)
|
| Attachment 5: 220623_0958_Stats.png
|
|
| Attachment 6: 220623_0959_Temp.png
|
|
| Attachment 7: 220623_1000_Bias.png
|
|
| Attachment 8: 220623_1413_Stats.png
|
|
| Attachment 9: 220623_1413_Temp.png
|
|
| Attachment 10: 220623_1414_bias.png
|
|
| Attachment 11: R3_156_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260453729 ( 852518.1 Hz)
Other data format: 1466271 ( 4799.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1362 ( 4.5 Hz)
RESUME: 1362 ( 4.5 Hz)
SYNC100: 33138 ( 108.5 Hz)
WR48-63: 33138 ( 108.5 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1397271 ( 4573.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 435545 ( 1425.6 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 305.511 s
FEE elapsed dead time(s) elapsed idle time(s)
0 11.365 0.000
1 54.806 0.000
2 0.562 0.000
3 5.946 0.000
4 0.000 35.352
5 0.068 0.000
6 0.334 0.000
7 0.054 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 30150644 28922 0 0 254 254 3938 3938 0 20538 0 108754
1 64977250 18108 0 0 757 757 8297 8297 0 0 0 62895
2 28023912 358551 0 0 83 83 3523 3523 0 351339 0 69361
3 44222269 352853 0 0 180 180 5536 5536 0 341421 0 37017
4 5005333 1272 0 0 0 0 636 636 0 0 0 36369
5 19466878 5006 0 0 20 20 2483 2483 0 0 0 42187
6 36422600 339861 0 0 37 37 4700 4700 0 330387 0 59710
7 32184843 361698 0 0 31 31 4025 4025 0 353586 0 19252
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 35.418s ( 903.496 blocks/s, 56.469 Mb/s)
|
|
|
496
|
Fri Jun 24 06:56:06 2022 |
OH | Friday 24 June 008:00-16:00 |
07:56 Took over from AM. No issues reported overnight
Stats ok - attachment 1
Temperature ok - attachment 2
Bias and leakage current ok - attachment 3
System wide checks:
Clocks ok
WR
Base Current Difference
aida07 fault 0xc4fe : 0xc53b : 61
aida08 fault 0xf0e9 : 0xf1b7 : 206
White Rabbit error counter test result: Passed 6, Failed 2
FPGA Errors
Base Current Difference
aida07 fault 0x11 : 0x29 : 24
aida08 fault 0x1a : 0x2c : 18
FPGA Timestamp error counter test result: Passed 6, Failed 2
Analysis of R3_359 - attachment 4
Dead time still around 15.9%
Merger item rate around 2E6-4E6
Tape server rate at 7750 kB/s
Current HDD free space 1.7 TB
Time left in HDD 2.59 days
Will run out of space at some point on Sunday
Have started compression of uncompressed raw data on the HDD. Using nice +10
08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
Am regularly checking the statistics
Was able to stop the DAQ by relaunching the merger.
Reset the FEEs and recovered without a powercylce
Started R4 following this stop
08:58 They have taken the beam to change the ion source for maybe 2 hours
Will stop writing data but continue forwarding to MBS
No storage ticked. Following break will be on R5
11:10 Beam is starting to come back R5 started
Stats ok - attachment 6
Temps ok - attachment 7
Bias and leakage currents ok - attachment 8
13:05 Statistics ok - attachment 9
Temps ok - attachment 10
Bias and leakage curents ok - attachment 11
System wide checks:
WR
Base Current Difference
aida07 fault 0xc53d : 0xc540 : 3
aida08 fault 0xf1be : 0xf1d0 : 18
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_33 - attachment 12
15:28 Statistics ok - attachment 13
Temperature ok - attachment 14
Bias and leakage current ok - attachment 15
System wide checks:
Base Current Difference
aida07 fault 0xc53d : 0xc546 : 9
aida08 fault 0xf1be : 0xf1e0 : 34
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_63 - attachment 16 |
| Attachment 1: 220624_0753_stats.png
|
|
| Attachment 2: 220624_0754_Temp.png
|
|
| Attachment 3: 220624_0755_Bias.png
|
|
| Attachment 4: R3_359_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260715255 ( 897318.9 Hz)
Other data format: 1204745 ( 4146.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1323 ( 4.6 Hz)
RESUME: 1323 ( 4.6 Hz)
SYNC100: 33082 ( 113.9 Hz)
WR48-63: 33082 ( 113.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1135935 ( 3909.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 353220 ( 1215.7 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 290.549 s
FEE elapsed dead time(s) elapsed idle time(s)
0 10.171 0.000
1 46.409 0.000
2 1.006 0.000
3 4.601 0.000
4 0.000 37.574
5 0.031 0.000
6 0.191 0.000
7 0.254 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 30903987 28078 0 0 236 236 4023 4023 0 19560 0 86851
1 61208192 17252 0 0 718 718 7908 7908 0 0 0 52897
2 26785987 291397 0 0 72 72 3425 3425 0 284403 0 56573
3 43119726 289440 0 0 153 153 5319 5319 0 278496 0 29859
4 4604412 1194 0 0 0 0 597 597 0 0 0 28791
5 18336996 4656 0 0 16 16 2312 2312 0 0 0 34152
6 36059875 277258 0 0 42 42 4514 4514 0 268146 0 48935
7 39696080 295470 0 0 86 86 4984 4984 0 285330 0 15162
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.877s ( 917.511 blocks/s, 57.344 Mb/s)
|
| Attachment 5: 220624_0842_Stats_Stop.png
|
|
| Attachment 6: 220624_1111_Stats.png
|
|
| Attachment 7: 220624_1111_Temp.png
|
|
| Attachment 8: 220624_1112_Bias.png
|
|
| Attachment 9: 220624_1303_stats.png
|
|
| Attachment 10: 220624_1303_Temp.png
|
|
| Attachment 11: 220624_1304_Bias.png
|
|
| Attachment 12: R5_33_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260717099 ( 869523.1 Hz)
Other data format: 1202901 ( 4011.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1344 ( 4.5 Hz)
RESUME: 1344 ( 4.5 Hz)
SYNC100: 33117 ( 110.4 Hz)
WR48-63: 33117 ( 110.4 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1133979 ( 3782.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 354749 ( 1183.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 299.839 s
FEE elapsed dead time(s) elapsed idle time(s)
0 11.504 0.000
1 42.102 0.000
2 0.275 0.000
3 6.677 0.000
4 0.000 31.976
5 0.051 0.000
6 0.250 0.000
7 0.412 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 31758241 28867 0 0 238 238 4156 4156 0 20079 0 87480
1 58453105 16104 0 0 635 635 7417 7417 0 0 0 53128
2 20104679 289945 0 0 41 41 2565 2565 0 284733 0 56849
3 41709424 287290 0 0 188 188 5262 5262 0 276390 0 29572
4 5133297 1264 0 0 0 0 632 632 0 0 0 29278
5 17857346 4470 0 0 22 22 2213 2213 0 0 0 34539
6 40732502 278221 0 0 59 59 5175 5175 0 267753 0 48740
7 44968505 296740 0 0 161 161 5697 5697 0 285024 0 15163
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.742s ( 921.070 blocks/s, 57.567 Mb/s)
|
| Attachment 13: 220624_1538_Stats.png
|
|
| Attachment 14: 220624_1538_Temp.png
|
|
| Attachment 15: 220624_1538_Bias.png
|
|
| Attachment 16: R5_63_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260796374 ( 834020.0 Hz)
Other data format: 1123626 ( 3593.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1191 ( 3.8 Hz)
RESUME: 1191 ( 3.8 Hz)
SYNC100: 33165 ( 106.1 Hz)
WR48-63: 33165 ( 106.1 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1054914 ( 3373.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 350507 ( 1120.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 312.698 s
FEE elapsed dead time(s) elapsed idle time(s)
0 7.889 0.000
1 39.498 0.000
2 0.306 0.000
3 5.270 0.000
4 0.000 26.707
5 0.036 0.000
6 0.217 0.000
7 0.175 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 30903831 29785 0 0 188 188 4074 4074 0 21261 0 87155
1 61666403 16932 0 0 646 646 7820 7820 0 0 0 55280
2 20140383 267008 0 0 42 42 2575 2575 0 261774 0 55290
3 41652352 266495 0 0 164 164 5306 5306 0 255555 0 28879
4 4871487 1204 0 0 0 0 602 602 0 0 0 28340
5 17605547 4430 0 0 11 11 2204 2204 0 0 0 33538
6 41056242 264471 0 0 55 55 5126 5126 0 254109 0 47278
7 42900129 273301 0 0 85 85 5458 5458 0 262215 0 14747
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 41.305s ( 774.730 blocks/s, 48.421 Mb/s)
|
|
|
504
|
Mon Jun 27 06:45:22 2022 |
OH | Monday 27th June 08:00-16:00 |
07:45 Spoke to David and the beam has been gone since about 05:30
Reason for the loss of beam is a vacuum issue before the FRS
They are waiting for the experts
08:31 Statistics ok - attachment 1
Temperature ok - attachment 2
Bias and leakage currrents ok - attachment 3
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
Currently on file R5_771
Analysis of file R5_770 (No beam) - attachment 4
Around 0.25% deadtime on AIDA02 rest even less
09:02 Current free HDD space 965 GB
Current tape server rate 4979 kB/s
Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours
11:41 Statistics ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
FPGA Timestamp error counter test result: Passed 7, Failed 1
Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently? |
| Attachment 1: 220626_0830_Stats.png
|
|
| Attachment 2: 220626_0830_Temp.png
|
|
| Attachment 3: 220626_0831_Bias.png
|
|
| Attachment 4: R5_770_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 261823568 ( 639881.5 Hz)
Other data format: 96432 ( 235.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 451 ( 1.1 Hz)
RESUME: 451 ( 1.1 Hz)
SYNC100: 33524 ( 81.9 Hz)
WR48-63: 33524 ( 81.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 28482 ( 69.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 0 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 409.175 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.033 0.000
1 1.307 0.000
2 0.000 0.000
3 0.206 0.000
4 0.000 157.882
5 0.000 0.000
6 0.000 0.000
7 0.139 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 20319054 33950 0 0 1 1 2733 2733 0 28482 0 0
1 68104715 18442 0 0 318 318 8903 8903 0 0 0 0
2 24150534 6320 0 0 0 0 3160 3160 0 0 0 0
3 55928626 14410 0 0 131 131 7074 7074 0 0 0 0
4 2503858 618 0 0 0 0 309 309 0 0 0 0
5 12980974 3340 0 0 0 0 1670 1670 0 0 0 0
6 41271038 10344 0 0 0 0 5172 5172 0 0 0 0
7 36564769 9008 0 0 1 1 4503 4503 0 0 0 0
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 41.158s ( 777.488 blocks/s, 48.593 Mb/s)
|
|
|
663
|
Fri Jun 14 15:00:47 2024 |
Norah , Muneerah | 16:00-20:00 Friday 14 June |
Everything appears to be running smoothly. Attachments 1-4 were taken around 16:00
DSSSD Bias & Leakage Current - Check: OK (see attachment 1)
FEE64 Temperature - Check: OK (see attachment 2)
ADC Data Item Stats and Spectrum Browser - Check: OK (see attachments 3 and 4)
Another set of screenshots was taken around 18:00, and everything looks good.
DSSSD bias & leakage current ok - attachment # 5-6
FEE64 temperatures ok - attachment # 7
ADC data item stats - attachment # 8
Merger ok - Attachement #9
Tape service - attachement # 10
ucesb - attachment #11
At 19:17, AIDA03 temperatures gave no response (see attachment 12), but after two minutes, they returned to normal.
I noted that the bplse in ucesb has no data (see attachment 13). It may need to change the disc thresholds again
it does not affect AIDA
At 19:30 most aida temperature are high and I spoke to people on site via zoom one of them she is working with AIDA so she was able to help. She called Nick and it is found that there is a problem in the cooling system so she stopped AIDA
At 22 the cooling system fixed and Nic and the other person on site restarted AIDA
|
| Attachment 1: Screenshot_from_2024-06-14_16-06-09.png
|
|
| Attachment 2: Screenshot_from_2024-06-14_16-04-58.png
|
|
| Attachment 3: Screenshot_from_2024-06-14_16-04-23.png
|
|
| Attachment 4: Screenshot_from_2024-06-14_16-05-33.png
|
|
| Attachment 5: Screenshot_from_2024-06-14_18-02-36.png
|
|
| Attachment 6: Screenshot_from_2024-06-14_18-02-08.png
|
|
| Attachment 7: Screenshot_from_2024-06-14_18-04-40.png
|
|
| Attachment 8: Screenshot_from_2024-06-14_18-05-44.png
|
|
| Attachment 9: Screenshot_from_2024-06-14_18-08-50.png
|
|
| Attachment 10: Screenshot_from_2024-06-14_18-11-04.png
|
|
| Attachment 11: Screenshot_from_2024-06-14_18-17-15.png
|
|
| Attachment 12: Screenshot_from_2024-06-14_19-23-32.png
|
|
| Attachment 13: Screenshot_from_2024-06-14_19-35-11.png
|
|
|
|
608
|
Fri Apr 26 07:01:26 2024 |
Norah | 08:00 - 16:00 Friday 26 April |
10:00 Checks:
All seems smooth.
DSSD bias and leakage current ( Attachment 1)
FEE64 temperatures (Attachment 2)
Per FEE64 Rate spectra (Attachment 3)
ADC data item stats -(Attachment 4)
Ucesb (Attachment 5)
10:21
Aida05 was zero not reading had to call Nic to fix it.
10:34
Now, Aida05 is running well.
.......................................................
12:00 Checks:
Everything appears to be going smoothly, and I took screenshot of them.
12:28, Aida02 showed as not reading , had to email Nic to fix it.
12:30, it resumed reading and returned to normal operation.
.........................................................
14:00 Checks:
Everything is going well , and I have captured screenshots of each one.
......................................................
16:00 Checks:
All seems smooth, and I took screenshot of them.
|
| Attachment 1: Screenshot_from_2024-04-26_10-00-39.png
|
|
| Attachment 2: Screenshot_from_2024-04-26_10-02-35.png
|
|
| Attachment 3: Screenshot_from_2024-04-26_10-04-40.png
|
|
| Attachment 4: Screenshot_from_2024-04-26_10-06-18.png
|
|
| Attachment 5: Screenshot_from_2024-04-26_10-09-52.png
|
|
|
|
613
|
Sat Apr 27 15:13:06 2024 |
Norah | 16:00 - 00:00 Saturday 27 April |
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
| Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
|
|
| Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
|
|
| Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
|
|
| Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
|
|
|
|
81
|
Fri Nov 1 14:17:15 2019 |
Nic, Patrick | Reply: WR Timestamps |
> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> >
> > HDMI cables in aida09 checked and good.
>
> The problem would be the cable , one end or the other.
> I think ( if I recall ) a setup would restart the WR decoder.
>
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
Cable will be replaced once a spare is available.
Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
WR rate was chosen by Vic I believe, I am unsure of reasoning myself. |
|
|
90
|
Thu Nov 7 10:22:26 2019 |
Nic, Patrick | Reply: WR Timestamps |
> > > All 12 FEEs have valid WR Timestamps
> > > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > >
> > > HDMI cables in aida09 checked and good.
> >
> > The problem would be the cable , one end or the other.
> > I think ( if I recall ) a setup would restart the WR decoder.
> >
> > I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
>
> Cable will be replaced once a spare is available.
>
> Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
>
> WR rate was chosen by Vic I believe, I am unsure of reasoning myself.
An update/clarification:
Although the upper bits are zero I believe actually it is a total failure to synchronise to WR:
aida09
WR Time Item 0x80500000 0x0fbd8000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80400249 0x0fbd8000; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80500000 0x0fbdc000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80400249 0x0fbdc000; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80500000 0x0fbe0000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
WR Time Item 0x80400249 0x0fbe0000; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
aida10
WR Time Item 0x8050022e 0x0bde8000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x804e29d5 0x0bde8000; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x8050022e 0x0bdec000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x804e29d5 0x0bdec000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x8050022e 0x0bdf0000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
WR Time Item 0x804e29d5 0x0bdf0000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000 |
|
|
36
|
Tue Mar 26 14:58:09 2019 |
NH, VP | How to reset merger |
If the merger doesn't show "xfer Links => Merger" at the top, it is probably the merger setting has been corrupted somehow. This is how to reset it
If you go to the web page from which you started the Merge and TapeServer
(probably localhost:8115)
Then select the item NetVar Service which is at the left hand end.
Click on Inquire Registers
then enter NetVar.MERGE.RunOptions into the "Register Name" entry and when finished move the mouse out of the entry box.
NetVar.MERGE.RunOptions will appear in the center of the top row. (it is an item in the menu but not so easy to find)
Click on Read and I expect you will get 0
Type 1 into the data entry box and click on Write
Now return to the Merger and click on Reload
The Merger State will now end as xfer Links => Merger
The button Output: will now toggle between this and xfer links => Merger => Storage. |
|
|
534
|
Tue Mar 12 16:16:49 2024 |
NH, TD, JB, HA | Summary of DSSSD Biasing 12.03 |
DSSD#1 undergoes a breakdown at 90V, two of the three wafers show this
- The adapter PCBs themselves have no breakdown at 100V, indicating the issue is internal to the snout
If we are lucky it may be a loose or misaligned kapton connector inside the snout. If not we will have to remove DSSD#2 and inspect DSSD#1 for damage/lint/etc. If we see nothing it should be replaced
DSSD#2 biases perfectly fine, but the leakage current is unstable with biasing -ve to p+n and gnd (return) to n+n. Leakage current is stable biasing +ve to n+n and gnd to p+n. (https://elog.ph.ed.ac.uk/DESPEC/240311_093933/Downstream_positive_bias_vs._Current_(uA).png very nice curve)
We see the same fluctuations just daisy chaining 3 p+n PCBs togerher with the bias (no DSSSD or gnd links). The fluctuations were reduced by connecting all 8 adapter boards of the DSSD together.
When biasing just *one* PCB in this way, the current is stable
The fluctuations happened changing the HV channel, adapter board PCB and LEMO cables: it doesn't seem to be a defect with a specific thing.
The test today confirmed the HV cables (the only ones used) were correctly isolated from everything (OL = "infinite" resistance to ground)
We saw unstable current indications using the Mesytec MHV-4 to apply voltage to the PCB as well.
The behaviour is odd but doesn't seem to be related to the DSSD, which I suspect is OK. One idea is to try connecting the 3 adapter boards to FEEs and repeating the test, as this introduces another (substantial) path to gnd. Maybe this eliminates the current instability? It's about the only difference left from December.
-
For the next steps I believe the snout must be dismounted to inspect DSSD#1. It would be best to coordinate this with replacing the broken(?) bPlas. If we are lucky we may not have to remove the DSSDs but it is likely we have to remove both DSSDs to swap out #1
After replacing the DSSD(s) we should cover the snout with a clean black bin bag and bias it on the MH table to confirm both detectors work. This saves the effort of carrying it to S4 just to find another problem.
If both detectors make it to 120V we can mount it again more confidently
Once we have two biased detectors we can rearrange some FEEs to get 8 for DSSD#1 to do the noise tests. I suggest we wire them up for the numbering plan in https://elog.ph.ed.ac.uk/DESPEC/532 but in the DHCP renumber 9,10,15,16 to 5,6,7,8 temporarily to allow data to be sent to MBS for the dry run (merger limitation)
When the remaining FEEs are recovered from UK+CRYRING we can instrument DSSD#2 and renumber the FEEs to match the cables |
|
|
119
|
Fri Dec 20 12:04:03 2019 |
NH, TD | Friday 20 December 2019 |
Fig 1: DSSD Leakage currents ok
Fig 2: FEE Temps ok
Fig 3: aida08 has a few timestamp errors in merger
Fig 4, 5: GSI WR OK
Fig 6: All System wide checks passed
FIg 7: Stats good
13:11 - Alpha Run Stopped, Final File: R9_6
FEEs powered off, DSSD bias off. |
| Attachment 1: Screenshot-CAEN_HV_Control.png
|
|
| Attachment 2: FeeTemps.png
|
|
| Attachment 3: mergerTSerrors.png
|
|
| Attachment 4: gsiwr1.png
|
|
| Attachment 5: gsiwr2.png
|
|
| Attachment 6: checks.png
|
|
| Attachment 7: stats.png
|
|
|
|
277
|
Sat Apr 24 13:25:23 2021 |
NH, TD | AIDA FEE64 crash log for S460 |
https://docs.google.com/spreadsheets/d/1CyL5L-WLsh9fsS7CN989GC3ktqiPQOkG8YIDJrrGxYA/edit#gid=0 |
|
|
375
|
Tue Aug 17 12:39:25 2021 |
NH, TD | AIDA Noise |
Investigate noise situation of AIDA (no DSSSDs connected)
Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
- aida16 very noisy ASICS 1&2, normal ASICS 3&4
- back-side aidas (02,04,06,08) worse resolution than front-side
- high-frequency noise still noticeable in waveforms
Figs 7-8: Remove pulser in (from NIM rack) only
- no noticable change
Figs 9-10: Remove all pulser cables from adapter boards
- looks improved
Figs 11-12: HV Braid->GND Jumpers removed
Fig 13-14: All HV cables removed
- aida06, aida14 isn't updating
- very noticable improvement
Fig 15-16: Intermediate HV cables re-added
- signal worse in FEE64s with HV cables
- aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
Pulser widths now:
- aida14 (-ve amplitude) = 13.5 channels
- aida08 (+ve amplitude) = 40 channels (?) |
| Attachment 1: Screenshot-Waves1.png
|
|
| Attachment 2: Screenshot-Waves2.png
|
|
| Attachment 3: Screenshot-Pulsers.png
|
|
| Attachment 4: Screenshot-Width11.png
|
|
| Attachment 5: Screenshot-PulserWidthBack.png
|
|
| Attachment 6: Screenshot-Width02.png
|
|
| Attachment 7: Screenshot-PulserInRemov-Front.png
|
|
| Attachment 8: Screenshot-PulserInRemov-Back.png
|
|
| Attachment 9: PulserAllOut-Front.png
|
|
| Attachment 10: PulserAllremov-Back.png
|
|
| Attachment 11: HVJumpers-Front.png
|
|
| Attachment 12: HVJumpers-Back.png
|
|
| Attachment 13: AllHVRemoved-Front.png
|
|
| Attachment 14: HVout-Back.png
|
|
| Attachment 15: HVintermediate-Front.png
|
|
| Attachment 16: HVintermediate-Back.png
|
|
|
|
404
|
Mon Mar 7 12:20:26 2022 |
NH, TD | AIDA Grounding Checks Mon 7 Mar |
Remove all DSSD ribbon cable braid ground from FEE64s to check n+n ground loops
Removing all on n+n side only and test resistance to ground: 4, 6 and 8 show continuity to ground, 2 shows no connection (braid ring terminal to copper bar)
Remove all p+n braids as wll and check:
- See no continuity to ground (good)
- Snout has no continuity to ground (good)
- n+n side fees 4 and 8 to seem to have continuity to snout though (bad)
- p+n checked and don't seem to have continuity to gnd (except when braid was touching a LEMO barrel)
- n+n aida06 maybe has high resistance connection to ground (bad) or (about 0.7 V drop according to RS multimeter)
Bias+Power up and check noise
aida09 Peak width = 86.03+/- 1.721
aida02 Peak width = 509.48+/- 11.1812
Low frequency noise on wavesforms corresponds to ca 100 kHz - switching power supply?
Attachment 6 - ADC data item stats
all p+n/n+n cable braids disconnected
all FEE64 adaptor PCBS grounded to Copper busbar OK
Remove guard ring jumper on upper-middle p+n FEE (rear guard ring of middle DSSSD)
Attachment 7 - ADC data item states
No real change to rates
Remove field plate jumpers on all p+n FEEs
Attachment 8 - ADC data item states
No real change to rates
Remove bias LK1 jumpers on n+n FEES:
Generally worse across the p+n side, no real change to n+n side
Attachment 9 - Stats with LK1 removed:
aida09 Peak Width = 181.77. p+n see Attachment 10
aida02 Peak width = 505.23. n+n see Attachment 11
Attachment 12 - p+n waveforms
Attachment 13 - n+n waveforms
High frequency component reduced but 100 kHz low frequency component stronger
Remove LK3 from aida03,aida07 (no jumpers at all on any FEE) - DSSSDs floating
No significant change, see attachment 14 stats |
| Attachment 1: 2022-03-07_14-00_Stats.png
|
|
| Attachment 2: 2022-03-07_14-03-03_pn_widths.png
|
|
| Attachment 3: 2022-03-07_14-04-13_nn_widths.png
|
|
| Attachment 4: 022-03-07_14-05-12_pn_waves.png
|
|
| Attachment 5: 2022-03-07_14-05-49_nn_waves.png
|
|
| Attachment 6: Screenshot_from_2022-03-07_15-28-41.png
|
|
| Attachment 7: 2022-03-07_15-48-02_stats_no_middle_rear_GR.png
|
|
| Attachment 8: 2022-03-07_16-08-11_stats_no_fieldplane_ground.png
|
|
| Attachment 9: 2022-03-07_16-22-00_stats_no_LK1.png
|
|
| Attachment 10: 2022-03-07_16-23-50_pn_widths_no_lk1.png
|
|
| Attachment 11: 2022-03-07_16-24-55_nn_widths_no_lk1.png
|
|
| Attachment 12: 2022-03-07_16-25-50_pn_waves_no_lk1.png
|
|
| Attachment 13: 2022-03-07_16-27-09_nn_waves_no_lk1.png
|
|
| Attachment 14: 2022-03-07_17-12-34_no_detector_gnd__stats.png
|
|
|
|
474
|
Tue May 31 09:05:00 2022 |
NH, TD | Tuesday 31 May |
DSSD stack will be as follows
1 3208-7 1019 125 upstream
2 3208-13 1020 120 downstream
First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance
3208-7:
Top and bottom: 22 Ohm
Left and right: 22 Ohm
3208-13:
Top and bottom: 22 Ohm
Left and right: 22 Ohm
Conclusion: All bias bond wires are intact on both DSSDs
Visual inspection is both DSSDs look good with no damage or bond wire issues
Installed in snout: full stack
3 mm bPlas 1
17 mm Air
1 mm AIDA 1 (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2 |
|
|
56
|
Mon Jul 8 16:02:52 2019 |
NH, PJCS | Aida 28/6/2019 - WATER OFF |
It occurs to to wonder if there is now water inside the modules. If the metal of the cooling plates is damp on the outside then it is safe to assume there is water on the inside too . Can I suggest that one of the modules is dismounted, disassembled and checked?
Patrick
| Quote: |
|
The humidity sensor hasn't reported humdities under 90% yet. In order to try and help dry the pipes and turn AIDA the water has been turned off the weekend.
Sunday temperatures might reach 40 C which means it may struggle anyway.
Will consider trying to get a temperature/humidity monitor for S4 to check if the environment is fine too.
|
|
|
|
127
|
Wed Feb 19 10:58:29 2020 |
NH, PJCS | Report: aida09 Kernel Panic & Lost WR |
> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
>
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
>
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
>
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5
>
> Attach 5: Timestamp shown by merger
Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a
logical "rollover" of the lower 14 bits of this 10nS timestamp.
Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?
It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.
The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.
If this occurs again then the system wide check results would be interesting. |