| ID |
Date |
Author |
Subject |
|
388
|
Thu Oct 21 16:18:51 2021 |
OH | FEE and Adaptor board groundings |
During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
- The adaptor boards have no connection to the FEE ground over the ERNI connector
- Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
- Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
- With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
- The cost of this is the rate of FEE8 goes up to 300k
- It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
|
|
391
|
Fri Oct 29 10:49:01 2021 |
OH | Noise tests - 28th October -29th October |
Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12
This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.
It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.
Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9 60.25
1 64.18
10 55.76
11 64.81
3 56.21
12 48.54nt 5
We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees
Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10
Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13
Pulser was then switched on
Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16
Pulser widths
FEE Width
9 60.25
1 64.18
10 55.76
11 64.81
3 56.21
12 48.54
n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18
FEE Width
9 62.88
1 65.30
10 55.18
11 64.47
3 51.77
12 50.22
2 196.00
4 271.57
29 October
Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22
Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23
Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24 |
| Attachment 1: 20211028-1300-rates-groundedfees.png
|
|
| Attachment 2: 20211028-1300-ratespectra.png
|
|
| Attachment 3: 20211028-1300-ratespectraymax150.png
|
|
| Attachment 4: 20211028-1300-aida11wave.png
|
|
| Attachment 5: 20211028-1300-aida02wave.png
|
|
| Attachment 6: 20211018-1326-rates-psuchange.png
|
|
| Attachment 7: 20211028-1326-ratesspectra-newpsu.png
|
|
| Attachment 8: 20211028-1326-rates-linymax10-newpsu.png
|
|
| Attachment 9: 20211028-1326-wavesaida12-pnpulser-newpsu.png
|
|
| Attachment 10: 20211028-1326-wavesaida02-newpsupng
|
| Attachment 11: 20211028-1410-ratespulserconnoff.png
|
|
| Attachment 12: 20211028-1412-ratespectra-normal.png
|
|
| Attachment 13: 20211028-1412-ratesspectra-ymax10.png
|
|
| Attachment 14: 20211028-1413-ratesymax10-pulserONpng
|
| Attachment 15: 20211028-1414-ratesspectra-pulseron.png
|
|
| Attachment 16: 20211028-1424-pulserpeakspn.png
|
|
| Attachment 17: 20211028_1449_nn_18L.png
|
|
| Attachment 18: 20211028_1451_pn_18l.png
|
|
| Attachment 19: PXL_20211028_121721708.jpg
|
|
| Attachment 20: PXL_20211028_121738665.jpg
|
|
| Attachment 21: PXL_20211028_121728173.jpg
|
|
| Attachment 22: 20211029_1218_stats.png
|
|
| Attachment 23: 20211029-1400-pnlk1234nnlk124.png
|
|
| Attachment 24: 20211029-1436-nolk3bar1and3.png
|
|
|
398
|
Wed Mar 2 09:08:44 2022 |
OH | DAQ Status 2nd March 2022 |
DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
Temperature menu is still responsive and all FEEs report a temperature
Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
Merger shows no data arriving as expected - Attachment 3
Merger data buffers for each FEE - attachment 4
Merger time errors - attachment 5
DAQ has not yet been restarted in case Patrick wanted to look. |
| Attachment 1: 220302_1002_goodevents.png
|
|
| Attachment 2: 220302_1002_white_rabbit.png
|
|
| Attachment 3: 220302_1003_merger.png
|
|
| Attachment 4: 220302_1003_merger_data_buffers.png
|
|
| Attachment 5: 220302_1004_merger_merge_time_errors.png
|
|
|
399
|
Wed Mar 2 10:05:13 2022 |
OH | DAQ Status 2nd March 2022 |
> DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
> Temperature menu is still responsive and all FEEs report a temperature
> Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
> Merger shows no data arriving as expected - Attachment 3
> Merger data buffers for each FEE - attachment 4
> Merger time errors - attachment 5
>
> DAQ has not yet been restarted in case Patrick wanted to look.
Looking at the first attachment..... aida09 has rebooted. So it is not setup at all.
This has caused the NewMerger to hang .... which surprises me .... as I thought it was designed to spot a bad FEE and continue with the rest.
Yes I will have a look today as I'm in Daresbury.
More later ...... |
|
417
|
Mon May 2 09:52:20 2022 |
OH | Shifter checklist 2022 |
Every 30 minutes:
- Reload statistics (Web browser tab screen 1, workspace 3)
- Are all FEEs still showing a rate?
- If one shows 0 after multiple reloads Call Oscar or Tom.
- IGNORE FOR TIME BEING Options database monitor (Top middle terminal screen 2, workspace 3)
- Has the message changed since the last check?
- If so copy the message and place in elog
- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 5)
- If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
- Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 3)
- Ensure "Act on all FEE" and "Act on all asic" selected
- Click on drop down menu in the bottom of the screen and select check ASIC control
- It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
- A pop up will appear check that all options say ok
- Has the implant rate gone to a more normal amount?
- If not feel free to call
Every two hours:
- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 3)
- Compare to the previous screenshot to check if anything has drastically changed
- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 3)
- Do they all appear stable?
- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 3)
- Perform the five system wide checks (Web browser tab screen 1, workspace 3)
- Ignore the system wide checks for ADC calibration (ADCs are turned off so they will all fail)
- Check memory information also fails.
- Make a note in the elog of which tests are successful
- If failed copy the text from the white box into the elog and make a note of it
- Enter the screenshots into the elog
- Update the leakage current spreadsheet (Web browser, screen 2, workspace 5) |
|
456
|
Sun May 15 07:40:52 2022 |
OH | Sunday 15th May 08:00-16:00 |
08:40 FEE Temperatures ok - attachment 1
FEE statistics - Attachment 2
Bias and leakage currents ok - attachment 3
ASIC check ok
All system wide checks ok - Except aida09 fails clock check with bit 6
11:00 FEE statistics - attachment 4
FEE temperatures all ok - attachment 5
BIAS and leakage currents - attachment 6
ASIC check ok
All system wide checks as before
Currently on file R4_1030
13.1 MB/s to disk
4.8 million items per second
13:13 FEE statistics - attachment 7
FEE Temperatures all ok - attahcment 8
Bias and leakage currents ok - attachment 9
ASIC check ok
All system wide checks as before
Currently on file R4_1078
13.1 MB/s to disk
4.8 million items per second
15:35 Analysis of R4_1128 - attachment 10 |
| Attachment 1: Screenshot_from_2022-05-15_08-38-04.png
|
|
| Attachment 2: Screenshot_from_2022-05-15_08-38-34.png
|
|
| Attachment 3: Screenshot_from_2022-05-15_08-39-12.png
|
|
| Attachment 4: Screenshot_from_2022-05-15_11-00-32.png
|
|
| Attachment 5: Screenshot_from_2022-05-15_11-00-50.png
|
|
| Attachment 6: Screenshot_from_2022-05-15_11-01-02.png
|
|
| Attachment 7: Screenshot_from_2022-05-15_13-13-50.png
|
|
| Attachment 8: Screenshot_from_2022-05-15_13-14-12.png
|
|
| Attachment 9: Screenshot_from_2022-05-15_13-14-31.png
|
|
| Attachment 10: R4_1128
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 259195371 ( 1637875.8 Hz)
Other data format: 2724629 ( 17217.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 2455 ( 15.5 Hz)
RESUME: 2456 ( 15.5 Hz)
SYNC100: 32590 ( 205.9 Hz)
WR48-63: 32590 ( 205.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 2654538 ( 16774.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 38026 ( 240.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 158.251 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 4.735 0.000
2 0.000 0.000
3 16.674 0.000
4 0.000 0.000
5 77.034 0.000
6 0.000 0.000
7 33.677 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 12.872
11 17.247 0.000
12 0.000 0.000
13 0.000 8.200
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 7464928 2802 0 0 0 0 927 927 0 948 0 661
1 38695355 10162 0 0 254 254 4827 4827 0 0 0 32843
2 6890959 1397581 0 0 0 0 1028 1028 0 1395525 0 633
3 41303989 1261446 0 0 398 398 5227 5227 0 1250196 0 331
4 4066993 1070 0 0 0 0 535 535 0 0 0 242
5 37291979 10883 0 0 737 738 4704 4704 0 0 0 222
6 12390385 7719 0 0 0 0 1626 1626 0 4467 0 394
7 42901955 15088 0 0 626 626 5217 5217 0 3402 0 167
8 10872099 2752 0 0 0 0 1376 1376 0 0 0 592
9 4061050 1014 0 0 0 0 507 507 0 0 0 432
10 2113468 548 0 0 0 0 274 274 0 0 0 291
11 41561869 11172 0 0 440 440 5146 5146 0 0 0 468
12 6798748 1722 0 0 0 0 861 861 0 0 0 436
13 2781594 670 0 0 0 0 335 335 0 0 0 314
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 33.348s ( 959.588 blocks/s, 59.974 Mb/s)
|
|
464
|
Tue May 17 06:58:51 2022 |
OH | Tuesday 17th May |
08:00 Still have beam for the time being.
Currently on file R4_2049
FEE Stats - attachment 1
FEE Temps ok - attachment 2
Bias and leakage currents - attachment 3
System wide checks all ok *except aida09 fails clock check with value 6 which is ok*
Plan for the morning is once beam is lost is Ge group will enter and place source on the snout to perform efficiency measurement.
Once they have left the area and are taking data a pulser walkthrough will be performed.
Patrick can then have access to the system
08:16 Beam taken away
Data taken stopped on file R4_2055
Will perform a pulser walkthrough now
Pulser walkthrough started on R5
1V->0.1V in 0.1V steps
Pulser setting - attachment 4
p+n pulser peaks - attachment 5
n+n pulser "peaks" - attachment 6 |
| Attachment 1: Screenshot_from_2022-05-17_08-00-50.png
|
|
| Attachment 2: Screenshot_from_2022-05-17_08-01-13.png
|
|
| Attachment 3: Screenshot_from_2022-05-17_08-01-25.png
|
|
| Attachment 4: Screenshot_from_2022-05-17_08-49-02.png
|
|
| Attachment 5: Screenshot_from_2022-05-17_08-51-19.png
|
|
| Attachment 6: Screenshot_from_2022-05-17_08-51-52.png
|
|
|
484
|
Sun Jun 19 12:36:10 2022 |
OH | Sunday 19th June |
Yesterday the sum/inverter was found to be a considerable source of noise in the system.
Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace
Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow
Put inverter back in CAEN crate in S4 and confirm similar behavior.
Move CAEN crate to new mains port, same behaviour
Move CAEN crate to AC voltage stabiliser, same behaviour
Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears
Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.
MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd
CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?
MACBs and inverter checked on another crate - no signal
Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected
Tested adding bias filters - No change
15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
We get ok stats even with waveforms on at 0xa- attachment 5
Waveforms still have the 100kHz oscillation in them though attachments 6 and 7
Without waveforms we get similar performance to before the switch to the new mains - attachment 8
15:56 Move CAEN NIM crate across to this mains network
300+kHz on all FEEs with waveforms on. Can recover waveform off performance though
16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.
17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9 |
| Attachment 1: 1.jpg
|
|
| Attachment 2: 2.jpg
|
|
| Attachment 3: 3.jpg
|
|
| Attachment 4: 4.jpg
|
|
| Attachment 5: 220618_1753_stats_no_waveform.png
|
|
|
493
|
Thu Jun 23 06:56:02 2022 |
OH | Thursday 23 June 08:00-16:00 |
07:56 Taken over from AM
Current White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc51a : 28
aida08 fault 0xf0e9 : 0xf132 : 73
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
FPGA Timestamp error counter test result: Passed 6, Failed 2
Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link
Statistics ok - attachment 1
FEE temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
Analysis of file R3_93 - attachment 4
Still deadtime in aida02 of ~15%
Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
Currently 2.2 TB of space on /media/SecondDrive
Current data rate of 6.23MB/s
At current data rate this should last 4 days
08:27 Currently no beam
Problem with Alvarez 3 now
Tomorrow will have a 1 hour break due to changing of the source
09:36 Beam back
09:59 System wide checks
Base Current Difference
aida07 fault 0xc4fe : 0xc51b : 29
aida08 fault 0xf0e9 : 0xf138 : 79
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
Stats ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
11:14 FRS group is changing from Hg setting back to Pt
11:36 Back on Pt R3_129
14:14 Stats ok - attachment 8
Temperature ok - attachment 9
Bias and leakage currents ok - attachment 10
System wide checks:
WR:
Base Current Difference
aida07 fault 0xc4fe : 0xc523 : 37
aida08 fault 0xf0e9 : 0xf156 : 109
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x11 : 0x1b : 10
aida08 fault 0x1a : 0x20 : 6
Analysis of R3_156 - attachment 11 |
| Attachment 1: 220623_0758_Stats.png
|
|
| Attachment 2: 220623_0759_Temp.png
|
|
| Attachment 3: 220623_0759_Bias.png
|
|
| Attachment 4: R3_95_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260243782 ( 817307.6 Hz)
Other data format: 1676218 ( 5264.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1230 ( 3.9 Hz)
RESUME: 1231 ( 3.9 Hz)
SYNC100: 33186 ( 104.2 Hz)
WR48-63: 33186 ( 104.2 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1607385 ( 5048.1 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 440709 ( 1384.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 318.416 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.258 0.000
1 47.390 0.000
2 1.040 0.000
3 6.094 0.000
4 0.003 59.113
5 0.094 0.000
6 0.304 0.000
7 0.013 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 20778195 27751 0 0 46 46 2749 2749 0 22161 0 72140
1 62611625 17407 0 0 705 706 7998 7998 0 0 0 66280
2 31916579 410233 0 0 83 83 4122 4122 0 401823 0 94767
3 52868439 408659 0 0 312 312 6874 6874 0 394287 0 28791
4 5535248 1334 0 0 1 1 666 666 0 0 0 39988
5 19704728 4912 0 0 25 25 2431 2431 0 0 0 52543
6 35689179 392725 0 0 47 47 4446 4446 0 383739 0 67737
7 31139789 413197 0 0 11 11 3900 3900 0 405375 0 18463
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.666s ( 923.094 blocks/s, 57.693 Mb/s)
|
| Attachment 5: 220623_0958_Stats.png
|
|
| Attachment 6: 220623_0959_Temp.png
|
|
| Attachment 7: 220623_1000_Bias.png
|
|
| Attachment 8: 220623_1413_Stats.png
|
|
| Attachment 9: 220623_1413_Temp.png
|
|
| Attachment 10: 220623_1414_bias.png
|
|
| Attachment 11: R3_156_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260453729 ( 852518.1 Hz)
Other data format: 1466271 ( 4799.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1362 ( 4.5 Hz)
RESUME: 1362 ( 4.5 Hz)
SYNC100: 33138 ( 108.5 Hz)
WR48-63: 33138 ( 108.5 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1397271 ( 4573.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 435545 ( 1425.6 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 305.511 s
FEE elapsed dead time(s) elapsed idle time(s)
0 11.365 0.000
1 54.806 0.000
2 0.562 0.000
3 5.946 0.000
4 0.000 35.352
5 0.068 0.000
6 0.334 0.000
7 0.054 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 30150644 28922 0 0 254 254 3938 3938 0 20538 0 108754
1 64977250 18108 0 0 757 757 8297 8297 0 0 0 62895
2 28023912 358551 0 0 83 83 3523 3523 0 351339 0 69361
3 44222269 352853 0 0 180 180 5536 5536 0 341421 0 37017
4 5005333 1272 0 0 0 0 636 636 0 0 0 36369
5 19466878 5006 0 0 20 20 2483 2483 0 0 0 42187
6 36422600 339861 0 0 37 37 4700 4700 0 330387 0 59710
7 32184843 361698 0 0 31 31 4025 4025 0 353586 0 19252
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 35.418s ( 903.496 blocks/s, 56.469 Mb/s)
|
|
496
|
Fri Jun 24 06:56:06 2022 |
OH | Friday 24 June 008:00-16:00 |
07:56 Took over from AM. No issues reported overnight
Stats ok - attachment 1
Temperature ok - attachment 2
Bias and leakage current ok - attachment 3
System wide checks:
Clocks ok
WR
Base Current Difference
aida07 fault 0xc4fe : 0xc53b : 61
aida08 fault 0xf0e9 : 0xf1b7 : 206
White Rabbit error counter test result: Passed 6, Failed 2
FPGA Errors
Base Current Difference
aida07 fault 0x11 : 0x29 : 24
aida08 fault 0x1a : 0x2c : 18
FPGA Timestamp error counter test result: Passed 6, Failed 2
Analysis of R3_359 - attachment 4
Dead time still around 15.9%
Merger item rate around 2E6-4E6
Tape server rate at 7750 kB/s
Current HDD free space 1.7 TB
Time left in HDD 2.59 days
Will run out of space at some point on Sunday
Have started compression of uncompressed raw data on the HDD. Using nice +10
08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
Am regularly checking the statistics
Was able to stop the DAQ by relaunching the merger.
Reset the FEEs and recovered without a powercylce
Started R4 following this stop
08:58 They have taken the beam to change the ion source for maybe 2 hours
Will stop writing data but continue forwarding to MBS
No storage ticked. Following break will be on R5
11:10 Beam is starting to come back R5 started
Stats ok - attachment 6
Temps ok - attachment 7
Bias and leakage currents ok - attachment 8
13:05 Statistics ok - attachment 9
Temps ok - attachment 10
Bias and leakage curents ok - attachment 11
System wide checks:
WR
Base Current Difference
aida07 fault 0xc53d : 0xc540 : 3
aida08 fault 0xf1be : 0xf1d0 : 18
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_33 - attachment 12
15:28 Statistics ok - attachment 13
Temperature ok - attachment 14
Bias and leakage current ok - attachment 15
System wide checks:
Base Current Difference
aida07 fault 0xc53d : 0xc546 : 9
aida08 fault 0xf1be : 0xf1e0 : 34
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_63 - attachment 16 |
| Attachment 1: 220624_0753_stats.png
|
|
| Attachment 2: 220624_0754_Temp.png
|
|
| Attachment 3: 220624_0755_Bias.png
|
|
| Attachment 4: R3_359_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260715255 ( 897318.9 Hz)
Other data format: 1204745 ( 4146.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1323 ( 4.6 Hz)
RESUME: 1323 ( 4.6 Hz)
SYNC100: 33082 ( 113.9 Hz)
WR48-63: 33082 ( 113.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1135935 ( 3909.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 353220 ( 1215.7 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 290.549 s
FEE elapsed dead time(s) elapsed idle time(s)
0 10.171 0.000
1 46.409 0.000
2 1.006 0.000
3 4.601 0.000
4 0.000 37.574
5 0.031 0.000
6 0.191 0.000
7 0.254 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 30903987 28078 0 0 236 236 4023 4023 0 19560 0 86851
1 61208192 17252 0 0 718 718 7908 7908 0 0 0 52897
2 26785987 291397 0 0 72 72 3425 3425 0 284403 0 56573
3 43119726 289440 0 0 153 153 5319 5319 0 278496 0 29859
4 4604412 1194 0 0 0 0 597 597 0 0 0 28791
5 18336996 4656 0 0 16 16 2312 2312 0 0 0 34152
6 36059875 277258 0 0 42 42 4514 4514 0 268146 0 48935
7 39696080 295470 0 0 86 86 4984 4984 0 285330 0 15162
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.877s ( 917.511 blocks/s, 57.344 Mb/s)
|
| Attachment 5: 220624_0842_Stats_Stop.png
|
|
| Attachment 6: 220624_1111_Stats.png
|
|
| Attachment 7: 220624_1111_Temp.png
|
|
| Attachment 8: 220624_1112_Bias.png
|
|
| Attachment 9: 220624_1303_stats.png
|
|
| Attachment 10: 220624_1303_Temp.png
|
|
| Attachment 11: 220624_1304_Bias.png
|
|
| Attachment 12: R5_33_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260717099 ( 869523.1 Hz)
Other data format: 1202901 ( 4011.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1344 ( 4.5 Hz)
RESUME: 1344 ( 4.5 Hz)
SYNC100: 33117 ( 110.4 Hz)
WR48-63: 33117 ( 110.4 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1133979 ( 3782.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 354749 ( 1183.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 299.839 s
FEE elapsed dead time(s) elapsed idle time(s)
0 11.504 0.000
1 42.102 0.000
2 0.275 0.000
3 6.677 0.000
4 0.000 31.976
5 0.051 0.000
6 0.250 0.000
7 0.412 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 31758241 28867 0 0 238 238 4156 4156 0 20079 0 87480
1 58453105 16104 0 0 635 635 7417 7417 0 0 0 53128
2 20104679 289945 0 0 41 41 2565 2565 0 284733 0 56849
3 41709424 287290 0 0 188 188 5262 5262 0 276390 0 29572
4 5133297 1264 0 0 0 0 632 632 0 0 0 29278
5 17857346 4470 0 0 22 22 2213 2213 0 0 0 34539
6 40732502 278221 0 0 59 59 5175 5175 0 267753 0 48740
7 44968505 296740 0 0 161 161 5697 5697 0 285024 0 15163
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.742s ( 921.070 blocks/s, 57.567 Mb/s)
|
| Attachment 13: 220624_1538_Stats.png
|
|
| Attachment 14: 220624_1538_Temp.png
|
|
| Attachment 15: 220624_1538_Bias.png
|
|
| Attachment 16: R5_63_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260796374 ( 834020.0 Hz)
Other data format: 1123626 ( 3593.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1191 ( 3.8 Hz)
RESUME: 1191 ( 3.8 Hz)
SYNC100: 33165 ( 106.1 Hz)
WR48-63: 33165 ( 106.1 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1054914 ( 3373.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 350507 ( 1120.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 312.698 s
FEE elapsed dead time(s) elapsed idle time(s)
0 7.889 0.000
1 39.498 0.000
2 0.306 0.000
3 5.270 0.000
4 0.000 26.707
5 0.036 0.000
6 0.217 0.000
7 0.175 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 30903831 29785 0 0 188 188 4074 4074 0 21261 0 87155
1 61666403 16932 0 0 646 646 7820 7820 0 0 0 55280
2 20140383 267008 0 0 42 42 2575 2575 0 261774 0 55290
3 41652352 266495 0 0 164 164 5306 5306 0 255555 0 28879
4 4871487 1204 0 0 0 0 602 602 0 0 0 28340
5 17605547 4430 0 0 11 11 2204 2204 0 0 0 33538
6 41056242 264471 0 0 55 55 5126 5126 0 254109 0 47278
7 42900129 273301 0 0 85 85 5458 5458 0 262215 0 14747
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 41.305s ( 774.730 blocks/s, 48.421 Mb/s)
|
|
504
|
Mon Jun 27 06:45:22 2022 |
OH | Monday 27th June 08:00-16:00 |
07:45 Spoke to David and the beam has been gone since about 05:30
Reason for the loss of beam is a vacuum issue before the FRS
They are waiting for the experts
08:31 Statistics ok - attachment 1
Temperature ok - attachment 2
Bias and leakage currrents ok - attachment 3
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
Currently on file R5_771
Analysis of file R5_770 (No beam) - attachment 4
Around 0.25% deadtime on AIDA02 rest even less
09:02 Current free HDD space 965 GB
Current tape server rate 4979 kB/s
Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours
11:41 Statistics ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
FPGA Timestamp error counter test result: Passed 7, Failed 1
Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently? |
| Attachment 1: 220626_0830_Stats.png
|
|
| Attachment 2: 220626_0830_Temp.png
|
|
| Attachment 3: 220626_0831_Bias.png
|
|
| Attachment 4: R5_770_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 261823568 ( 639881.5 Hz)
Other data format: 96432 ( 235.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 451 ( 1.1 Hz)
RESUME: 451 ( 1.1 Hz)
SYNC100: 33524 ( 81.9 Hz)
WR48-63: 33524 ( 81.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 28482 ( 69.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 0 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 409.175 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.033 0.000
1 1.307 0.000
2 0.000 0.000
3 0.206 0.000
4 0.000 157.882
5 0.000 0.000
6 0.000 0.000
7 0.139 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 20319054 33950 0 0 1 1 2733 2733 0 28482 0 0
1 68104715 18442 0 0 318 318 8903 8903 0 0 0 0
2 24150534 6320 0 0 0 0 3160 3160 0 0 0 0
3 55928626 14410 0 0 131 131 7074 7074 0 0 0 0
4 2503858 618 0 0 0 0 309 309 0 0 0 0
5 12980974 3340 0 0 0 0 1670 1670 0 0 0 0
6 41271038 10344 0 0 0 0 5172 5172 0 0 0 0
7 36564769 9008 0 0 1 1 4503 4503 0 0 0 0
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 41.158s ( 777.488 blocks/s, 48.593 Mb/s)
|
|
663
|
Fri Jun 14 15:00:47 2024 |
Norah , Muneerah | 16:00-20:00 Friday 14 June |
Everything appears to be running smoothly. Attachments 1-4 were taken around 16:00
DSSSD Bias & Leakage Current - Check: OK (see attachment 1)
FEE64 Temperature - Check: OK (see attachment 2)
ADC Data Item Stats and Spectrum Browser - Check: OK (see attachments 3 and 4)
Another set of screenshots was taken around 18:00, and everything looks good.
DSSSD bias & leakage current ok - attachment # 5-6
FEE64 temperatures ok - attachment # 7
ADC data item stats - attachment # 8
Merger ok - Attachement #9
Tape service - attachement # 10
ucesb - attachment #11
At 19:17, AIDA03 temperatures gave no response (see attachment 12), but after two minutes, they returned to normal.
I noted that the bplse in ucesb has no data (see attachment 13). It may need to change the disc thresholds again
it does not affect AIDA
At 19:30 most aida temperature are high and I spoke to people on site via zoom one of them she is working with AIDA so she was able to help. She called Nick and it is found that there is a problem in the cooling system so she stopped AIDA
At 22 the cooling system fixed and Nic and the other person on site restarted AIDA
|
| Attachment 1: Screenshot_from_2024-06-14_16-06-09.png
|
|
| Attachment 2: Screenshot_from_2024-06-14_16-04-58.png
|
|
| Attachment 3: Screenshot_from_2024-06-14_16-04-23.png
|
|
| Attachment 4: Screenshot_from_2024-06-14_16-05-33.png
|
|
| Attachment 5: Screenshot_from_2024-06-14_18-02-36.png
|
|
| Attachment 6: Screenshot_from_2024-06-14_18-02-08.png
|
|
| Attachment 7: Screenshot_from_2024-06-14_18-04-40.png
|
|
| Attachment 8: Screenshot_from_2024-06-14_18-05-44.png
|
|
| Attachment 9: Screenshot_from_2024-06-14_18-08-50.png
|
|
| Attachment 10: Screenshot_from_2024-06-14_18-11-04.png
|
|
| Attachment 11: Screenshot_from_2024-06-14_18-17-15.png
|
|
| Attachment 12: Screenshot_from_2024-06-14_19-23-32.png
|
|
| Attachment 13: Screenshot_from_2024-06-14_19-35-11.png
|
|
|
608
|
Fri Apr 26 07:01:26 2024 |
Norah | 08:00 - 16:00 Friday 26 April |
10:00 Checks:
All seems smooth.
DSSD bias and leakage current ( Attachment 1)
FEE64 temperatures (Attachment 2)
Per FEE64 Rate spectra (Attachment 3)
ADC data item stats -(Attachment 4)
Ucesb (Attachment 5)
10:21
Aida05 was zero not reading had to call Nic to fix it.
10:34
Now, Aida05 is running well.
.......................................................
12:00 Checks:
Everything appears to be going smoothly, and I took screenshot of them.
12:28, Aida02 showed as not reading , had to email Nic to fix it.
12:30, it resumed reading and returned to normal operation.
.........................................................
14:00 Checks:
Everything is going well , and I have captured screenshots of each one.
......................................................
16:00 Checks:
All seems smooth, and I took screenshot of them.
|
| Attachment 1: Screenshot_from_2024-04-26_10-00-39.png
|
|
| Attachment 2: Screenshot_from_2024-04-26_10-02-35.png
|
|
| Attachment 3: Screenshot_from_2024-04-26_10-04-40.png
|
|
| Attachment 4: Screenshot_from_2024-04-26_10-06-18.png
|
|
| Attachment 5: Screenshot_from_2024-04-26_10-09-52.png
|
|
|
613
|
Sat Apr 27 15:13:06 2024 |
Norah | 16:00 - 00:00 Saturday 27 April |
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
| Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
|
|
| Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
|
|
| Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
|
|
| Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
|
|
|
81
|
Fri Nov 1 14:17:15 2019 |
Nic, Patrick | Reply: WR Timestamps |
> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> >
> > HDMI cables in aida09 checked and good.
>
> The problem would be the cable , one end or the other.
> I think ( if I recall ) a setup would restart the WR decoder.
>
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
Cable will be replaced once a spare is available.
Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
WR rate was chosen by Vic I believe, I am unsure of reasoning myself. |
|
90
|
Thu Nov 7 10:22:26 2019 |
Nic, Patrick | Reply: WR Timestamps |
> > > All 12 FEEs have valid WR Timestamps
> > > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > >
> > > HDMI cables in aida09 checked and good.
> >
> > The problem would be the cable , one end or the other.
> > I think ( if I recall ) a setup would restart the WR decoder.
> >
> > I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
>
> Cable will be replaced once a spare is available.
>
> Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
>
> WR rate was chosen by Vic I believe, I am unsure of reasoning myself.
An update/clarification:
Although the upper bits are zero I believe actually it is a total failure to synchronise to WR:
aida09
WR Time Item 0x80500000 0x0fbd8000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80400249 0x0fbd8000; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80500000 0x0fbdc000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80400249 0x0fbdc000; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80500000 0x0fbe0000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
WR Time Item 0x80400249 0x0fbe0000; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
aida10
WR Time Item 0x8050022e 0x0bde8000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x804e29d5 0x0bde8000; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x8050022e 0x0bdec000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x804e29d5 0x0bdec000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x8050022e 0x0bdf0000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
WR Time Item 0x804e29d5 0x0bdf0000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000 |
|
36
|
Tue Mar 26 14:58:09 2019 |
NH, VP | How to reset merger |
If the merger doesn't show "xfer Links => Merger" at the top, it is probably the merger setting has been corrupted somehow. This is how to reset it
If you go to the web page from which you started the Merge and TapeServer
(probably localhost:8115)
Then select the item NetVar Service which is at the left hand end.
Click on Inquire Registers
then enter NetVar.MERGE.RunOptions into the "Register Name" entry and when finished move the mouse out of the entry box.
NetVar.MERGE.RunOptions will appear in the center of the top row. (it is an item in the menu but not so easy to find)
Click on Read and I expect you will get 0
Type 1 into the data entry box and click on Write
Now return to the Merger and click on Reload
The Merger State will now end as xfer Links => Merger
The button Output: will now toggle between this and xfer links => Merger => Storage. |
|
534
|
Tue Mar 12 16:16:49 2024 |
NH, TD, JB, HA | Summary of DSSSD Biasing 12.03 |
DSSD#1 undergoes a breakdown at 90V, two of the three wafers show this
- The adapter PCBs themselves have no breakdown at 100V, indicating the issue is internal to the snout
If we are lucky it may be a loose or misaligned kapton connector inside the snout. If not we will have to remove DSSD#2 and inspect DSSD#1 for damage/lint/etc. If we see nothing it should be replaced
DSSD#2 biases perfectly fine, but the leakage current is unstable with biasing -ve to p+n and gnd (return) to n+n. Leakage current is stable biasing +ve to n+n and gnd to p+n. (https://elog.ph.ed.ac.uk/DESPEC/240311_093933/Downstream_positive_bias_vs._Current_(uA).png very nice curve)
We see the same fluctuations just daisy chaining 3 p+n PCBs togerher with the bias (no DSSSD or gnd links). The fluctuations were reduced by connecting all 8 adapter boards of the DSSD together.
When biasing just *one* PCB in this way, the current is stable
The fluctuations happened changing the HV channel, adapter board PCB and LEMO cables: it doesn't seem to be a defect with a specific thing.
The test today confirmed the HV cables (the only ones used) were correctly isolated from everything (OL = "infinite" resistance to ground)
We saw unstable current indications using the Mesytec MHV-4 to apply voltage to the PCB as well.
The behaviour is odd but doesn't seem to be related to the DSSD, which I suspect is OK. One idea is to try connecting the 3 adapter boards to FEEs and repeating the test, as this introduces another (substantial) path to gnd. Maybe this eliminates the current instability? It's about the only difference left from December.
-
For the next steps I believe the snout must be dismounted to inspect DSSD#1. It would be best to coordinate this with replacing the broken(?) bPlas. If we are lucky we may not have to remove the DSSDs but it is likely we have to remove both DSSDs to swap out #1
After replacing the DSSD(s) we should cover the snout with a clean black bin bag and bias it on the MH table to confirm both detectors work. This saves the effort of carrying it to S4 just to find another problem.
If both detectors make it to 120V we can mount it again more confidently
Once we have two biased detectors we can rearrange some FEEs to get 8 for DSSD#1 to do the noise tests. I suggest we wire them up for the numbering plan in https://elog.ph.ed.ac.uk/DESPEC/532 but in the DHCP renumber 9,10,15,16 to 5,6,7,8 temporarily to allow data to be sent to MBS for the dry run (merger limitation)
When the remaining FEEs are recovered from UK+CRYRING we can instrument DSSD#2 and renumber the FEEs to match the cables |
|
119
|
Fri Dec 20 12:04:03 2019 |
NH, TD | Friday 20 December 2019 |
Fig 1: DSSD Leakage currents ok
Fig 2: FEE Temps ok
Fig 3: aida08 has a few timestamp errors in merger
Fig 4, 5: GSI WR OK
Fig 6: All System wide checks passed
FIg 7: Stats good
13:11 - Alpha Run Stopped, Final File: R9_6
FEEs powered off, DSSD bias off. |
| Attachment 1: Screenshot-CAEN_HV_Control.png
|
|
| Attachment 2: FeeTemps.png
|
|
| Attachment 3: mergerTSerrors.png
|
|
| Attachment 4: gsiwr1.png
|
|
| Attachment 5: gsiwr2.png
|
|
| Attachment 6: checks.png
|
|
| Attachment 7: stats.png
|
|
|
277
|
Sat Apr 24 13:25:23 2021 |
NH, TD | AIDA FEE64 crash log for S460 |
https://docs.google.com/spreadsheets/d/1CyL5L-WLsh9fsS7CN989GC3ktqiPQOkG8YIDJrrGxYA/edit#gid=0 |