 Sun May 5 10:22:24 2024, TD, Sunday 5 May 
|
11.22 DSSSD#1 bias -120V leakage current -17.1uA
DSSSD#2 bias -120V leakage current -10.6uA
S4 temperature 25 deg C
cf. 07.00 29.4.24 S4 temperature 25 deg C
DSSSD#1 -21.8uA
DSSSD#2 -12.8uA
Leakage currents currently c. 80% of 29.4.24 values at same temperature |
|
Sat May 4 07:36:07 2024, TD, Saturday 4 May 11x
|
08.32 DSSSD bias & leakage currents - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC, correlation scalers data item stats OK attachments 4-5
11/16 <20k max aida16 174k
per FEE64 Rate spectra - attachmnt 6
09.44 per p+n FEE64 1.8.L spectra - attachment 7
aida09 pulser peak width 58 ch FWHM
per FEE64 1.8.W spectra - attachments 8-10
09.47 bias cycle - off and then on
no significant change in leakage current
09.57 FEE64 power OFF
DSSSD bias ON |
|
Thu May 2 23:08:43 2024, TD, Friday 3 May
|
00.05 FEE64 temperatures OK
ADC data item stats OK
09.31 FEE64 temperatures OK
ADC data item stats OK
13.15 FEE64 temperatures OK
ADC data item stats OK
9/16 <20k max aida16 164k
20.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida03 no data - DAQ reset
9/16 <20k max aida16 158k
23.15 FEE64 temperatures OK
ADC data item stats
aida03 no data - DAQ reset
10/16 <20k max aida16 153k |
|
Thu May 2 00:16:22 2024, TD, NH, Thursday 2 May
|
01.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
6/16 < 20k
10.25 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 restarted
7/16 < 20k, max 220k aida16
all histograms zero'd
13.20 per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 59 ch FWHM
FEE64 temperatures OK
ADC data item stats
aida04 no data
5/16 < 20k, max 332k aida08
14.37
Moved Time Machine delayed scaler from aida04 to aida07
aida04 no data
Reset merger etc
aida05 no data
Reset merger etc
All FEEs with data again
S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C
17.26 FEE64 temperatures OK
ADC data item stats OK
19.28 FEE64 temperatures OK
ADC data item stats
aida02 no data - DAQ reset
|
|
Wed May 1 08:59:15 2024, TD, Wednesday 1 May
|
04.55 FEE64 temperatures OK
ADC data item stats OK *except* aida04 no data
TapeServer no storage mode
09.55 FEE64 temperatures OK
ADC data item stats OK *except* aida02 & aida04 no data
DSSSD bias and leakage current - Grafana - attachment 1
19.28 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
noted increase in rates for aid08 and aida16
5/16 < 20k |
|
Tue Apr 30 09:06:45 2024, TD, NH, Tuesday 30 April 7x
|
10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
9/16 FEE64s < 20k
per FEE64 Rate spectra - attachment 5
Merger etc - attachments 6-7
aida02, aida04 inactive
11.20 DAQ restarted
aida02 and aida04 now producing data
11.30 analysis data file R22_313
all deadtimes <1%
no timewarps
HEC data item rate 560Hz (all from one channel of aida08 asic #3)
aida03 & aida04 MBS rates c. 24kHz
14.32 FEE64 temperatures OK
ADC data item stats OK
all FEE64s producing data
14.43 all histograms & stats zero'd
[NH checks here]
21.26 aida04 stopped forwarding to merger
the "quick fix" (stop merger, rerun) worked
MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability
System Wide Checks
clocks PASSED
ADC IGNORED
WR:
Base Current Difference
aida02 fault 0xe4d8 : 0x65c0 : -32536
aida03 fault 0x2371 : 0x2380 : 15
aida04 fault 0xa210 : 0xaf58 : 3400
aida06 fault 0x3624 : 0x3626 : 2
White Rabbit error counter test result: Passed 12, Failed 4
FPGA:
Base Current Difference
aida02 fault 0xaa3b : 0xc455 : 6682
aida04 fault 0x5a4 : 0x6ba : 278
aida06 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1
Baselines reset
22.50 FEE64 temps ok
merger ok
all links active
ADC data item stats ok |
|
Mon Apr 29 08:11:16 2024, TD, Monday 29 April 41x
|
08.52 Beam off
09.11 New data file R22
Monitor accumulated implanted activity in bPlas and DSSSDs
ASIC settings 2024Apr26-01.29.53
DSSSD bias & leakage current OK - attachments 1-3
FEE64 temperatures OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
Merger etc - attachment 7
analysis data file R22_19 - attachment 8
all deadtimes 0.3%, or less
no timewarps
HEC data item rate 390Hz
10.00 all histograms & stats zero'd
13.24 DSSSD bias & leakage current OK - attachments 9-10
FEE64 temperatures OK - attachment 11
ADC data item stats - attachment 12
per FEE64 Rate spectra - attachment 13
per p+n FEE64 1.8.L spectra - attachments 14-16
common x/y axes - peak height proxy for width
aida09 pulser peak width 58 ch FWHM (~41keV FWHM) cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
electronic noise today very similar to electronic noise Monday 22 April
per FEE64 1.8.W spectra - 20us FSR - attachments 17-19
Merger etc - attachment 20
analysis data file R22_60 - attachment 21
max deadtime 1.7% (aida11), all others <1%
no timewarps
HEC data item rate 710Hz
14.13 ucesb - attachment 22
14.15 aida04 not producing data
14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
15.27 *all* FEE64s stop producing data - reason unknown
16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
16.29 DSSSD bias & leakage current OK - attachments 23-24
FEE64 temperatures OK - attachment 25
ADC data item stats - attachment 26
per FEE64 Rate spectra - attachment 27
Merger etc - attachment 28
20.45 DSSSD bias & leakage current OK - attachments 29-30
FEE64 temperatures OK - attachment 32
ADC data item stats - attachment 31
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
20.54 analysis data file R22_145 - attachment 35
max deadtime 1.5% (aida11), all others <1%
no timewarps
HEC data item rate 775Hz
03.45 DSSSD bias & leakage current OK - attachments 36-37
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41 |
|
Sun Apr 28 22:52:06 2024, Marc, 0:00-08:00 Monday 29 April 7x
|
Just another manic magic Monday, ....
Taking over Betool's AIDA shift. All is good.
This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.
01:15: AIDA04 is down again. DAQ shift crew informed.
01:27: AIDA04 is back and shift crew has been informed.
01:30 Checks:
Bias and leakage current - Attachment 1-2
Temperatures - Attachment 3
ADC Data item - Attachment 4
Rate Spectra - Attachment 5
UCESB - Attachment 6
Timestamp checks - Attachement 7
06:00 It has run very smoothly.
|
|
Sun Apr 28 20:35:57 2024, Betool Alayed, 16:0-00:00 28 April 2024 10x
|
21:30 screenshots
and
23:24 screenshots |
|
Sun Apr 28 16:26:47 2024, Betool Alayed, 16:00-00:00 Sunday 28 April 10x
|
19:24 pm screenshots
and
17:18 pm screenshots |
|
Sun Apr 28 16:07:59 2024, TD, [How To] Recover DAQ when one FEE64 stops sending data - short(er) version
|
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
6) Start DAQ
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
8) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
9) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
10) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance. |
|
Sun Apr 28 15:49:58 2024, TD, DAQ restart short(er) cut
|
16.45 aida04 stopped producing data
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
aida04 did not stop, all other FEE64s stopped
restart Merger (workspace 3, display 2, select 'NewMerger' icon)
SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
GO Merger (workspace 3, display 1, browser 'NewMerger' tab)
aida04 now stops
disable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
enable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
|
|
Sun Apr 28 15:09:34 2024, Betool Alayed, 16:0-00:00 28 April 2024
|
16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
|
Sun Apr 28 06:57:59 2024, Magda Satrazani, 08:00-16:00 Sunday 28 April 36x
|
Took over the shift from Marc.
10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19
Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD
10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).
Grafana (AIDA) is now reporting data correctly too (attachment 21).
Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )
Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.
Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.
14.45 analysis data file R21_500 - attachment 36
max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%
no timewarps
HEC data item rate 2.7kHz
|
|
Sat Apr 27 23:04:42 2024, Marc, 00:00-08:00 Sunday 28 April 12x
|
Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C
0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.
After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.
I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.
I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.
We are now back in business with AIDA back in main DAQ at 01:40
01:45 full checks:
Bias: AIDA-Graphana - Attachment 1
Leakage current ramping up again ! S4 temperaturenot really going down either: 25C
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 5
ucesb - attachment 6
03:00 full checks
No change to be highlighted. All pretty much the same as during previous full checks.
05:00 full checks
Expt still running very smoothly.
Bias: AIDA-Graphana - Attachment 7-8
Leakage current going down slightly
FEE64 temperatures OK - attachment 9
ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
per FEE64 Rate spectra - attachment 11
Merger etc - attachment 12
ucesb - attachment 13
07:15 all running smoothly.
07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.
07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.
|
|
Sat Apr 27 15:13:06 2024, Norah, 16:00 - 00:00 Saturday 27 April 15x
|
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
|
Sat Apr 27 04:59:31 2024, TD, 08:00-16:00 Saturday 27 April 37x
|
06.00 DAQ continues file S100_alpha/R20_740
FRS setting 162Eu
DSSSD bias & leakage current - Grafana - attachments 1-2
leakage currents tracking S4 ambient temperature
FEE64 temperatures OK - attachment 3
All system wide checks OK *except*
as reported - attachments 4-5
*all* ADCs not calibrated (not calibrated post-synchronise ASIC clocks at most recent restart)
WR timestamps OK - attachment 6
ADC, PAUSE, RESUME, Correlation scaler data item stats OK - attachments 7-10
per FEE64 Rate spectra - attachment 11
per p+n FEE64 1.8.L specra - attachments 12-13
aida09 pulser peak width 55 ch FWHM (~38keV FWHM)
per FEE64 1.8.H spectra - attachments 14-15
Merger, TapeServer etc - attachments 16-17
12Mb/s to disk
ucesb - attachment 18
spill length 3.5s, extraction 1.5s
06.30 analysis data file R20_748 - attachment 19
max deadtime 12%
no timewarps
HEC data item rate 2.1kHz
06.35 new run file S100_alpha/R21
09.30 DAQ continues - file S100_alpha/R21_83
DSSSD bias & leakage currents - Grafana - attachment 20
FEE64 temperatures OK - attachment 21
ADC data item stats OK - attachments 22
per FEE64 Rate spectra - attachment 23
Merger etc - attachments 24
ucesb - attachment 25
spill length 3.5s, extraction 1.5s
09.50 per FEE64 1.8.W spectra - 20us FSR - attachments 26-27
no significant change in electronic noise cf. start of run
per p+n FEE64 1.8.L spectra - attachment 28
aida09 pulser peak width 62 ch FWHM
common x/y scale - peak heoght proxy for width
13.57 DAQ continues file R21_141
DSSSD bias & leakage current - Grafana - attachments 29-30
FEE64 temperatures OK - attachment 31
ADC data item stats OK - attachments 32
per p+n FEE64 1.8.L spectra - attachment 33
aida09 pulser peak width 55 ch FWHM (~39keV FWHM)
per FEE64 Rate spectra - attachment 34
Merger etc - attachments 35
ucesb - attachment 36
spill length 3.5s, extraction 1.5s
14.15 analysis data file R21_144 - attachment 35
max deadtime 16% (aida04)
no timewarps
HEC data item rate 2.0kHz |
|
Fri Apr 26 23:04:51 2024, Marc, 00:00-08:00 Saturday 27 April 12x
|
It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!
0.15 Checks:
DSSSD bias & leakage current - Grafana - attachments 1-2
leakage current dropped a bit since the evening shift ~22:00.
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
per FEE64 Rate spectra - attachment 5
Merger etc - attachment 6
ucesb - attachment 7
2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12
3:30 Still running smoothly.
5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.
DAQ restarted OK. Main DAQ informed.
|
|
Fri Apr 26 15:34:00 2024, RDP, 16:00 - 00:00 Friday 26 April 6x
|
This appeared in the system log:
Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
These messages seem to appear sporadically.
18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?
18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.
aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters
Thanks Nic!
22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)
22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.
|
|
Fri Apr 26 08:15:59 2024, TD, [How To] Recover DAQ when one FEE64 stops sending data
|
Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.
Typical symptoms are:
- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
active data links are green, inactive links are red
N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Reboot problem FEE64s
Launch a command terminal and telnet to the problem FEE64 for example
telnet aida04
Enter user 'root' and enter password. When logged in reboot the FEE64 using the command
reboot
6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
you should see two messages showing the FEE64 mounting filesystems
wait c. three minutes for boot to complete
7) Restart DAQ
Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
8) Restore ASIC settings
Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
Wait 10s
9) Check ASIC settings
Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
Wait for popup indicating completion of checks and dismiss
10) System wide checks
Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check clock status' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
make a note or take a screenshot of any error messages observed
11) Start DAQ
Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
13) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
14) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
15) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance.
|
|