| ID |
Date |
Author |
Subject |
|
427
|
Sun Nov 6 05:16:22 2016 |
OH | Update | Leakage current spreadsheet has now been uploaded to google docs for ease of access.
It can be accessed and edited from the following link: https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit?usp=sharing |
|
425
|
Sun Nov 6 01:34:15 2016 |
TD, OH, CG | Sun 6th November | 10:34 Detector Bias a leakage currents all ok. (Attachments 1 and 2)
Temperatures all ok (Attachment 3)
All system wide checks ok other than clock status see attachment 4
DAQ continues to run ok (R124) (No stop at 6:30am)
DAQ running at about 14MB/s to disc which is faster than before.
Most likely due to the increased rate in detector 1 which is represented at FEEs 17, 20, 22 and 23 in
attachment 5.
Also attached are the spectra for layouts 1-4 & 7-8 as attachments 6-11
When looking at the pulser spectrum in layouts 3 and 4 (attachments 9 and 10) we see that the pulser peaks
relating to this detector is broader shown by
the decreased peak height.
Comparing the waveforms in attachments 11 and 12 to those taken on Monday 31st Oct we see that the noise
in nnaida 5 and 6 has been greatly reduced.
On the others however there appears to be an increase in the amplitude of the noise.
11:25 The second putty window for leakage currents was found to have been frozen for the past 24 hours.
Was reloaded and updated leakage currents are in attachment 13 and 14
12.15 Analysis of R124_291 shows ADC & disc data synchronised - see attachment 15
Analysis of data file - see attachment 16
18:26 DAQ stopped, presumably during LN2 fill cycle.
Restarted the merger and tried "Stop" on the DAQ control page. ~50% of nnaida now show "error."
R124 closed 18.29
Could not regain control of the system. Jorge switched BRIKEN to its internal clock and FEEs power cycled @
19.20.
23.35 DAQ started (R126 - no storage mode)
DSSSD bias & leakage currents OK - see attachment |
|
424
|
Sat Nov 5 09:40:02 2016 |
OH | Updated Leakage Current Spreadsheet | Attached is the updated leakage current spreadsheet (Attachment 1) as pf 5/11/2016 it is up to date with the elog.
Also added is the recording of ambient temperature. This is recorded in degrees C and plotted relative to 21 degrees C to better show on the same plot as leakage currents. (Screenshot in attachement 2)
Also attached is the spreadsheet in csv formatting. |
|
423
|
Sat Nov 5 01:55:15 2016 |
CG, TD, OH | Saturday 5th November | 10.55 DAQ found to have stopped at 06:47 during LN fill cycle.
10:57 Screen shots of system after stop
11.20 R122 started.
System check screen shots attached at 11:08
18.20 System been running stably all day. Leakage currents attachments 14+15.
LN2 fill cycle shortly, anticipate DAQ stop.
19.16 DAQ stopped.
Closed file R122.
19.37 Temp 25.3 DP 6.1 Hum 31.1
Added 2 extra stacks of lead bricks to protect AIDA. Now ~15cm of lead (Attachment 17)
23.30 DSSSD bias & leakage currents OK - see attachments 18-19
FEE74 temperatures OK - see attachment 20
23.54 DAQ starts (R124)
Merge rate 1.8M items/s, c. 13Mb/s to disk c. x2 earlier rates
Dominated by significantly higher rates from DSSSD#1
FEE64 #17 ASIC 2*L pulser peak widths ~300 ch FWHM
Needs to be investigated at next entry opportunity |
|
422
|
Fri Nov 4 08:10:46 2016 |
CG | Pb bricks upstream | Stack of Pb bricks upstream of AIDA+BRIKEN to protect from incoming beam. |
|
421
|
Fri Nov 4 04:35:03 2016 |
TD, CG, DK | Friday 4 November part 2 | 13.35 Swapped HDMI cables from master MACB to MACB #3 & #7
for HDMI cables from MACB #7 ports 1 & 2 to MACB modules #8 & #9
ReSYNC and stsrem wide test failures - see attachments 1-5
15.00 Swap HDMI cable from MACB #2 port 1 to MACB #3 next
ReSYNC and system wide test fails - see attachments 6-10
*Conclusion* all of HDMI cables between HDMI cable from nnaida6 and master MACB
have now been swapped for other HDMI cables between MACB # 7 and MACB # 8, 9, 10
but we still observe ReSYNC and clock issues
16.05 CG points out that I have misinterpreted Elog 258 - switch back MAC
mode settings to the following
MACB # 1 2 3 4 5 6 7 8 9 10 11 spare 1 spare 2
Mode 5 2 2 3 3 3 3 3 3 3 3
Rev B y y 50MHz? y
FEE64 power cycle
ReSYNC and system wide fails - see attachments 11-16
16,50 Re-considering options
Install MACB #11 (rev B) mode 0
Will attempt to use 25MHz correlation scaler clock NIM output from master MACB to BRIKEN+RIBFDAQ
Power cycle FEE64s
ReSYNC OK
System wide checks OK but some issues remain - see attachments 17-25
17.10 DAQ starts
Writing to disk file R115, merge rate 0.9M data items/, TapeServer ~7Mb/s
17.20 Analysis of R115_1 confirms ADC data synchronised - see attachment 26
17.25 MERGE stats look OK - see attachments 27-29
17.35 System wide tests SYNC errors & SYNC pulses OK
17.38 DAQ stalled - manual start of ORNL clover LN2 fill cycle by Nishimura-san
Most recent file R115_5 last accessed 17.38
DAQ stop OK
Disable histogram and data transfer #1
DAQ RESET
DAQ SETUP
Restart merger and TapeServer
Enable histogram and data transfer #1
17.47 DAQ start (R116)
17.58 DAQ stopped again at end of fill cycle.
1. Disable histogram and data transfer #1
2. DAQ RESET
3. DAQ SETUP
4. Restart merger and TapeServer
5. Enable histogram and data transfer #1
Had to repeat 1-5 multiple times to get all links working with the merger. R117 created during this (empty).
18.08 Run R118 started.
18.45 TD changing things for runs R119-R120.
18.50 Run R121 started.
First few minutes are pulser run 30-10,000 @ 250Hz on coarse dial.
18.54 Beam ON.
Continuing R121. Pb shield immediately after final MUSIC detector.
21.35 DAQ continues OK
Most recent file R121_30
All system-wide tests OK *but* note nnaida5 now fails clock status test in addition to combinations of
nnaida7/8/15/16
Merge/Tapeserver desktop, Merge statistics
- see attachments 30-36
21.45 R121_32 ADC data synchronised
Overnight
Phong Vi completed the pulser walkthrough interrupted earlier this evening.
BNC PB-4 amplitude settings 30,000 - 20,000 @ 10,000 step, 8,000-2,000 @ 2,000 step
The data is in files R121_62, _63 and _64 (which have been copied to d05 for further analysis). |
|
419
|
Thu Nov 3 17:42:37 2016 |
TD, CG, DK | Friday 4 November |
00.40-02.30 Discussion with PJCS
Confirm both NIM bin voltages OK
Move master MACB to lower NIM bin (with BNC PB_4, Ortec 433, 2x CAEN N1419B, 1x CAEN N1419)
successful ReSYNC
system-wide checks clock test fails from zero to all of the following FEE64s - nnaida7, 8, 15 & 16
SYNC count test OK *except* nnaida19 which has greater number of SYNC pulses
online analysis indicates nnaida19 ADC data not synchronised with other FEE64s by ~200ms
Replace master MACB rev B mode 0 with MACB module #13 mode 5 (external clock 50MHz TTL)
Move MACB module 2 from upper NIM bin to lower NIM bin
Swap nnaida19/port1 <-> nnaida20/port2
Power cycle FEE64s and restart
System wide checks fail - see attachments 1-4
Re-seat HDMI cabling master MACB -> MACB modules 3-6
System wide checks fail - see attachments 7-10
Swap ports of 2x HDMI cables from master MACB
Power cycle FEE64s
Re_SYNC & system wide checks fail - see attachments 11-15
10.20 ReSYNC fails for all TS_SYNC_PHASE settings 0-3
DK checks MACB mode settings
master -> 5
module #2 2
module #3 2
modules #4-11 3
Check HDMI cable seating nnaida6 to master MACB
No change - ReSYNC fails, system wide checks fail as before
Per PJCS e-mail check MACB settings per https://elog.ph.ed.ac.uk/AIDA/258
and change MACB module # 7 from mode 3 to mode 2. MACB settings are now
MACB # 1 2 3 4 5 6 7 8 9 10 11 spare 1 spare 2
Mode 5 2 3 3 3 3 2 3 3 3 3
Rev B y y 50MHz? y
*BUT* remember the system has been in the previous state since Elog 258
abd has been working OK
No change - ReSYNC fails, system wide checks fail as before
Check 50MHz TTL external clock signal and observe transients and width changes
in clock signal - see attachment 16
This may not be *the* explanation of current problems but I doubt the the hardware
will be 'happy' with it either
JAR cycles NIM signal through NIM-TTL signal converter several times to produce
better (no transients or width changes) clock signal with c. 3.2V amplitude
Power cycle and restart FEE64s
No change - ReSYNC and system wide checks fail - see attachments 17-21
12.40 Re-checked HDMI cabling
found master MACB port 1 -> MACB #7 next, port 2 -> MACB #3 next
change to master MACB port 1 -> MACB #3 next, port 2 -> MACB #7 next
Power cycle FEE64s
ReSYNC partial failure (left side OK, right side of tree fails), system wide checks fail - see attachments
22-26
13.05 Swap HDMI cables from master MACB to parents, i.e. HDMI cable master port 1 to MACB #3 next
changes to master port 2 to MACB #7 next
ReSYNC and system wide checks fail - see attachments 30-34
|
|
418
|
Thu Nov 3 02:29:45 2016 |
TD | [How To] Startup AIDA (October 2016) | Assumes 3-phase ac power (to Julabo FL11006 recirculating chiller) and single-phase ac power
(to AIDA NIM bins, PSUs, network switches, USB-controlled ac mains relay and Raspberry Pi)
is connected and switched on.
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
1) Start Julabo FL11006 Recirculating Chiller
See manual (page 19) http://www2.ph.ed.ac.uk/~td/AIDA/Information/Julabo%20FL11006.pdf
Set point +20 deg C
Check flow of coolant by inspecting the two flow indicators which are located on top of
the AIDA stand
Do NOT proceed if the Julabo FL11006 Recirculating Chiller is not operating, or if there
is little, or no, flow in one, or both, flow indicators.
2) Start interface to USB-controlled ac mains relay to AIDA data acquisition servers
On the AIDA Dell T410 Workstation aidas1 switch to Desktop 7 and start a Firefox web browser
Enter URL http://nnrpi1:8015/AIDA/Rly16/
See attachment 1
Select 'Sequence ALL On' to powerup the AIDA data acquisition servers
Check all relays Relay 1 to Relay 8 move from the 'Switch ON' to 'Switch OFF' row
See attachment 2
Wait 10 minutes (yes - really!) before proceeding
3) Start Httpd Data Acquisition server
See attachment 3
Switch to Desktop 1 and select the yellow MIDAS icon 'Httpd for DataAcquisition' which will start a terminal
to display server messages as follows
System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-504.8.1.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS@aidas and MIDAS_LIBRARY = /MIDAS@aidas/TclHttpd/Linux64
PATH =
/MIDAS@aidas/bin_Linux64:/MIDAS@aidas/TclHttpd/Linux64:/homes/npg/root/bin:/homes/npg/root/bin:/homes/npg/root/lib:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "jhi+u8jzdbpl"
httpd started on port 8015
Custom startup from /MIDAS@aidas/config/TclHttpd/aidas1@8015/startup.tcl
DefineMessage defined with server at http://elog.ph.ed.ac.uk:8015/MessageLoggerServer
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
loading tcl/AIDARunControl.tcl for namespace ::
RunControlClient provided
RunControlClient loaded
Completed custom startup from /MIDAS@aidas/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
See attachment 4
4) Start web browser for interface to AIDA
Switch to Desktop 1 and start Firefox web browser
Enter URL localhost:8015
See attachment 5
5) Commence startup of AIDA interface
Select 'AIDA Experiment Control'
See attachment 6
6) From AIDA Experiment Control tab
Select 'AIDA Hardware Control'
See attachment 7
Select 'Data Acquisition Run Control'
7) Reset and setup all data acquisition servers
From 'Data Acquisition Run Control' tab
See attachment 8
Select 'Act On ALL Data Acquisition Servers'
The web page will update (~20s) to show the status of all of the configured AIDA FEE64
data acquisition servers which are named nnaida1, nnaida2 ... etc
See attachment 9
Select 'Reset'
Wait for update and check that the state of *all* data acquisition servers is 'reset'
Select 'Setup'
Wait for timestamp re-synchronisation pop-up
See attachments 10-11
Usually all data acquisition servers (nnaida1, nnaida2 ... etc) will ReSYNC OK and you may proceed to step 8.
Sometimes one, or more, data acquisition servers report ReSYNC not complete.
Re-select the 'AIDA Experiment Control' tab and select 'AIDA Hardware Control'
See attachments 12-13
From the 'AIDA Hardware Control' tab select 'Master Timestamp Control'
From the 'Master Timestamp Control' tab
Select 'Reload'
Select FEE64 -> nnaida6
Select 'Perform ReSYNC'
Wait for timestamp re-synchronisation pop-up.
All data acquisition servers 9Nnaida1, nnaoida2 .. etc) should ReSYNC OK - if they do not
please seek expert help
8) Start ASIC controls
Re-select the 'AIDA Experiment Control' tab and select 'AIDA Hardware Control'
See attachments 12-13
Select 'ASIC Control'
See attachment 14
From 'ASIC Control' tab
select 'Act on ALL FEE64 modules' and await page update
select 'Act on ALL ASICs' and await page update
See attachment 15
9) Check FEE64 operating temperatures
Re-select the 'AIDA Hardware Control' tab and select 'FEE Temperatures'
See attachment 16
Check temperatures
Virtex < 70 deg C
PSU < 45 deg C
ASIC < 70 deg C
Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0
*If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows
Desktop 6 - Firefox web browser
From 'Rly16 Control' select 'Switch ALL off' and *now* seek expert help
10) Perform system-wide diagnostic checks
Re-select the 'AIDA Hardware Control' tab and select 'System Wide Checks'
See attachments 17-18
11) Start servers for data merging and output to disk
Switch to Desktop 2 and start the 'Httpd for Merge/TapeServer', 'Merger' and 'TapeServer' servers
by selecting the three adjacent MIDAS in sequence
See attachment 19
Three terminal sessions will pop-up to display server messages. These terminals should be moved to
left of the desktop and arranged as shown
See attachment 20
This will enable you to simultaneously observe the latest messages from each server and the activity
spinners of the Merger and TapeServer servers
12) Start web interface to Merger and TapeServer
Start Firefox web browser
Enter URL localhost:8115
See attachment 21
13) Open data file
Select 'Tape Service'
See attachment 22
In the 'TapeServer Service' tab complete the Volume Name, Run Name and Run Number fields.
Typically the volume name will the experiment ID (e.g. NP1234) and Run Name (R).
See attachment 23
Select 'Go'
See attachment 24
This will create an experiment directory if it does not already exist (e.g. /TapeData/NP1234)
and will open a data file within this directory with a filename concatenating the Run Name
and Run Number fields - if the selected file already exists a new filename (Run Number) will
be suggested
14) Setup merger
Re-select the 'AIDA' tab and select 'MERGE Control'
See attachment 25
Select 'Setup & Configure'
See attachment 26
Select 'Go'
See attachment 27
Select 'Toggle Merge Program Pause State'
See attachment 28
15) Enable histogramming and data transfer #1 links
Switch to Desktop 1 and re-select 'Data Acquisition Run Control' tab
Select Histogramming 'Enable All' and Data Transfer #1 'Enable All'
16) AIDA is now ready to commence collecting data
- you can proceed from step 5 of https://elog.ph.ed.ac.uk/AIDA/412 |
|
417
|
Thu Nov 3 01:25:30 2016 |
TD, CG | Thursday 3 November | 10.20 DAQ found stalled
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
AIDA SYNC stats - see attachment 4 - stats 10.1M/381/3600 => DAQ stalled at 23.05 + 7.4h => c.06.29
System wide checks OK *except* SYNC pulses & SYNC errors - see attachments 5-8
Conclude DAQ stalling at c. 06.30/18.30 due to start of ORNL LN2 dewar refill cycle
but unrelated to either the BRIKEN-supplied 50MHz TTL clock signal to external clock
input of Master MACB or the ground connection provided by this coaxial cable.
Which implies the problem is either EMI/RFI or ac mains electrical ground.
AIDA powered by two single phase outlets from one of four sections of a single phase ac mains distribution
box with power supplied by two heavy duty (3cm dia) cables from Denkenseiki Noisecut AVR type PMN (AVR4, B3F
104V/51A). AIDA is connected to both NEMA 5-15R-style outlets of section A, LN2 refill to section D, BRIKEN
to sections B, C and D.
Ambient temperature +24.8 deg C, d.p. +4.1 deg C, RH 26.3%
15.00 DSSSD cabling drain wire attachment to AIDA adaptor PCB (FEE64 # 5 & 6) not tightly secured.
CG re-secured drain wire and this has significantly reduced 100kHz period transient noise
as observed at preamplifier outputs.
14.48 Restart DAQ (R110)
Master MACB mode 5, 50MHz TTL clock to external clock input
14.57 DAQ stalled
All system wide checks OK *except* SYNC errors - see attachment 9
ASIC clock timestamp - nnaida19 fails
N.B. SYNC pulses check OK, counters incrementing between successive tests
The latter is a problem we have seen previously but has not stopped DAQ running OK
aidas1> ./analyser /TapeData/Sept2016/R110_2
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 11861
ADC data format: 82556471 ( 877427.0 Hz)
Other data format: 14217428 ( 151105.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 27 ( 0.3 Hz)
RESUME: 27 ( 0.3 Hz)
SYNC100: 861083 ( 9151.8 Hz)
FEE64 disc: 13353468 ( 141923.4 Hz)
MBS info: 2823 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 1976 ( 21.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 94.089 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.084 0.000
2 0.076 0.000
3 0.000 0.000
4 0.000 0.000
5 0.109 0.000
6 0.012 0.000
7 0.000 0.000
8 0.000 0.000
9 0.003 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.032 0.000
16 0.013 0.000
17 0.000 0.000
18 0.000 0.000
19 0.125 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.142 0.149
24 0.076 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS Other
HEC Data
0 0 0 0 0 0 0 0 0 0 0
0
1 942953 5806860 0 0 3 3 35855 5770999 0 0
42
2 4036003 4268394 0 0 2 2 35861 4232529 0 0
0
3 3038422 39132 0 0 0 0 35892 3240 0 0
0
4 1192440 42063 0 0 0 0 35889 6174 0 0
68
5 5685966 39310 0 0 5 5 35849 3451 0 0
0
6 3670394 46605 0 0 2 2 35885 7893 2823 0
70
7 2661022 43508 0 0 0 0 35890 7618 0 0
231
8 2165335 41382 0 0 0 0 35890 5492 0 0
359
9 3037520 863964 0 0 1 1 35889 828073 0 0
0
10 2616647 179334 0 0 0 0 35889 143445 0 0
0
11 1083633 39064 0 0 0 0 35889 3175 0 0
0
12 1759629 38368 0 0 0 0 35888 2480 0 0
0
13 2419715 280498 0 0 0 0 35891 244607 0 0
0
14 3189205 1325559 0 0 0 0 35887 1289672 0 0
747
15 4886680 39482 0 0 2 2 35871 3607 0 0
7
16 5087977 38869 0 0 1 1 35886 2981 0 0
0
17 2905638 329411 0 0 0 0 35888 293523 0 0
0
18 3327386 46016 0 0 0 0 35888 10128 0 0
0
19 6770177 40711 0 0 4 4 35842 4861 0 0
7
20 2225755 38524 0 0 0 0 35891 2633 0 0
344
21 2061728 40299 0 0 0 0 35887 4412 0 0
90
22 3052329 512109 0 0 0 0 35890 476219 0 0
7
23 9325689 38673 0 0 3 3 35834 2833 0 0
3
24 5414228 39293 0 0 4 4 35862 3423 0 0
1
25 0 0 0 0 0 0 0 0 0 0
0
26 0 0 0 0 0 0 0 0 0 0
0
27 0 0 0 0 0 0 0 0 0 0
0
28 0 0 0 0 0 0 0 0 0 0
0
29 0 0 0 0 0 0 0 0 0 0
0
30 0 0 0 0 0 0 0 0 0 0
0
31 0 0 0 0 0 0 0 0 0 0
0
32 0 0 0 0 0 0 0 0 0 0
0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 1.500s ( 7907.333 blocks/s, 494.208 Mb/s)
15.48 Restart aidas1
Power cycle FEE64s
Restart AIDA DAQ
Same problem - DAQ stall (no output from merger) at, or shortly after, Start ASIC Readout with
multiple FEE64s with zero AIDA SYNC rate
*All* system wide checks OK
Cannot recover DAQ control of FEE64s with merger restart - power cycle FEE64s again
23.58 Attempt to change master MACB from MACB module #11 mode 5 to MACB module #13 mode 0
Successive fuse failures of USB ac mains relay
System wide ASIC clock timestamp check nnaida 5/6/7/8 13/14/15/16 21/22/23/24 i.e. first half of MACB tree
Re-seat HDMI cabling from master MACB root module to first half of MACB tree - ReSYNC fails - see attachments
10 & 11
|
|
416
|
Tue Nov 1 23:55:33 2016 |
TD, CG | Wednesday 2 November | 08.56 DAQ continues OK
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses - see attachments 4-6
Most recent file R101_249 ADC & disc data synchronised - see attachments 7-8
spectrum s1000 - per ASIC data items as function of timestamp (20us/channel)
spectra 1-32 and 100-132 per FEE64 data items as function of timestamp (20us/channel)
09.03 DAQ stopped
17.10 DAQ started (R103) no storage mode
Master MACB module #1 mode 0, 50MHz TTL clock signal cable disconnected from MACB rear panel.
18.55 Checked DAQ status and it has stopped. No file write times so can't easily check exactly when (see SYNC
stat page to calculate).
All system checks passed except SYNC errors (see attachments 9-12)
Upon attempting to Stop the DAQ, could not contact nnaida11.
AIDA SYNC statistics c. 1.76M SYNCs per FEE64. SYNC rate = 381Hz => elapsed time c. 1.3h => DAQ stalled c.
18.30
23.05 DAQ started (R104) no storage mode
Master MACB module #1 mode 0, 50MHz TTL clock signal cable disconnected from MACB rear panel. |
|
415
|
Tue Nov 1 02:24:05 2016 |
TD | Tuesday 1 November | 11.24 DAQ found stalled
Most recent file R95_168 last accessed 06.31
R95_168 ADC & disc data synchronised - see attachment 8
spectrum s1000 - per ASIC data items as function of timestamp (20us/channel)
aidas1> ./analyser /TapeData/Sept2016/R95_168
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 14568
ADC data format: 108169612 ( 1126585.0 Hz)
Other data format: 10690700 ( 111343.5 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 31 ( 0.3 Hz)
RESUME: 30 ( 0.3 Hz)
SYNC100: 877566 ( 9139.8 Hz)
FEE64 disc: 9810241 ( 102173.5 Hz)
MBS info: 2832 ( 29.5 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 2160 ( 22.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 96.015 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 0.000
2 0.000 0.000
3 0.000 0.000
4 0.000 0.000
5 1.994 2.016
6 1.408 1.439
7 0.000 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.015 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.045 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.099 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS
Other HEC Data
0 0 0 0 0 0 0 0 0 0
0 0
1 350112 43200 0 0 0 0 36623 6577 0
0 48
2 1247240 43778 0 0 0 0 36622 7156 0
0 0
3 2547061 39987 0 0 0 0 36626 3361 0
0 0
4 881007 40879 0 0 0 0 36623 4256 0
0 5
5 23965264 2536358 0 0 13 12 35831 2500502 0
0 38
6 23702370 1860427 0 0 11 11 36086 1821487 2832
0 844
7 1895595 40743 0 0 0 0 36623 4120 0
0 1
8 1603094 41634 0 0 0 0 36626 5008 0
0 325
9 3180182 1849501 0 0 0 0 36625 1812876 0
0 0
10 2576178 97708 0 0 0 0 36622 61086 0
0 0
11 447806 39812 0 0 0 0 36625 3187 0
0 0
12 905803 39001 0 0 0 0 36618 2383 0
0 0
13 2226595 147679 0 0 0 0 36624 111055 0
0 0
14 3021228 2593022 0 0 0 0 36626 2556396 0
0 513
15 4146089 40401 0 0 2 2 36617 3780 0
0 0
16 4264696 39555 0 0 0 0 36619 2936 0
0 0
17 2934508 416504 0 0 0 0 36623 379881 0
0 0
18 3319005 56583 0 0 0 0 36619 19964 0
0 0
19 6346836 41482 0 0 2 2 36604 4874 0
0 1
20 2392653 39249 0 0 0 0 36623 2626 0
0 283
21 2056421 41512 0 0 0 0 36622 4890 0
0 96
22 2824822 522339 0 0 0 0 36624 485715 0
0 5
23 7402741 39352 0 0 3 3 36590 2756 0
0 1
24 3932306 39994 0 0 0 0 36625 3369 0
0 0
25 0 0 0 0 0 0 0 0 0
0 0
26 0 0 0 0 0 0 0 0 0
0 0
27 0 0 0 0 0 0 0 0 0
0 0
28 0 0 0 0 0 0 0 0 0
0 0
29 0 0 0 0 0 0 0 0 0
0 0
30 0 0 0 0 0 0 0 0 0
0 0
31 0 0 0 0 0 0 0 0 0
0 0
32 0 0 0 0 0 0 0 0 0
0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 2.449s ( 5948.019 blocks/s, 371.751 Mb/s)
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses & errors - see attachments 4-7
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
11.46 Start DAQ (R96)
16.28 Stop DAQ OK
16.30 Start DAQ (R97)
BNC PB-4 pulser OFF (background run)
17.04 Stop DAQ
17.08 Ambient temperature +25.0 deg C, d.p. +8.4 deg C, RH 34.8%
17.11 Start DAQ (R98)
BNC PB-4 pulser ON
60Co cource (10-7023) placed on top of AIDA snout approximately above DSSSD #3
17.30 DAQ stalled due to excessive diak activity from data copying & analysis tasks
Power cycle required
18.09 DAQ start (R99)
18.38 DAQ stalled
Most recent file R99_7 last accessed 18.38
Power cycle rquired
19.42 DAQ start (R101)
BNC PB-4 pulser ON
60Co cource (10-7023) placed on top of AIDA snout approximately above DSSSD #3
(Phong's correction: 60Co source position was in front of AIDA snout toward upstream)
ASIC settings 2016Nov01-18.02.17 |
|
414
|
Mon Oct 31 04:50:59 2016 |
TD | [How to] Monitor AIDA (October 2016) |
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
Every 4h
1) Desktop 7 - 2x PUTTY terminal sessions
Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply
Addr 00
Ch 0 = DSSSD # 1
:
:
Ch 3 = DSSSD # 4
Addr 01
Ch 0 = DSSSD # 5
Ch 1 = DSSSD # 6
Sequentially select each terminal and obtain a screenshot (Alt-PrtScn) of DSSSD #1 = #6 detector
bias voltages and leakage currents
Upload screenshots to BRIKEN Elog
If B2F/F11 ambient temperature information is available please add this information to the Elog
Compare DSSSD leakage currents (and temperatures) with previous observations
2) Desktop 1 - Firefox web browser - Temperatures tab
Select 'Reload'
Check temperatures
Virtex < 70 deg C
PSU < 45 deg C
ASIC < 70 deg C
Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0
*If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows
Desktop 6 - Firefox web browser
From 'Rly16 Control' select 'Switch ALL off'
and *now* seek expert help
*If* DAQ running also
3) Desktop 1 - Firefox web browser - Run Control tab
Select 'Update'
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags
5) Desktop 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check merge rate (~0.5-2M data items/s) consistent with value observed at start of run
Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser
6) Desktop 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check data rate consistent with value observed at start of run |
|
413
|
Mon Oct 31 04:02:11 2016 |
TD | [How To] Stop Runs (October 2016) |
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
How to Stop Run
1) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check number of blocks written to disk is increasing
See attachment 1
If TapeServer data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303
2) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check merge rate (~0.5-2M data items/s)
See attachment 2
If MERGER data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303
3) Workspace 1 - Firefox web browser - Run Control tab
Select 'Update'
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags
Stop DAQ - select red 'STOP' button
Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates
Check state of *all* data acquisition servers is 'stopped'
Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags
See attachment 3
N.B. Occasionally the state of all data acquisition servers does not change to 'stopped'
and if this happens simply re-try by re-selecting 'STOP'
4) Workspace 2 - Firefox web browser - MERGE tab
Pause MERGE input - select 'Toggle Merge Program Pause State'
See attachment 4
5) Workspace 2 - Firefox web browser - TapeServer tab
Stop TapeServer and close disk file - select red 'STOP' button
See attachment 5
6) Complete BRIKEN Elog with the following information
AIDA
run filename (e.g. R99)
[# data blocks]
start date & time
[stop date & time]
ASIC settings DB key
additional comments
your name
Complete run sheet for written logbook |
|
412
|
Mon Oct 31 02:26:32 2016 |
TD | [How To] Start Runs (October 2016) | All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
How To Start Run
1) Workspace 1 - Firefox web browser - Run Control tab
Select 'Update'
Check DAQ stopped (green 'GO' button displayed)
Check state of *all* data acquisition servers is 'stopped'
Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags
See attachment 1
2) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check TapeServer stopped and disk file closed (green GO button displayed)
See attachment 2
3) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check Merge Program paused - see banner
Merge State = GOing : paused : xfer enabled : output paused
If not paused select 'Toggle Merge Program Pause State'
See attachment 3
4) Workspace 1 - Firefox web browser - ASIC Control tab
Select 'Reload'
Check 'Act on ALL FEE64 modules?' selected
Check 'Act on ALL ASICs' selected
Select 'Select required function' -> 'Check ASIC Control'
After ~60s a pop-up will appear showing the response from each ASIC of each data acquisition server.
Check all ASICs are OK and status = 0x001F0000.
Dismiss pop-up by selecting OK
See attachments 4-5
5) Workspace 1 - Firefox web browser - Run Control tab
Start DAQ - select green GO button
Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags
See attachment 6
6) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload' several times and check *all* enabled data links (0-23) are active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are inactive call for expert help and do NOT proceed!
Observe active spinner in MERGER server terminal window
See attachment 7
7) Workspace 2 - Firefox web browser - TapeServer tab
Check 'Volume Name' and 'File Name'
Start TapeServer and open disk file - select green 'GO' button
See attachment 8
8) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload' several times and check all enabled data links (0-23) are always active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are *always* inactive call for expert help and do NOT proceed!
Select 'Toggle Merge Program Pause State' to enable data merging
Select 'Reload' to observe merge rate (~ 9000 data items/s)
Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser
See attachment 9
9) Workspace 1 - Firefox web browser - Run Control tab
Select 'Start ASIC Data Readout'
See attachment 10
10) Workspace 2 - Firefox web browser - MERGE tab
Select Reload several times and check all enabled data links (0-23) are active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are *never* active call for expert help and do NOT proceed!
Select 'Reload' to observe merge rate (~ 0.5-2M data items/s)
See attachment 11
11) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload' and observe number of blocks written to disk increasing
See attachment 12
12) Workspace 1 - Firefox web browser - Control tab
Select 'Save/Restore Settings'
From 'Save/Restore Module Settings' tab
select 'Reload'
select 'Module Names' whose settings should be saved (usually all)
select 'Generate new DB key' and make a note of the DB key generated
the DB key will have the format YYYYMMMDD-HH.MM.SS
The DB key will be used to name a directory to store AIDA settings
and is created uid: npg, gid: npgstaff, perms 755. However, its
contents are written uid: nfsnobody, gid nfsnobody. The default
directory (DB Key) permissions do not work - change the permissions
to 777 - before attempting to save data to this directory (DB Key).
launch a terminal and change the permissions of the DB key directory using the command
chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/[YYYYMMMDD-HH.MM.SS]
select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
the directory (DB key)
11) Update BRIKEN Elog with the following information
AIDA
run filename (e.g. R99)
[# data blocks]
start date & time
[stop date & time]
ASIC settings DB key
additional comments
your name
Complete run sheet for written logbook |
|
411
|
Mon Oct 31 02:00:01 2016 |
TD | Monday 31 October | 11.00 DAQ continues OK
DAQ STOP OK
Most recent files R91_568 last accessed 10.59
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses & errors - see attachments 4-7
14.20 Per FEE64 screenshots p+n junction and n+n ohmic strip waveforms - see attachments 8-11
Preamplifier output noise looks OK for all FEE64s *except* nnaida5 & nnaida6 (which
have significantly higher ADC and disc rates than other FEE64s)
21.10 DAQ found stalled
Most recent file R94_99 last accessed 18.35
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
21.16 DAQ start (R96) |
|
410
|
Sun Oct 30 04:37:31 2016 |
TD | Spectrum layouts created | The following spectrum layouts have been created
1 All Rate spectra
2 All HitRate Spectra
3 p+n junction strip 1.8.L spectra
4 n+n ohmic strip 1.8.L spectra
5 p+n junction strip 1.8.H spectra
6 n+n ohmic strip 1.8.H spectra
5 p+n junction strip 1.8.W waveforms
6 n+n ohmic strip 1.8.W waveforms
The layouts can be found in the directory ~/LayOut and should be copied to /tmp
N.B Currently you cannot restore a layout to a spectrum display with > 16 galleries.
If spectrum display uses > 16 galleries, switch spectrum display to 16 galleries
or less before restoring a spectrum layout. |
|
409
|
Sun Oct 30 03:26:55 2016 |
TD | Report - medium - cannot restore layout if currently displaying >16 galleries |
Cannot restore layout if currently displaying >16 galleries
Obtain the error message 'loading Spectrum.tcl > can't read "dbtagx(31,1)": no such element in array' |
|
408
|
Sun Oct 30 01:53:13 2016 |
TD | Sunday 30 October | 10.53 DAQ found stalled
Most recent files R89_102 last accessed 07.17
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
All system wide checks OK *except* Clock status, SYNC errors & SYNC pulses - see attachments 4-8 - unusual
messages/reports
One merger error messages - link 18 - see attachment 9
Restart merger
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
Standard soft reset did not work - one or more FEE64s dropped data transfer #1 link at DAQ GO
Retry - one, more FEE64s indicate error state - run control tab unresponsive
Power cycle FEE64s |
|
407
|
Sat Oct 29 07:26:59 2016 |
TD | AIDA ADC & disc data synchronisation check |
Following analysis of AIDA ADC & disc readout ( see https://elog.ph.ed.ac.uk/AIDA/404 )
changed timestamp resolution from 1us/channel to 20.48us/channel
Offline sort (from disk file)
Attachments 1-3 s1-s32 # ADC data items per FEE64, s100-132 # disc data items per FEE64 (20.48us/channel)
Attachments 4-5 s1000 4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)
Attachments 6-7 MIDASsort terminal session logs for *offline* sort
Online sort (from TapeServer)
Attachment 8 s1-s32 # ADC data items per FEE64, s100-132 # disc data items per FEE64 (20.48us/channel)
Attachments 9-10 s1000 4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)
Attachments 11-12 MIDASsort terminal session logs for *online* sort
Attachment 13 MIDASsort FORTRAN sort routine sync.f
Attachment 14 Spectrum titles for the above sort routine
Note
1) data is discontinuous as a function of timestamp because we are sampling buffers
from the TapeServer rather than sorting all data as above
2) a low rate of timewarps is reported which appears to be an artifact of the buffer
sampling - no timewarps reported by analyses of the corresponding data files
|
|
406
|
Sat Oct 29 03:31:03 2016 |
TD | Saturday 29 October |
11.30 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 1-4
11.36 DSSSD bias & leakage currents OK - see attachments 5-8
FEE64 temperatures OK - see attachment 9
16.46 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 5-8
18.55 DAQ stalled
Most recent file R87_274 last accessed 18.47
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.10 DAQ starts (R89) |
|