 Wed Jun 15 01:43:47 2016, TD, Wednesday 15 June 2016 6x
|
09.19 DAQ stalled AIDA72_87 c. 01.25
SYNC errors since baseline (start of AIDA72) - attachment 1
Power-cycle & reboot
09.23 Detector biases & leakage currents -OK - attachments 2-3
09.38 System-wide checks
All FEE74s pass all tests *except* nnaida3 & nnaida16 fail FADC calibration
All FEE64s but oneshow 1 SYNC error *before* DAQ GO - attachment 4
09.39 AIDA73
ASICs checked OK
FADC disabled
ASIC settings 2016Jun12-09.09.37
SyncCheck OK (EURICA-AIDA)
13.28 DAQ stalled AIDA73_73
All FEE64s pass system-wide clock and ASIC clock tests
SYNC errors since baseline (start of AIDA73) - attachment 5
19.06 DAQ stalled AIDA74_64
All FEE74s pass all tests *except* FADC calibration
SYNC errors since baseline (start of AIDA74) - attachment 6
21.47 AIDA75
ASICs checked OK
FADC disabled
All FEE74s pass all tests *except* FADC calibration
ASIC settings 2016Jun12-09.09.37
LEC/MEC slow comparator 0x40
LEC/MEC fast comparator 0x20
BNC PB-4 Pulser
Frequency 250Hz range
Merge ~600k data items/s (~4600kb/s)
|
|
Thu Jun 16 03:40:43 2016, TD, Thursday 16 June 2016
|
10.40 DAQ stalled AIDA75_30 c. 01.36
SYNC errors since baseline (start of AIDA75) - attachment 1
SYstem-wide checks ASIC clock timestamp and clock status OK for all FEE64s
Power-cycle & reboot
11.14 AIDA76
ASIC settings 2016Jun12-10.10.08
BNC PB-4 amplitude 50,000
11.19 AIDA77
BNC PB-4 amplitude 30,000
11.23 AIDA78
BNC PB-4 amplitude 10,000
11.28 AIDA79
BNC PB-4 amplitude 5,000
All *.L histograms saved c. 11.30 Thursday 16 June 2016
Example spectra - attachment 5
11.34 Water on floor by AIDA NIM bins/forward AIDA stand/EURICA - it would appear
that one or more cluster LN2 cryostat inlets have thawed are or thawing - failed fill
and/or blocked manifold?
Julabo FL11006 water level and flow looks OK
FEE temperatures OK - see attachment 4
Detector bias off
FEE64 power off
All AIDA power off
Source of water was determined to be wheeled air dehumdifier/cooler used to cool crates
in 19" rack to right of AIDA (looking upstream). This had a small tray tank to collect
water which was full to overflowing.
2x 60Co sources (#2530 and CD395) returned to cupboard B3F (between E12 & E17 stairwells)
|
|
Thu Jun 16 08:00:10 2016, TD, AIDA photographs - June 2016 - 1
|
| |
|
Thu Jun 16 08:07:37 2016, TD, AIDA photographs - June 2016 - 2
|
| |
|
Thu Jun 16 08:12:51 2016, TD, AIDA photographs - June 2016 - 3
|
| |
|
Thu Jun 16 08:18:50 2016, TD, AIDA photographs - May 2016
|
| |
|
Sat Jun 18 14:35:31 2016, TD, AIDA photographs - June 2016 - 4
|
| |
|
Fri Jul 8 18:54:47 2016, TD, Analysis July2016/R1 data
|
See https://elog.ph.ed.ac.uk/AIDA/324
Analysis of first and last data file - see attachments 1-2
Rates per FEE64 ~20k data items/s
analyser2.f program - see attachment 3
To compile the analyser2.f program (SL6)
gfortran analyser2.f -o analyser2
analyser2 <filename> to produce summary statistics
analyser2 v <filename> to produce verbose analysis
ADC multiplicity as a function of time (1 channel = 0.1ms) for file R1_114 - see attachment 4
Pulser running at ~270Hz clearly identifiable
Conclusion
DAQ ran overnight without stalling
DAQ did not lose synchronisation
Decrease comparator thresholds to increase rates to experiment-realistic levels ~50k data items/s/FEE64 |
|
Mon Jul 25 08:38:56 2016, TD, Analysis of R4
|
| |
|
Sun Jul 31 09:38:07 2016, TD, Analysis of AIDA75_1
|
Attachment 2 - analysis from *end* of file AIDA75_1
2D spectrum - FEE # versus timestamp (100us/channel)
1D spectrum - projection of timestamp data for FEE # = 6
Conclude data synchronised at beginning and end of AIDA75_1.
Period of pulser data ~36ch = 3.6ms => pulser frequency ~ 280Hz
Attachment 3 - analysis around ts=0x18bacd75c4 +/- 20000000 (dec)
Conclude FEE # 6 stops producing ADC data c. ch 353
Attachment 4 - analysis around ts=0x18cac6144b +/- 20000000 (dec)
Conclude FEE # 6 starts producing ADC data c. ch 28
Spectrum s4100 y-channel calculated as follows
C IF ( ts.GE.z'18bacd75c4' - 20000000
C----67---
C + .AND.
C + ts.LE.z'18bacd75c4' + 20000000 ) THEN
IF ( ts.GE.z'18cac6144b' - 20000000
C----67---
+ .AND.
+ ts.LE.z'18cac6144b' + 20000000 ) THEN
C
C
C Increment ts spectra per FEE64 module
j = MOD( INT( ts*10.0D-09*1.0D+04 ), 4096 )
IF ( j.EQ.0 ) THEN
DO k = 0, 4095
DO l = 1, 31
CALL set2d( 4100, l, k, 0 )
ENDDO
ENDDO
ENDIF
CALL inc2d( 4100, module, j )
ENDIF |
|
Mon Aug 22 10:35:06 2016, TD, Saturday 20 August
|
|
|
Mon Aug 22 10:35:31 2016, TD, Monday 20 August 17x
|
11.00 Restore power to AIDA following Saturday's shutdoen in preparation for the planned maintenance
outage on Sunday.
Restart Julabo FL11006 chiller - set point +20 deg C
Restart AIDA
Temperature +24.3 deg C, DP +15.2 deg C, RH 57.5%
DSSSD #1-6 bias - see attachments 1-2
FEE64 temperatures - attachment 3
System-wide checks - see attachments 4-8
Note SYNC pulses received by *all* slave FEE64s = SYNC pulses sent by master FEE64 (nnaida6)
11.35 R6 starts
slow comparator 0x10
LEC/MEC fast comparator 0x10
All FEE64 discriminator masks -> 0x0
observe very high rates (~400-500kHz/FEE64) from majority ofp+n junction FEE64s
LEC/MEC fast comparator -> 0xff
18.45 DAQ running OK
No merger error messages since change of LEC/MEC fast comparator threshold -> 0xff
Offline analysis of R6_79 confirms that all FEE64s are synchronised
Added correlation trigger input
BNC PB-4 Pulser trigger output -> Ortec SCA + NIM output width 0.5us -> LeCroy 429A -> root MACB trigger
#4 input
21.05 R7 starts
Attachments 9-16 show FEE64 temperatures and results of each system-wide check.
Note the result of the Master Timestamp Control -> Check Clock Lock States
indicating clock state *not* locked
slow comparator 0x10
LEC/MEC fast comparator 0xff
All FEE64 discriminator masks -> 0x0 |
 Mon Aug 22 13:37:30 2016, TD, Monday 20 August
|
> 11.00 Restore power to AIDA following Saturday's shutdoen in preparation for the planned maintenance
> outage on Sunday.
>
> Restart Julabo FL11006 chiller - set point +20 deg C
> Restart AIDA
>
> Temperature +24.3 deg C, DP +15.2 deg C, RH 57.5%
>
> DSSSD #1-6 bias - see attachments 1-2
>
> FEE64 temperatures - attachment 3
>
> System-wide checks - see attachments 4-8
> Note SYNC pulses received by *all* slave FEE64s = SYNC pulses sent by master FEE64 (nnaida6)
>
> 11.35 R6 starts
> slow comparator 0x10
> LEC/MEC fast comparator 0x10
>
> All FEE64 discriminator masks -> 0x0
>
> observe very high rates (~400-500kHz/FEE64) from majority ofp+n junction FEE64s
>
> LEC/MEC fast comparator -> 0xff
>
> 18.45 DAQ running OK
> No merger error messages since change of LEC/MEC fast comparator threshold -> 0xff
> Offline analysis of R6_79 confirms that all FEE64s are synchronised
>
> Added correlation trigger input
> BNC PB-4 Pulser trigger output -> Ortec SCA + NIM output width 0.5us -> LeCroy 429A -> root MACB trigger
> #4 input
>
> 21.05 R7 starts
>
> Attachments 9-16 show FEE64 temperatures and results of each system-wide check.
>
> Note the result of the Master Timestamp Control -> Check Clock Lock States
> indicating clock state *not* locked
>
> slow comparator 0x10
> LEC/MEC fast comparator 0xff
> All FEE64 discriminator masks -> 0x0
The status is OK. I changed the system wide check to reflect the new value but forgot about the individual check.
I will update the software.
Note attachment 5 shows all is well.
Updated version of /MIDAS/TclHttpd/Html/AIDA/MasterTS/MasterTS.tcl installed with correction. |
|
Tue Aug 23 01:49:01 2016, TD, Tuesday 23 August 18x
|
09.50 DAQ found stopped
No error messages merge/tapeserver - see attachments 1-2
Most recent data file R7_107 last written to 08.13 23.8.16
10.00 Electrical power to AIDA is off for reasons unknown (power saving/maintenance/trip)
3-ph circuit to Julabo FL1106 recirculating chiller is unaffected
10.20 Analysis of R7_107 (most recent data file) shows data synchronised OK
11.30 Yoshida-san informs us that the experimental power to F11 is off for annual inspection/maintenance.
Power should be restored this evening, or possibly, sometime tomorrow.
19.00 Power restored
FEE64 power *off* - as expected following power outage
Detector bias *off* - as expected following power outage
Startup AIDA
System wide checks, FEE64 temperatures and DSSSD bias/leakage currents - see attachments 3-11
19.24 R8 starts
slow comparator 0x10
LEC/MEC fast comparator 0xff
ASIC settings 2016Aug20-15.34.53
Discriminator masks -> 0x0
BNC PB-4 Pulser
Amplitude 90,000
x5 attenuator IN
t_r 50ns tau_d 50us
- polarity
INT ref, delay MIN, tail pulse
19.31 AIDA SYNC, PAUSE, RESUME, SYNCtime warp, Good data etc statistics - see attachments 12-18 |
|
Wed Aug 24 03:43:44 2016, TD, Wednesday 24 August 32x
|
11.40 R8 continues
B2F/FF temp +25 deg C, d.p. +14 deg C, RH 51%
DSSSD bias/leakage current OK - see attachments 1-2
FEE64 temperatures OK - see attachment 3
System-wide checks OK - see attachments 4-9
AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6 statistics - see attachments
10-16
Some error messages observed for merger - see attachment 17-18
All related to Q5 (nnaida6)
Analysis of R8_231 shows data synchronised
18.07 R8 stops
At end of run (file R8_310) BNC PB-4 pulser ref switch INT -> EXT to check for possible skew in data
synchronisation
Analysis of R8_310 shows no evidence of skew, i.e. all FEE64s are observed to stop producing pulser events
at same time
19.21 R9 starts
ASIC settings 2016Aug20-15.34.53
Discriminator masks -> 0x0
BNC PB-4 Pulser
Amplitude 90,000
x5 attenuator IN
t_r 50ns tau_d 50us
- polarity
INT ref, delay MIN, tail pulse
BNC PB-4 pulser triggered by EG&G Ortec 462 Start output - frequency ~120Hz - Range 81.92us
From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5
ASIC Control -> Check ASIC Control required after DAQ GO
BNC PB-4 pulser triggered by EG&G Ortec 462 Start output - frequency ~120Hz - Range 81.92us
From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5
System-wide checks OK - see attachments 19-24
AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6 statistics x2 - see attachments
25-32
|
|
Thu Aug 25 02:10:10 2016, TD, Thursday 25 August 21x
|
11.40 R9 continues
ASIC settings 2016Aug20-15.34.53
Discriminator masks -> 0x0
From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5
BNC PB-4 Pulser
Amplitude 90,000
x5 attenuator IN
t_r 50ns tau_d 50us
- polarity
INT ref, delay MIN, tail pulse
External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz
B2F/F11 temp +25.2 deg C, d.p. +14.5 deg C, RH 50.4%
DSSSD bias/leakage current OK - see attachments 1-2
FEE64 temperatures OK - see attachment 3
Some error messages observed for merger - see attachment 4-6
All related to Q5 (nnaida6)
System-wide checks OK - see attachments 7-12
AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6, nnaida5 & nnaida20
statistics - see attachments
13-20
Analysis of R9_213 shows data synchronised - see attachment 21
FEE # versus timestamp (0.1ms / channel)
Note nnaida19 not currently configured |
|
Fri Aug 26 02:10:31 2016, TD, Friday 26 August 28x
|
10.11 R9 continues
ASIC settings 2016Aug20-15.34.53
Discriminator masks -> 0x0
From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5
BNC PB-4 Pulser
Amplitude 90,000
x5 attenuator IN
t_r 50ns tau_d 50us
- polarity
INT ref, delay MIN, tail pulse
External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz
B2F/F11 temp +24.8 deg C, d.p. +12.9 deg C, RH 46.8%
DSSSD bias/leakage current OK - see attachments 1-2
FEE64 temperatures OK - see attachment 3
Some error messages observed for merger - see attachment 4-12
All related to Q5 (nnaida6)
System-wide checks OK - see attachments 13-18
AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6, nnaida5 & nnaida20
statistics - see attachments
19-28
Analysis of R9_561 shows data synchronised - see attachmen
FEE # versus timestamp (0.1ms / channel)
Note nnaida19 not currently configured
10.49 R9 stops |
|
Mon Aug 29 06:57:32 2016, TD, Monday 29 August 7x
|
Per the instructions provided by https://elog.ph.ed.ac.uk/AIDA/331 added Servisol Heat Sink Compound
to each of the 24x FEE64s currently installed (plus spare FEE module stored in the B3F AIDA storage
area).
Before adding the Servisol Heat Sink Compound - https://elog.ph.ed.ac.uk/AIDA/349 (attachment 3)
After adding the Servisol Heat Sink Compound - see attachment 1
With exception of nnaida8 & nnaida10 Virtex temperatures now ~ +44-55 deg C cf. ~ +55-70 deg C
nnaida6 shows an increase in temperature for the PSU from ~ +35 deg C to ~ +40 deg C
All system-wide checks OK
Rate hit patterns for nnaida1-nnaida24 - see attachments 2-3
Should show uniform rate ~130Hz for all channels
Observe some channels are missing
N.B. 1x AIDA adaptor PCB rev D replaced by spare due to (single) bent ERNI pin
All ERNI connectors of all AIDA adaptor PCBs checked before re-installation of FEE modules
- it is possible that there are other AIDA adaptor PCBs with bent pins
- the system is currently fairly noisy and this could be an alternative explanation of missing channels
16.55 R10
FEE64 configuration includes nnaida19
ASIC settings 2016Aug20-15.34.53
Discriminator masks -> 0x0
From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5
BNC PB-4 Pulser
Amplitude 90,000
x5 attenuator IN
t_r 50ns tau_d 50us
- polarity
INT ref, delay MIN, tail pulse
External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz
B2F/F11 temp +25.0 deg C, d.p. +14.3 deg C, RH 51.6% (Julabo FL11006 set point +20 deg C)
FEE64 temperatures - attachment 6 |
|
Mon Aug 29 08:53:55 2016, TD, nnaida19 restored
|
nnaida19 restored to configuration - /MIDAS/Aida/tclhttpd/aidas1/startup.tcl - see attachment 1
Latest version of firmware loaded - /MIDAS/Aida/FEE_GF_Feb16_21.bin
System power-cycled
ASIC settings 2016Aug20-15.34.53
slow comparator 0x20
LEC/MEC fast comparator 0xff
discriminator mask -> 0x0
All system-wide checks OK *except* nnaida19 fails ASIC clock timestamp synchronisation |
|
Tue Aug 30 08:26:33 2016, TD, Tuesday 30 August 28x
|
16.18 R10 stops
ASIC settings 2016Aug20-15.34.53
Discriminator masks -> 0x0
From Correlation Status & Control tab change Trigger delay 0x5
BNC PB-4 Pulser
Amplitude 90,000
x5 attenuator IN
t_r 50ns tau_d 50us
- polarity
INT ref, delay MIN, tail pulse
External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz
B2F/F11 temp +25.1 deg C, d.p. +14.1 deg C, RH 51.6%
DSSSD bias/leakage current OK - see attachments 1-2
FEE64 temperatures OK - see attachment 3
Some error messages observed for merger - see attachment 4-10
All related to Q5 (nnaida6)
System-wide checks OK - see attachments 11-16
AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6, nnaida5 & nnaida20
statistics - see attachments
17-25
Analysis of R10_272 shows data synchronised - see attachment 26
FEE # versus timestamp (0.1ms / channel)
nnaida1 - nnaid24 (stat) hit patterns - see attachments 27-28
aidas1> ./analyser2 /TapeData/Aug2016/R10_272
*** GREAT format 3.2.0 analyser - TD - May 2014
*** FEE64 discriminator: block: 15670 ptr: 2407 data: 0x81640080 module: 1 information type: 6 information
field: 0x00040080 ts: 0x000007BD800000A0 OLD: ts: 0x000007BD9000009E tag: 2 type: 6
*** ERROR: READ I/O error: 5002
blocks: 24294
ADC data format: 57829496 ( 296793.6 Hz)
Other data format: 140380504 ( 720463.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 59 ( 0.3 Hz)
RESUME: 59 ( 0.3 Hz)
SYNC100: 1783050 ( 9151.0 Hz)
FEE64 disc: 138570073 ( 711171.8 Hz)
MBS info: 27263 ( 139.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 9410 ( 48.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 1 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 194.848 s
FEE module #: 1 elapsed dead time 0.210 s
FEE module #: 2 elapsed dead time 0.261 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.206 s
FEE module #: 6 elapsed dead time 0.857 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.000 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.055 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.027 s
FEE module #: 18 elapsed dead time 0.000 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.013 s
FEE module #: 23 elapsed dead time 0.000 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 11.883s ( 2044.465 blocks/s, 127.779 Mb/s) |
|