09.19 DAQ stalled AIDA72_87 c. 01.25

      SYNC errors since baseline (start of AIDA72) - attachment 1

      Power-cycle & reboot

09.23 Detector biases & leakage currents -OK - attachments 2-3

09.38 System-wide checks 
      All FEE74s pass all tests *except* nnaida3 & nnaida16 fail FADC calibration
      All FEE64s but oneshow 1 SYNC error *before* DAQ GO - attachment 4

09.39 AIDA73

      ASICs checked OK
      FADC disabled
      ASIC settings 2016Jun12-09.09.37
      SyncCheck OK (EURICA-AIDA)

13.28 DAQ stalled AIDA73_73

      All FEE64s pass system-wide clock and ASIC clock tests
      SYNC errors since baseline (start of AIDA73) - attachment 5

19.06 DAQ stalled AIDA74_64

      All FEE74s pass all tests *except* FADC calibration
      SYNC errors since baseline (start of AIDA74) - attachment 6

21.47 AIDA75

      ASICs checked OK
      FADC disabled
      All FEE74s pass all tests *except* FADC calibration
      ASIC settings 2016Jun12-09.09.37
       LEC/MEC slow comparator 0x40
       LEC/MEC fast comparator 0x20
      BNC PB-4 Pulser 
       Frequency 250Hz range
      Merge ~600k data items/s (~4600kb/s)



   

10.40 DAQ stalled AIDA75_30 c. 01.36

      SYNC errors since baseline (start of AIDA75) - attachment 1
      SYstem-wide checks ASIC clock timestamp and clock status OK for all FEE64s

      Power-cycle & reboot

11.14 AIDA76 

      ASIC settings 2016Jun12-10.10.08
      BNC PB-4 amplitude 50,000

11.19 AIDA77

      BNC PB-4 amplitude 30,000

11.23 AIDA78

      BNC PB-4 amplitude 10,000

11.28 AIDA79

      BNC PB-4 amplitude 5,000

      All *.L histograms saved c. 11.30 Thursday 16 June 2016
      Example spectra - attachment 5

11.34 Water on floor by AIDA NIM bins/forward AIDA stand/EURICA - it would appear
      that one or more cluster LN2 cryostat inlets have thawed are or thawing - failed fill
      and/or blocked manifold?

      Julabo FL11006 water level and flow looks OK
      FEE temperatures OK - see attachment 4

      Detector bias off
      FEE64 power off
      All AIDA power off

      Source of water was determined to be wheeled air dehumdifier/cooler used to cool crates
      in 19" rack to right of AIDA (looking upstream). This had a small tray tank to collect
      water which was full to overflowing.

      2x 60Co sources (#2530 and CD395) returned to cupboard B3F (between E12 & E17 stairwells)
 

See https://elog.ph.ed.ac.uk/AIDA/324

Analysis of first and last data file - see attachments 1-2
Rates per FEE64 ~20k data items/s

analyser2.f program - see attachment 3

To compile the analyser2.f program (SL6)

gfortran analyser2.f -o analyser2

analyser2 <filename> to produce summary statistics
analyser2 v <filename> to produce verbose analysis

ADC multiplicity as a function of time (1 channel = 0.1ms) for file R1_114 - see attachment 4
Pulser running at ~270Hz clearly identifiable

Conclusion
DAQ ran overnight without stalling
DAQ did not lose synchronisation

Decrease comparator thresholds to increase rates to experiment-realistic levels ~50k data items/s/FEE64

Attachment 2 - analysis from *end* of file AIDA75_1

2D spectrum - FEE # versus timestamp (100us/channel)

1D spectrum - projection of timestamp data for FEE # = 6

Conclude data synchronised at beginning and end of AIDA75_1. 
Period of pulser data ~36ch = 3.6ms => pulser frequency ~ 280Hz

Attachment 3 - analysis around ts=0x18bacd75c4 +/- 20000000 (dec)

Conclude FEE # 6 stops producing ADC data c. ch 353

Attachment 4 - analysis around ts=0x18cac6144b +/- 20000000 (dec)

Conclude FEE # 6 starts producing ADC data c. ch 28

Spectrum s4100 y-channel calculated as follows

C        IF ( ts.GE.z'18bacd75c4' - 20000000
C----67---
C     +       .AND.
C     +       ts.LE.z'18bacd75c4' + 20000000 ) THEN
        IF ( ts.GE.z'18cac6144b' - 20000000
C----67---
     +       .AND.
     +       ts.LE.z'18cac6144b' + 20000000 ) THEN
C
C
C       Increment ts spectra per FEE64 module

        j = MOD( INT( ts*10.0D-09*1.0D+04 ), 4096 )
        IF ( j.EQ.0 ) THEN
         DO k = 0, 4095
          DO l = 1, 31
           CALL set2d( 4100, l, k, 0 )
          ENDDO
         ENDDO
        ENDIF
        CALL inc2d( 4100, module, j )

        ENDIF

11.00 Restore power to AIDA following Saturday's shutdoen in preparation for the planned maintenance
      outage on Sunday.

      Restart Julabo FL11006 chiller - set point +20 deg C
      Restart AIDA

      Temperature +24.3 deg C, DP +15.2 deg C, RH 57.5%

      DSSSD #1-6 bias - see attachments 1-2
     
      FEE64 temperatures - attachment 3

      System-wide checks - see attachments 4-8
      Note SYNC pulses received by *all* slave FEE64s = SYNC pulses sent by master FEE64 (nnaida6)

11.35 R6 starts
      slow comparator 0x10
      LEC/MEC fast comparator 0x10

      All FEE64 discriminator masks -> 0x0

      observe very high rates (~400-500kHz/FEE64) from majority ofp+n junction FEE64s

      LEC/MEC fast comparator -> 0xff

18.45 DAQ running OK
      No merger error messages since change of LEC/MEC fast comparator threshold -> 0xff
      Offline analysis of R6_79 confirms that all FEE64s are synchronised

      Added correlation trigger input
      BNC PB-4 Pulser trigger output -> Ortec SCA + NIM output width 0.5us -> LeCroy 429A -> root MACB trigger
#4 input

21.05 R7 starts

      Attachments 9-16 show FEE64 temperatures and results of each system-wide check. 

      Note the result of the Master Timestamp Control -> Check Clock Lock States 
      indicating clock state *not* locked

      slow comparator 0x10
      LEC/MEC fast comparator 0xff
      All FEE64 discriminator masks -> 0x0

> 11.00 Restore power to AIDA following Saturday's shutdoen in preparation for the planned maintenance
>       outage on Sunday.
> 
>       Restart Julabo FL11006 chiller - set point +20 deg C
>       Restart AIDA
> 
>       Temperature +24.3 deg C, DP +15.2 deg C, RH 57.5%
> 
>       DSSSD #1-6 bias - see attachments 1-2
>      
>       FEE64 temperatures - attachment 3
> 
>       System-wide checks - see attachments 4-8
>       Note SYNC pulses received by *all* slave FEE64s = SYNC pulses sent by master FEE64 (nnaida6)
> 
> 11.35 R6 starts
>       slow comparator 0x10
>       LEC/MEC fast comparator 0x10
> 
>       All FEE64 discriminator masks -> 0x0
> 
>       observe very high rates (~400-500kHz/FEE64) from majority ofp+n junction FEE64s
> 
>       LEC/MEC fast comparator -> 0xff
> 
> 18.45 DAQ running OK
>       No merger error messages since change of LEC/MEC fast comparator threshold -> 0xff
>       Offline analysis of R6_79 confirms that all FEE64s are synchronised
> 
>       Added correlation trigger input
>       BNC PB-4 Pulser trigger output -> Ortec SCA + NIM output width 0.5us -> LeCroy 429A -> root MACB trigger
> #4 input
> 
> 21.05 R7 starts
> 
>       Attachments 9-16 show FEE64 temperatures and results of each system-wide check. 
> 
>       Note the result of the Master Timestamp Control -> Check Clock Lock States 
>       indicating clock state *not* locked
> 
>       slow comparator 0x10
>       LEC/MEC fast comparator 0xff
>       All FEE64 discriminator masks -> 0x0

The status is OK. I changed the system wide check to reflect the new value but forgot about the individual check.
I will update the software.
Note attachment 5 shows all is well.

Updated version of /MIDAS/TclHttpd/Html/AIDA/MasterTS/MasterTS.tcl installed with correction.

09.50 DAQ found stopped
      No error messages merge/tapeserver - see attachments 1-2
      Most recent data file R7_107 last written to 08.13 23.8.16

10.00 Electrical power to AIDA is off for reasons unknown (power saving/maintenance/trip)
      3-ph circuit to Julabo FL1106 recirculating chiller is unaffected

10.20 Analysis of R7_107 (most recent data file) shows data synchronised OK 

11.30 Yoshida-san informs us that the experimental power to F11 is off for annual inspection/maintenance.
      Power should be restored this evening, or possibly, sometime tomorrow.

19.00 Power restored

      FEE64 power *off* - as expected following power outage
      Detector bias *off* - as expected following power outage

      Startup AIDA
      System wide checks, FEE64 temperatures and DSSSD bias/leakage currents - see attachments 3-11

19.24 R8 starts

      slow comparator 0x10
      LEC/MEC fast comparator 0xff

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse

19.31 AIDA SYNC, PAUSE, RESUME, SYNCtime warp, Good data etc statistics - see attachments 12-18

11.40 R8 continues

      B2F/FF temp +25 deg C, d.p. +14 deg C, RH 51%

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3
      System-wide checks OK - see attachments 4-9

      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6 statistics - see attachments
10-16

      Some error messages observed for merger - see attachment 17-18
      All related to Q5 (nnaida6)

      Analysis of R8_231 shows data synchronised 

18.07 R8 stops

      At end of run (file R8_310) BNC PB-4 pulser ref switch INT -> EXT to check for possible skew in data
synchronisation

      Analysis of R8_310 shows no evidence of skew, i.e. all FEE64s are observed to stop producing pulser events
at same time

19.21 R9 starts

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse
      BNC PB-4 pulser triggered by EG&G Ortec 462 Start output - frequency ~120Hz - Range 81.92us
      From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5

      ASIC Control -> Check ASIC Control required after DAQ GO

      BNC PB-4 pulser triggered by EG&G Ortec 462 Start output - frequency ~120Hz - Range 81.92us
      From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5

      System-wide checks OK - see attachments 19-24
      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6 statistics x2 - see attachments
25-32
      

11.40 R9 continues

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0
      From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse
      External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz

      B2F/F11 temp +25.2 deg C, d.p. +14.5 deg C, RH 50.4%

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      Some error messages observed for merger - see attachment 4-6
      All related to Q5 (nnaida6)

      System-wide checks OK - see attachments 7-12

      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6, nnaida5 & nnaida20
statistics - see attachments
13-20

      Analysis of R9_213 shows data synchronised - see attachment 21

      FEE # versus timestamp (0.1ms / channel)
      Note nnaida19 not currently configured

10.11 R9 continues

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0
      From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse
      External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz

      B2F/F11 temp +24.8 deg C, d.p. +12.9 deg C, RH 46.8%

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      Some error messages observed for merger - see attachment 4-12
      All related to Q5 (nnaida6)

      System-wide checks OK - see attachments 13-18

      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6, nnaida5 & nnaida20
statistics - see attachments
19-28

      Analysis of R9_561 shows data synchronised - see attachmen

      FEE # versus timestamp (0.1ms / channel)
      Note nnaida19 not currently configured

10.49 R9 stops

Per the instructions provided by https://elog.ph.ed.ac.uk/AIDA/331 added Servisol Heat Sink Compound
to each of the 24x FEE64s currently installed (plus spare FEE module stored in the B3F AIDA storage
area).

Before adding the Servisol Heat Sink Compound - https://elog.ph.ed.ac.uk/AIDA/349 (attachment 3)

After adding the Servisol Heat Sink Compound - see attachment 1

With exception of nnaida8 & nnaida10 Virtex temperatures now ~ +44-55 deg C cf. ~ +55-70 deg C 

nnaida6 shows an increase in temperature for the PSU from ~ +35 deg C to ~ +40 deg C

All system-wide checks OK

Rate hit patterns for nnaida1-nnaida24 - see attachments 2-3
Should show uniform rate ~130Hz for all channels
Observe some channels are missing

N.B. 1x AIDA adaptor PCB rev D replaced by spare due to (single) bent ERNI pin
     All ERNI connectors of all AIDA adaptor PCBs checked before re-installation of FEE modules
     - it is possible that there are other AIDA adaptor PCBs with bent pins
     - the system is currently fairly noisy and this could be an alternative explanation of missing channels
 

16.55 R10

      FEE64 configuration includes nnaida19

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0
      From Correlation Status & Control tab change Trigger delay from 0x180 to 0x5

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse
      External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz

      B2F/F11 temp +25.0 deg C, d.p. +14.3 deg C, RH 51.6% (Julabo FL11006 set point +20 deg C)

      FEE64 temperatures - attachment 6

nnaida19 restored to configuration - /MIDAS/Aida/tclhttpd/aidas1/startup.tcl - see attachment 1

Latest version of firmware loaded - /MIDAS/Aida/FEE_GF_Feb16_21.bin

System power-cycled

ASIC settings 2016Aug20-15.34.53

slow comparator 0x20
LEC/MEC fast comparator 0xff

discriminator mask -> 0x0

All system-wide checks OK *except* nnaida19 fails ASIC clock timestamp synchronisation

16.18 R10 stops

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0
      From Correlation Status & Control tab change Trigger delay 0x5

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse
      External (random) trigger from EG&G Ortec 462 Start output (Range 81.92us) - frequency c. 120Hz

      B2F/F11 temp +25.1 deg C, d.p. +14.1 deg C, RH 51.6%

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      Some error messages observed for merger - see attachment 4-10
      All related to Q5 (nnaida6)

      System-wide checks OK - see attachments 11-16

      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, nnaida6, nnaida5 & nnaida20
statistics - see attachments
17-25

      Analysis of R10_272 shows data synchronised - see attachment 26

      FEE # versus timestamp (0.1ms / channel)

      nnaida1 - nnaid24 (stat) hit patterns - see attachments 27-28

aidas1> ./analyser2 /TapeData/Aug2016/R10_272
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** FEE64 discriminator: block: 15670 ptr:  2407 data: 0x81640080 module:  1 information type:  6 information
field: 0x00040080 ts: 0x000007BD800000A0 OLD: ts: 0x000007BD9000009E tag: 2 type: 6
 *** ERROR: READ I/O error:       5002
                   blocks:      24294
          ADC data format:   57829496 (  296793.6 Hz)
        Other data format:  140380504 (  720463.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         59 (       0.3 Hz)
                               RESUME:         59 (       0.3 Hz)
                              SYNC100:    1783050 (    9151.0 Hz)
                           FEE64 disc:  138570073 (  711171.8 Hz)
                             MBS info:      27263 (     139.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       9410 (      48.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      194.848 s
 FEE module #:   1 elapsed dead time       0.210 s
 FEE module #:   2 elapsed dead time       0.261 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.206 s
 FEE module #:   6 elapsed dead time       0.857 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.000 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.055 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.027 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.013 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   11.883s ( 2044.465 blocks/s, 127.779 Mb/s)