AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 18 of 46  ELOG logo
ID Date Author Subject
  592   Wed May 24 03:20:35 2017 CB, TDWednesday 24 May
11.00 Merger found stopped at 03:27 - current file R24_7
      Restarted Merge. Stopped Run control. Run control stopped, got error message - see attachment 1
      nnaida 1-16 show error, nnaida 17-24 show stopped - see attachment 2
      No new panics in Raspberry Pi console monitor
      
13.10 Found FEE64 #1-16 power off - fuse failed in section #1 of USB-controlled ac mains relay - this fuse was
replaced yesterday

13.15 Closed file May2017/R24
      Closed DAQ, Merger & Tapeserver browsers & terminated terminal servers

13.22 Replaced 250V/16A/fast fuse *and* carrier in section #1 of USB-controlled ac mains relay - when fuse fails
there is some solder leaks from base of fuse into carrier 

13.25 Switch on relays # 1-8 at ~10s intervals
      Count 24 authenticated mount requests 13.26.02 to 13.27.30 - see attachment 5

13.31 E17/F11 ambient temperature +24.4 deg C, RH 40.0%, d.p. +9.9 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 65%)
      Detector biases & leakage currents OK - see attachments 3 & 4

13.39 Checked Pi-monitor - no further panics detected in FEE64 system logs

13.54 Turned on DAQ.
      FEE temperatures OK - see attach 6
      System wide checks ok *except* nnaida1 clock status (status 6) and nnaida1,7,8,9,13,21,24
      Manually recalibrated all *except* nnaida21

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x64
      LEC/MEC fast comparator 0x20

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Current file R26

      ttyUSB0 attached - 7

14.04 good events statistics - see attachment 8
  590   Tue May 23 03:01:52 2017 CB, TD, NN, OHTuesday 23 May
11.02 DAQ continues OK 
      Current file May2017/R22_192

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida7, 9, 13, 22

11.05 E17/F11 ambient temperature +23.9 deg C, RH 41.3%, d.p. +10.5 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)
      Voltage on BRIKEN 3He tubes now ON

11.07 DAQ stops OK
      Current file May2017/R22_193
      Deleted R22*

      Loaded new firmware to test ADC calibration clock issues - attachment 6
      Firmware loaded - attachment 7
      Restarted Firefox - temperatures still look fine - see attachment 7
      System wide checks - nnaida1 clock failed, other clocks OK - attachment 8
                           ADC calibration failed for nnaida1, 3, 21 - attachment 9
                           Other system wide checks OK
      Statistics - few SYNC pulses received by nnaida 9 and 12. See attachement 10


      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

13.10 DAQ started
      File May2017/R23
      Run Control GO - Tape server GO - Merger Unpause - Merger data rate 0, but Tapeserver sees events and
Merger links are green - see attachment 11
      Start ASIC data readout - Merger sees data rate of ~800 000 items/s (OK!)
      FPGA clock monitor page (http://localhost:8015/AIDA/MONLOCK/MONLOCK.tml) screenshot attached (12)

      Ras Pi system console logs for USB0 attached - see attachment 13

18.00 Merger found stopped. We were unable to stop the DAQ. Powered off FEEs. Power on, only eight were supplied
power. 
      Blown fuse in USB bias controller - replaced.
      Power ON the FEEs. All 24 FEEs mounted.
       
      Current file May2017/R23_27 - stopped at 15:27

18.31 nnaida20 rebooted on its own
      
      
18.37 E17/F11 ambient temperature +23.7 deg C, RH 39.8%, d.p. +9.5 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 70%)

18.41 Restarted DAQ 
      
      Detector biases; leakage currents OK - see attachment 14
      System wide checks - OK *except* several modules failed ADC calibration - see attachment 15
      FPGA clock monitor screenshot attached (16)

18.51 Stopped DAQ. Flashed previous firmware on FEEs (FEE_GF_Feb16_24.bin)
      Power cycled. All 24 modules mounted OK.
      nnaida22 rebooted on its own
      Time out Setting up the DAQ on nnaida2 and others. Power cycled again.
      nnaida17 rebooted on its own.
      Time out Setting up the DAQ on nnaida1. Power cycled again.
      
19.56 DAQ finally started.
      Current run R24

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x64
      LEC/MEC fast comparator 0x20

      *all* system wide checks OK *except* some ADC calibrations and clock on nnaida1
      Temperatures OK
      Statistics OK
      
      Logs for nnaida 17,20,22 attached

      
  589   Mon May 22 10:29:11 2017 TDFEE64 system console to Raspberry pi /dev/ttyUSB mapping
ttyUSB0 =>  last host=nnaida24, => last addr=10.1.2.24, ; #panics => 0 ; Not Finished 
ttyUSB1 =>  last host=nnaida23, => last addr=10.1.2.23, ; #panics => 0 ; Not Finished 
ttyUSB10 =>  last host=nnaida18, => last addr=10.1.1.18, ; #panics => 0 ; Not Finished 
ttyUSB11 =>  last host=nnaida17, => last addr=10.1.1.17, ; #panics => 0 ; Not Finished 
ttyUSB12 =>  last host=nnaida4, => last addr=10.1.2.4, ; #panics => 0 ; Not Finished 
ttyUSB13 =>  last host=nnaida3, => last addr=10.1.2.3, ; #panics => 0 ; Not Finished 
ttyUSB14 =>  last host=nnaida12, => last addr=10.1.2.12, ; #panics => 0 ; Not Finished 
ttyUSB15 =>  last host=nnaida11, => last addr=10.1.2.11, ; #panics => 0 ; Not Finished 
ttyUSB16 =>  last host=nnaida20, => last addr=10.1.2.20, ; #panics => 0 ; Not Finished 
ttyUSB17 =>  last host=nnaida19, => last addr=10.1.2.19, ; #panics => 0 ; Not Finished 
ttyUSB18 =>  last host=nnaida6, => last addr=10.1.1.6, ; #panics => 0 ; Not Finished 
ttyUSB19 =>  last host=nnaida5, => last addr=10.1.1.5, ; #panics => 0 ; Not Finished 
ttyUSB2 =>  last host=nnaida16, => last addr=10.1.2.16, ; #panics => 0 ; Not Finished 
ttyUSB20 =>  last host=nnaida14, => last addr=10.1.1.14, ; #panics => 0 ; Not Finished 
ttyUSB21 =>  last host=nnaida13, => last addr=10.1.1.13, ; #panics => 0 ; Not Finished 
ttyUSB22 =>  last host=nnaida22, => last addr=10.1.1.22, ; #panics => 0 ; Not Finished 
ttyUSB23 =>  last host=nnaida21, => last addr=10.1.1.21, ; #panics => 0 ; Not Finished 
ttyUSB3 =>  last host=nnaida15, => last addr=10.1.2.15, ; #panics => 0 ; Not Finished 
ttyUSB4 =>  last host=nnaida8, => last addr=10.1.2.8, ; #panics => 0 ; Not Finished 
ttyUSB5 =>  last host=nnaida7, => last addr=10.1.2.7, ; #panics => 0 ; Not Finished 
ttyUSB6 =>  last host=nnaida2, => last addr=10.1.1.2, ; #panics => 0 ; Not Finished 
ttyUSB7 =>  last host=nnaida1, => last addr=10.1.1.1, ; #panics => 0 ; Not Finished 
ttyUSB8 =>  last host=nnaida10, => last addr=10.1.1.10, ; #panics => 0 ; Not Finished 
ttyUSB9 =>  last host=nnaida9, => last addr=10.1.1.9, ; #panics => 0 ; Not Finished 
  588   Mon May 22 10:21:29 2017 PatrickConsole mapping to FEE64 please

Please attach the list of ttyUSB to FEE64 console connection .

 

  586   Mon May 22 09:12:56 2017 PatrickConsoles reporting Options not available.... defaults operational ?

I have noticed from the console logs reports concerning an inability to open the Options CONTENTS files for the FEE64s and hence failure to load the Options.

I have checked with similar files on nnsa here in Daresbury and they are set for permissions "-rwxrwxrwx.  1 npg npgstaff 1115 May  4 11:42 CONTENTS"

Please could you alter the permissions of all the Options files to agree with this and re-try the SETUP.

  585   Mon May 22 08:53:42 2017 CB, TD, PatrickMonday 22 May
> Concerning nnaida7 and the failure of calibration..... have you tried to calibrate it using the FADC Align  & Control ?
> If it still fails then there is a question about the quality of the clock signal being received by this module.
> I will log through and look at the console log for any anomolies but please try and calibrate.
> Let me know if you wish me to take remote control.

Yes - there have been multiple attempts to re-calibrate nnaida7 ADCs

Note *all* ADCs were calibrated before the most recent FEE64 power cycle.

It is not unusual for ~1-3 FEE64s to fail ADC calibration tests following
a power cycle - there is no obvious pattern to which ADCs fail or which (if
any) will be successfully re-calibrated.
  584   Mon May 22 08:43:10 2017 CB, TD, PatrickMonday 22 May
> 10.35 DAQ continues OK 
>       Current file May2017/R19_253
> 
>       ASIC settings 2017Mar27-16.12.39 *except*
>       slow comparator 0x0a
>       LEC/MEC fast comparator 0xff
> 
>       Detector biases; leakage currents OK - see attachment 1
>       FEE64 Temperatures OK - see attachment 2
>       good events; statistics OK - see attachment 3
>       No warnings/errors reported by merger since DAQ GO - see attachment 4
> 
>       *all* system wide checks OK *except* ADC calibration of nnaida7
> 
> 10.38 E17/F11 ambient temperature +24.7 deg C, RH 40.7%, d.p. +10.7 deg C
>       Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)
> 
> 10.39 DAQ stops OK
>       Current file May2017/R19_253
>       Deleted R19*
> 
>       ASIC settings 2017Mar27-16.12.39 *except*
>       slow comparator 0x0a
>       LEC/MEC fast comparator 0xff
> 
>       BNC PB-4 settings 
>       Amplitude 90,000
>       x5 attenuator IN
>       t_r 50ns, tau_d 50us
>       polarity -
>       rate 250Hz range
>       delay MIN, tail pulse
> 
>       Check ASIC Control x2
>       Zero all spectra
> 
> 10.47 DAQ starts
>       File May2017/R20
>       Ras Pi system console logs for USB0 attached - see attachment 5
>       Analysed R20_0. All appears to be working fine. Highest dead time ~0.3%  - see attachment 6
>       Off-line analysis of R20_0 confirms ADC data and discriminator are synchronised - see attachments 7,8,9

Concerning nnaida7 and the failure of calibration..... have you tried to calibrate it using the FADC Align  & Control ?
If it still fails then there is a question about the quality of the clock signal being received by this module.
I will log through and look at the console log for any anomolies but please try and calibrate.
Let me know if you wish me to take remote control.
  583   Mon May 22 08:29:26 2017 TDOffline analysis R25_12 to R25_22
Offline analysis of AIDA data files R25_12 to R25_22 using MIDASsort

Note

- no calibration (raw ADC data only)
- no thresholds 
- no clustering
- events delimited by > 2us time difference between successive ADC data words 
- HEC = high energy channel = 20GeV FSR range for high energy implants
- LEC = low energy channel = 20MeV FSR range for decays (and pulser)

Attachment 1-4
--------------
s2000-s2005 per DSSSD LEC m_lec_p versus m_lec_n

            following spectra require 0 < m_lec_p < 8 and 0 < m_lec_n < 8
            all combinations of x & y plotted

s2010-s2015 per DSSSD LEC x versus y
s2020-s2025 per DSSSD LEC E_p versus E_n        (20keV/ch nominal)
s2030-s2035 per DSSSD LEC-LEC time              (2.56us/channel)

Attachment 5-8
---------------
s2000-s2005 per DSSSD LEC m_hec_p versus m_hec_n

            following spectra require 0 < m_hec_p < 8 and 0 < m_hec_n < 8
            all combinations of x & y plotted

s2010-s2015 per DSSSD HEC x versus y           
s2020-s2025 per DSSSD HEC E_p versus E_n       (20MeV/channel nominal)
s2030-s2035 per DSSSD HEC-HEC time             (2.56us/channel)

Attachment 9-11
---------------
s2200-s2205 per pixel, per DSSSD HEC-LEC time  (2.56us/channel)

Attachment 12-14
----------------
s2210-s2215 per pixel, per DSSSD HEC-LEC time  (163.840us/channel)

Attachments 15-17
-----------------
s2300-s2305 per DSSSD HEC Energy versus HEC-LEC time (20MeV/channel, 163.840us/channel)
s2400-s2405 per DSSSD LEC Energy versus HEC-LEC time (20keV/channel, 163.840us/channel)
s2500-s2505 per DSSSD x & y strips versus HEC-LEC time (163.840us/channel)

Attachment 18
-------------
Projection s2503 y-strips 0-60 -> HEC-LEC time 163.804us/channel
  582   Mon May 22 02:34:51 2017 CB, TDMonday 22 May
10.35 DAQ continues OK 
      Current file May2017/R19_253

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida7

10.38 E17/F11 ambient temperature +24.7 deg C, RH 40.7%, d.p. +10.7 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

10.39 DAQ stops OK
      Current file May2017/R19_253
      Deleted R19*

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control x2
      Zero all spectra

10.47 DAQ starts
      File May2017/R20
      Ras Pi system console logs for USB0 attached - see attachment 5
      Analysed R20_0. All appears to be working fine. Highest dead time ~0.3%  - see attachment 6
      Off-line analysis of R20_0 confirms ADC data and discriminator are synchronised - see attachments 7,8,9

16.55 DAQ continues OK 
      Current file May2017/R20_71

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK - see attachment 10
      FEE64 Temperatures OK - see attachment 12
      good events; statistics OK - see attachment 13
      No warnings/errors reported by merger since DAQ GO - see attachment 14

      *all* system wide checks OK *except* ADC calibration of nnaida7

17.02 E17/F11 ambient temperature +24.0 deg C, RH 40.3%, d.p. +9.7 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

17.03 DAQ stops OK
      Current file May2017/R20_73
      Deleted R20*

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control x2
      Zero all spectra

17.08 DAQ starts
      File May2017/R21
      Raspberry Pi system console logs for USB0 attached - see attachment 15

17.50 Changed permissions of options files /MIDAS/DB/EXPERIMENTS/Options/nnaida<n>/CONTENTS 
      from mode 644 to mode 777

18.09 DAQ stops OK
      Current file May2017/R21_11
      Deleted R21*

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      DAQ RESET
      DAQ SETUP

      *all* system wide checks OK *except* nnaida7, 9, 13 and 22 ADC calibration failed

      Check ASIC Control x2

17.08 DAQ starts
      File May2017/R22
      Raspberry Pi system console logs for USB0 attached - see attachment 16
  581   Sun May 21 10:58:01 2017 TDJulabo FL11006 recirculating chiller refilled
18.55 Julabo FL11006 recirculating chiller refilled to "H" marker of the filling level indicator.

      See http://www2.ph.ed.ac.uk/~td/AIDA/Information/Julabo%20FL11006.pdf
      
      To refill from 60% L-H to 100% L-H required c. 4 litres of water.
      Following refill there is a trickle of water from the overflow connector which is being 
      collected by hose and bucket - this flow should cease when the fluid level falls below
      the overflow level.
      
19.20 FEE64 temperatures OK - see attachment 1
  580   Sun May 21 02:54:22 2017 CB, TD, OHSunday 21 May
10.54 DAQ continues OK 
      Current file May2017/R18_208

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida7

10.58 E17/F11 ambient temperature +24.6 deg C, RH 40.5%, d.p. +10.3 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 50%)

11.00 DAQ stops OK
      Current file May2017/R18_208
      Deleted R17*

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control x2
      Zero all spectra

11.16 DAQ starts
      File May2017/R19
      Ras Pi system console logs for USB0 attached - see attachment 5

18.26 DAQ continues OK 
      Current file May2017/R19_77

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases & leakage currents OK - see attachment 6 & 7
      FEE64 Temperatures OK - see attachment 8
      good events statistics OK - see attachment 9
      No warnings/errors reported by merger since DAQ GO - see attachment 10

      *all* system wide checks OK *except* ADC calibration of nnaida7

18.30 E17/F11 ambient temperature +24.7 deg C, RH 38.9%, d.p. +9.8 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 60%)
  579   Sat May 20 02:39:06 2017 CB, OHSaturday 20 May
10.35 DAQ continues OK 
      Current file May2017/R16

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida7

10.45 E17/F11 ambient temperature +24.8 deg C, RH 39.9%, d.p. +10.0 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 50%)

11.00 DAQ stops OK
      Current file May2017/R16

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control x2
      Zero all spectra

11.10 DAQ starts
      File May2017/R17

16.30 DAQ continues OK 
      Current file May2017/R17

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK - see attachment 5
      FEE64 Temperatures OK - see attachment 6
      good events; statistics OK - see attachment 7
      No warnings/errors reported by merger since DAQ GO - see attachment 8

      *all* system wide checks OK *except* ADC calibration of nnaida7

16.45 E17/F11 ambient temperature +24.3 deg C, RH 39.1%, d.p. +9.5 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 50%)

16.50 DAQ stops OK
      Current file May2017/R17

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control x2
      Zero all spectra

16.54 DAQ starts
      File May2017/R18
      Attached Ras Pi system console log for USB0
  578   Fri May 19 10:16:27 2017 CB, TD, OHDAQ sever rebooted
18:16 Finished rebooting DAQ server and acquisition. All OK, except ADC calibration on #7.
      Started run May2017/R16
      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases; leakage currents OK
      FEE64 Temperatures OK
      good events; statistics OK
      No warnings/errors reported by merger since DAQ GO
  577   Fri May 19 09:55:54 2017 CB, TD, OHLogs from Raspberry Pi System Console
17:50 Rebooted Server and FEE64. Connected remotely to Ras Pi System Console. Repeated USB connection procedure. All 24 modules can now be seen. Sample log attached
  576   Fri May 19 02:44:14 2017 TD, CB, OHFriday 19th May
10:44 System wide checks all ok
      Leakages and biases all ok (see attachment 1)
      Ambient Temperature 24.4C - Humidity 38.1% - Dew Point +9.1
      FEE temperatures ok (see attachment 2)
      DAQ, Merger and Tape server all still running ok
      Good events statistics (see attachment 3)

      Orientation of RS-232 cards was wrong. They have now been corrected will enable logging then do a halt and
go on the daq.
      Power supply also moved onto AIDA circuit.

11:00 Current file R15_333.
      
11:04 halt and go performed
      24 Log files produced on Rasp Pi console monitor.
  575   Thu May 18 15:20:44 2017 Patrick Coleman-Smith[HowTo] Connecting a USB console interface to the FEE64

The three attachments show the orientation of the console cable pcb relative to the FEE64 connectors.

The first with the other cables and the other two with no cables for clarity of position.

  574   Thu May 18 09:46:36 2017 TD, OH, CBRasPi system console diagnostics
> Can you let me know when you power-cycled the FEEs please ? Or maybe just do a stop/go.
> If there is nothing to log from the consoles then the Pi won't have made a log file. 
> The software opens and closes the log file each time a line of text comes in from the FEE64.

The most recent power cycle of the FEE64s was May 5

DAQ stop/go per your request - see Elog https://elog.ph.ed.ac.uk/AIDA/571
  573   Thu May 18 09:29:50 2017 TD, OH, CBRasPi system console diagnostics
> Connecting monitor to RasPi system console diagnostics
> 
> May 18 10:59:38 aidas1 dhcpd: DHCPREQUEST for 10.32.6.189 from b8:27:eb:6b:0a:a0 via p2p1: wrong network.
> May 18 10:59:38 aidas1 dhcpd: DHCPNAK on 10.32.6.189 to b8:27:eb:6b:0a:a0 via p2p1
> May 18 10:59:38 aidas1 dhcpd: DHCPDISCOVER from b8:27:eb:6b:0a:a0 via p2p1
> May 18 10:59:39 aidas1 dhcpd: DHCPOFFER on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 10:59:39 aidas1 dhcpd: Wrote 0 deleted host decls to leases file.
> May 18 10:59:39 aidas1 dhcpd: Wrote 0 new dynamic host decls to leases file.
> May 18 10:59:39 aidas1 dhcpd: Wrote 44 leases to leases file.
> May 18 10:59:39 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 (10.1.1.250) from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 10:59:39 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 11:05:46 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 11:05:46 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> 
> 
> ssh np-monitor -> could not resolve hostname
> ssh 10.1.1.117 -> OK
> 
> Connecting remotely to RasPi system console diagnostics
> 
> http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml -> Looks OK (see attached 1)
> Connected 24 USB cables. 6 cables per HUB. Connected all HUBs to RasPi. Pressed Update USB terminal list. We see 24
> USBs (0-23) (see attached 2)
> Pressed Connect USB terminal. Parse USB Log file for details. We can only see ttyUSB0,1,2,3. Full log:
> 
> ttyUSB0 =>  last host=nnaida14, => last addr=193.62.115.227, ; #panics => 0 ; Finished OK 
> ttyUSB1 =>  last host=nnlycca4, => last addr=193.62.115.240, ; #panics => 0 ; Finished OK 
> ttyUSB10 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB11 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB12 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB13 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB14 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB15 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB16 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB17 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB18 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB19 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB2 =>  last host=nnaida11, => last addr=193.62.115.232, ; #panics => 0 ; Finished OK 
> ttyUSB20 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB21 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB22 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB23 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB3 =>  last host=nnaida12, => last addr=193.62.115.233, ; #panics => 0 ; Finished OK 
> ttyUSB4 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB5 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB6 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB7 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB8 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB9 =>  last  => last  ; #panics => 0 ; Not Finished 
> 
> ttyUSB0
> ttyUSB1
> ttyUSB10
> ttyUSB11
> ttyUSB12
> ttyUSB13
> ttyUSB14
> ttyUSB15
> ttyUSB16
> ttyUSB17
> ttyUSB18
> ttyUSB19
> ttyUSB2
> ttyUSB20
> ttyUSB21
> ttyUSB22
> ttyUSB23
> ttyUSB3
> ttyUSB4
> ttyUSB5
> ttyUSB6
> ttyUSB7
> ttyUSB8
> ttyUSB9
> 
> Ssh'ed to RasPi. There are only four log files in ~/logs/
> 
> sudo lsusb -t results in
> 
> /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
>     |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/5p, 480M
>         |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
>         |__ Port 2: Dev 4, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 5, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 6, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 7, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 8, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 9, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 10, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 11, If 0, Class=vend., Driver=ftdi_sio, 12M
>         |__ Port 3: Dev 20, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 21, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 22, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 23, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 24, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 25, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 26, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 27, If 0, Class=vend., Driver=ftdi_sio, 12M
>         |__ Port 4: Dev 12, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 13, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 14, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 15, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 16, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 17, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 18, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 19, If 0, Class=vend., Driver=ftdi_sio, 12M
>         |__ Port 5: Dev 28, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 29, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 30, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 31, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 32, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 33, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 34, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 35, If 0, Class=vend., Driver=ftdi_sio, 12M

Can you let me know when you power-cycled the FEEs please ? Or maybe just do a stop/go.
If there is nothing to log from the consoles then the Pi won't have made a log file. 
The software opens and closes the log file each time a line of text comes in from the FEE64.
  572   Thu May 18 03:07:47 2017 TD, OH, CBRasPi system console diagnostics
Connecting monitor to RasPi system console diagnostics

May 18 10:59:38 aidas1 dhcpd: DHCPREQUEST for 10.32.6.189 from b8:27:eb:6b:0a:a0 via p2p1: wrong network.
May 18 10:59:38 aidas1 dhcpd: DHCPNAK on 10.32.6.189 to b8:27:eb:6b:0a:a0 via p2p1
May 18 10:59:38 aidas1 dhcpd: DHCPDISCOVER from b8:27:eb:6b:0a:a0 via p2p1
May 18 10:59:39 aidas1 dhcpd: DHCPOFFER on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 10:59:39 aidas1 dhcpd: Wrote 0 deleted host decls to leases file.
May 18 10:59:39 aidas1 dhcpd: Wrote 0 new dynamic host decls to leases file.
May 18 10:59:39 aidas1 dhcpd: Wrote 44 leases to leases file.
May 18 10:59:39 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 (10.1.1.250) from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 10:59:39 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 11:05:46 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 11:05:46 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1


ssh np-monitor -> could not resolve hostname
ssh 10.1.1.117 -> OK

Connecting remotely to RasPi system console diagnostics

http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml -> Looks OK (see attached 1)
Connected 24 USB cables. 6 cables per HUB. Connected all HUBs to RasPi. Pressed Update USB terminal list. We see 24
USBs (0-23) (see attached 2)
Pressed Connect USB terminal. Parse USB Log file for details. We can only see ttyUSB0,1,2,3. Full log:

ttyUSB0 =>  last host=nnaida14, => last addr=193.62.115.227, ; #panics => 0 ; Finished OK 
ttyUSB1 =>  last host=nnlycca4, => last addr=193.62.115.240, ; #panics => 0 ; Finished OK 
ttyUSB10 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB11 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB12 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB13 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB14 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB15 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB16 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB17 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB18 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB19 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB2 =>  last host=nnaida11, => last addr=193.62.115.232, ; #panics => 0 ; Finished OK 
ttyUSB20 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB21 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB22 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB23 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB3 =>  last host=nnaida12, => last addr=193.62.115.233, ; #panics => 0 ; Finished OK 
ttyUSB4 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB5 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB6 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB7 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB8 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB9 =>  last  => last  ; #panics => 0 ; Not Finished 

ttyUSB0
ttyUSB1
ttyUSB10
ttyUSB11
ttyUSB12
ttyUSB13
ttyUSB14
ttyUSB15
ttyUSB16
ttyUSB17
ttyUSB18
ttyUSB19
ttyUSB2
ttyUSB20
ttyUSB21
ttyUSB22
ttyUSB23
ttyUSB3
ttyUSB4
ttyUSB5
ttyUSB6
ttyUSB7
ttyUSB8
ttyUSB9

Ssh'ed to RasPi. There are only four log files in ~/logs/

sudo lsusb -t results in

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
        |__ Port 2: Dev 4, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 5, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 6, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 7, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 8, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 9, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 10, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 11, If 0, Class=vend., Driver=ftdi_sio, 12M
        |__ Port 3: Dev 20, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 21, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 22, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 23, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 24, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 25, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 26, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 27, If 0, Class=vend., Driver=ftdi_sio, 12M
        |__ Port 4: Dev 12, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 13, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 14, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 15, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 16, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 17, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 18, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 19, If 0, Class=vend., Driver=ftdi_sio, 12M
        |__ Port 5: Dev 28, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 29, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 30, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 31, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 32, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 33, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 34, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 35, If 0, Class=vend., Driver=ftdi_sio, 12M
  571   Thu May 18 02:16:05 2017 CB, TD, OHThursday 18 May
10.10 DAQ continues OK 
      Current file May2017/R10_346

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      Detector biases &amp; leakage currents OK - see attachments 1 &amp; 2
      FEE64 Temperatures OK - see attachment 3
      &#39;good events&#39; statistics OK - see attachment 4
      No warnings/errors reported by merger since DAQ GO - see attachment 5

      *all* system wide checks OK

10.20 E17/F11 ambient temperature +24.3 deg C, RH 39.0%, d.p. +9.3 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 65%)

10.25 DAQ stops OK
      Current file May2017/R13_294

      Files May2017/R13* *deleted* 

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control x2
      Zero all spectra

10.30 DAQ starts
      File May2017/R14

17.34 DAQ stops OK
      Current file May2017/R14_113

      Files May2017/R14* *deleted* 

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0xff

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

      Check ASIC Control
      Zero all spectra

      *all* system wide checks OK

17.37 DAQ starts
      File May2017/R15

      Detector biases & leakage currents OK - see attachments 6 & 7
      FEE64 Temperatures OK - see attachment 8
      good events statistics - higher rates for several FEE64s - due to addition of system consoles? - see
attachment 9
      No warnings/errors reported by merger since DAQ GO - ~ +30% data rate - see attachment 10
ELOG V3.1.4-unknown