AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 2 of 46  ELOG logo
    Reply  Fri Aug 19 09:43:27 2016, TD, VP, reply PJCS, Friday 19 August 
> 
> 14.00 Both AIDA NIM bins U/S 
>       Upper NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B - ac mains fuse 100V/8A/slow? blowing
>       Lower NIM bin - 11x MACB - not powering up
> 
>       Replaced both NIM bins.
>       Upper NIM bin - 11x MACB
>       Lower NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B
> 
> 15.50 Julabo FL11006 chiller ON - set point +20 deg C
>       B2F/F11 ambient temp +24.0 deg C, RH 54%, DP +13.8 deg C
> 
>       Bias 6x MSL type BB18 DSSSDs - see attachments 1-2
> 
> 16.10 Startup DAQ nnaida1-24 *except* nnaida19
> 
>       System-wide checks
> 
>       Check clock status OK
>       Check ADC calibration - nnaida7 failed
>       Check ASIC clock timestamp - multiple failures - see attachment 3
>       Check SYNC errors not counted Since Baseline - see attachment 4
>       Check SYNC pulses received - see attachment 5
> 
>       BNC PB-4 
> 
>       Amplitude 90,000, x5 attenuator IN
>       t_r 50ns, tau_d 50us
>       Polarity -
>       INT ref, delay MIN, tail pulse
> 
>       FEE64 temperatures - see attachment 6
> 
> 16.34 R1 starts (directory Aug2016)
> 
>       slow comparator 0x10, LEC/MEC fast comparator 0x10
> 
>       merge 555k data items/s, tapeserver 4430 kb/s
Looks like a problem with the inter-MACB wiring.
The screen shots show groups of 4 with the problems.
I suggest re-seating the short HDMI cables and reset the system.
    Reply  Fri Aug 19 11:07:44 2016, TD, VP, reply PJCS, Friday 19 August 7.png
> > 
> > 14.00 Both AIDA NIM bins U/S 
> >       Upper NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B - ac mains fuse 100V/8A/slow? blowing
> >       Lower NIM bin - 11x MACB - not powering up
> > 
> >       Replaced both NIM bins.
> >       Upper NIM bin - 11x MACB
> >       Lower NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B
> > 
> > 15.50 Julabo FL11006 chiller ON - set point +20 deg C
> >       B2F/F11 ambient temp +24.0 deg C, RH 54%, DP +13.8 deg C
> > 
> >       Bias 6x MSL type BB18 DSSSDs - see attachments 1-2
> > 
> > 16.10 Startup DAQ nnaida1-24 *except* nnaida19
> > 
> >       System-wide checks
> > 
> >       Check clock status OK
> >       Check ADC calibration - nnaida7 failed
> >       Check ASIC clock timestamp - multiple failures - see attachment 3
> >       Check SYNC errors not counted Since Baseline - see attachment 4
> >       Check SYNC pulses received - see attachment 5
> > 
> >       BNC PB-4 
> > 
> >       Amplitude 90,000, x5 attenuator IN
> >       t_r 50ns, tau_d 50us
> >       Polarity -
> >       INT ref, delay MIN, tail pulse
> > 
> >       FEE64 temperatures - see attachment 6
> > 
> > 16.34 R1 starts (directory Aug2016)
> > 
> >       slow comparator 0x10, LEC/MEC fast comparator 0x10
> > 
> >       merge 555k data items/s, tapeserver 4430 kb/s
> Looks like a problem with the inter-MACB wiring.
> The screen shots show groups of 4 with the problems.
> I suggest re-seating the short HDMI cables and reset the system.

Short HDMI cables re-seated. See attachment 1
    Reply  Fri Aug 19 12:41:43 2016, TD, VP, reply PJCS, Friday 19 August 
> > > 
> > > 14.00 Both AIDA NIM bins U/S 
> > >       Upper NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B - ac mains fuse 100V/8A/slow? blowing
> > >       Lower NIM bin - 11x MACB - not powering up
> > > 
> > >       Replaced both NIM bins.
> > >       Upper NIM bin - 11x MACB
> > >       Lower NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B
> > > 
> > > 15.50 Julabo FL11006 chiller ON - set point +20 deg C
> > >       B2F/F11 ambient temp +24.0 deg C, RH 54%, DP +13.8 deg C
> > > 
> > >       Bias 6x MSL type BB18 DSSSDs - see attachments 1-2
> > > 
> > > 16.10 Startup DAQ nnaida1-24 *except* nnaida19
> > > 
> > >       System-wide checks
> > > 
> > >       Check clock status OK
> > >       Check ADC calibration - nnaida7 failed
> > >       Check ASIC clock timestamp - multiple failures - see attachment 3
> > >       Check SYNC errors not counted Since Baseline - see attachment 4
> > >       Check SYNC pulses received - see attachment 5
> > > 
> > >       BNC PB-4 
> > > 
> > >       Amplitude 90,000, x5 attenuator IN
> > >       t_r 50ns, tau_d 50us
> > >       Polarity -
> > >       INT ref, delay MIN, tail pulse
> > > 
> > >       FEE64 temperatures - see attachment 6
> > > 
> > > 16.34 R1 starts (directory Aug2016)
> > > 
> > >       slow comparator 0x10, LEC/MEC fast comparator 0x10
> > > 
> > >       merge 555k data items/s, tapeserver 4430 kb/s
> > Looks like a problem with the inter-MACB wiring.
> > The screen shots show groups of 4 with the problems.
> > I suggest re-seating the short HDMI cables and reset the system.
> 
> Short HDMI cables re-seated. See attachment 1

ok, the discrepancy is odd. Not sure if it is due to an uneven reset or more noise in the wiring.
In system wide checks redo the baseline for the SYNC error counters and keep an eye on it.

Maybe a power cycle will bring all the SYNC received counters into line.

Certainly they should be exactly the same.

Is there any correlation between the ones that are wrong and the MACB layout/cabling ?
Entry  Tue Dec 6 06:38:46 2016, TD, Updated Patrick, CAEN N1419 HV output common mode noise 9x
There appears to be significant common mode noise on the output of the
CAEN N1419 4 Ch Reversible 500 V/200 µA NIM HV Power Supply Module (USB)
- the specification quotes 'Max Ripple smaller than < 5mVpp'. We configure
the CAEN N1419 outputs as floating wrt NIM module/crate chassis.

For example, attachment 1, shows the output of module #2 (Addr 01) channel #3
(unused channel) set to 10V (leakage current 0.030uA) - core (Ch 1)/braid (Ch 2).
Tektronix 3034C channels 1 & 2 are both set ac/1M.

Attachment 2 shows math function Ch 1 - Ch 2 (50mV/division cf. 200mV/division
for Ch 1 and Ch 2)

Attachment 3 shows CAEN N1419 #2 settings

Attachment 4 per attachments 1 & 2 *except* the cables from channels 0 and 1
of CAEN N1419 #2 have been disconnected from the NIM module. The structure of
the common mode noise changes but its amplitude remains significant. This
presumably indicates that there are other sources of common mode noise in
addition to that observed at the output of the CAEN N1419 HV outputs.

Attachments 5 & 6 show the NIM crate front panel +6V output wrt ground
for the lower (NIM Bin type REPIC,  BNC PB-4, Ortec 433, 2x MACB, 2x CAEN N1419,
1x CAEN 1419B) and upper (NIM Bin type OKEN, 9x MACB) AIDA NIM bins respectively.
All +/-24V, +/-12V and +/-6V front panel outputs show the same noise structure.

Attachment 7 shows equivalent of attachment 6 but using Tektronix DPO3054
which can display FFT.

Update -----------------------------------------
Tested the CAEN HV supply in T9 using the same method and observed similar results.

With help from Mos Kogimtzis a better method of measuring the HV noise was used.
The co-ax from the isolated output of the HV module was connected to one input of the 'scope. HV set to 1V.
Thus the screen of the HV signal is grounded at the 'scope amplifier input. This nearer to how the HV is used with a detector. 
The single signal input channel was set to DC 1MR with an offset of 1V.
The 'scope can now display at 4mV/division. 
The peak/peak was less than 5mV.See attachment 8 slide 1

There are features at both high and low frequency, attachments slides 2 and 3. NB 1mV/division.

If a differential measurement is required then a suitable cable with a shielded twisted pair is needed.

Update #2 -----------------------------------------
Moved the AIDA T9 HV supply to a CAEN NIM bin. Newly purchased this year.
This is the only module in the bin.
 Used the simple method of noise measuring as before with the same cables and 'scope.
Attachment 9 shows there is a different type of noise. Not regular like attachment 8 but more random and larger.

The HV is set to 1V.

Further investigation will be carried out using AC coupling and HV set to 50V.
    Reply  Thu Apr 23 09:02:57 2015, TD, Patrick, FEE64 Fast Discriminator OR o/p via MACB 
> Illustration of the trigger timing of the AIDA FEE64 fast discriminator OR output via the
> MACB front panel Trigger outputs.
> 
> Channel 1: BNC PB-4 Pulser Trigger Output
> Channel 2: BNC PB-4 Pulser Analogue Output
> Channel 3: MACB Trigger Output
> 
> MACB Trigger Output: NIM complement
>                      width ~ 7us
>                      delay ~ 750ns (with respect to BNC PB_4 Pulser Analogue Output)
> 
> Example illustrates MAC Trigger Output from nnaida5 with standard ASIC settings and
> LEC/MEC fast discriminator 0x40

It would be useful to show the Trigger signal at the different levels of the MACB distribution system... then 
the difference will show the delay through the MACB layer.

Are you able to estimate the length of cable between the pulser and the adaptor boards.

What are the conclusions about the delay .... is it too much to be useful ?
    Reply  Mon Jun 19 08:58:00 2017, TD, Patrick, Report - high - 'zero all histograms' doesn't 
> 
> The DAQ Run Control 'zero all histograms' function appears to have stopped working
> - an issue related to the new RunControl/sys.tcl ... ?
> 
> See https://elog.ph.ed.ac.uk/AIDA/662

Fault discovered and file edited at Daresbury.
A piece of random text was in the " zero histograms" process.

The repaired file has been installed in RIKEN.
Entry  Mon May 29 00:59:27 2017, TD, PV, CB, NN, OH, Monday 29 May 10x
09.00 DAQ continues OK 
      Current file May2017/R32_239

      Detector biases; leakage currents OK - see attachments 1 & 2
      FEE64 Temperatures OK - see attachment 3
      good events; statistics OK - see attachment 4
      Some error messages reported by merger yesterday but none overnight - see attachment 5

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status =6)

09.33 E17/F11 ambient temperature +24.0 deg C, RH 38.0%, d.p. +8.8 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

09.35 Analysis of R32_259 - OK see attachment 6

10.50 Offline analysis R32_277 - ADC & disc data synchronised - see attachments 7-9

11.00 DAQ stop OK 
      File May2017/R32_280

      Check ASIC control x2
      Zero all spectra

11.05 DAQ go
      File May2017/R33

      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0x20

15.00 Merger crashed (SEGV) - see https://elog.ph.ed.ac.uk/AIDA/607

      DAQ stop
      Restart Merger
      DAQ stop OK following transient nnaida1 timeout

      Close merger/tapeserver browser, kill httpd for merger/tapeserver,merger and tapeserver terminal server
sessions
      Disable histogramming and data transfer #1
      DAQ reset, setup
      Restart merger and tapeserver
      Enable histogramming and data transfer #1

      *all* system wide checks OK *except* nnaida1 & nnaida21 fail ADC calibration, nnaida1 clock status=6

15.18 DAQ start
      File May2017/R34

15.25 no merger error messages
      nnaida1 & nnaida2 good events high - ~100k and ~120k respectively - see attachment 10

16.24 removed R32
      System wide checks - ASIC clock timestamp test fails on all nnaidas. Synchronised clocks. ASIC clock test
      passes.
      FEE temperatures, leakage currents - OK
      Merger reports 

      Merge: Restarting from timestamp 1 or greater
      Merge: Starting at timestamp 27d1e55c00a0
      Merge: Trying to add items to Q index 21 (Link 21) but queue already full

      Probably as a result of ADC clock synchronisation.

17:03 Used analyser on R34_23. No timewarps, negligible dead time. All other values as normal.

17.19 FEE64 temperatures, leakage currents - OK
      nnaida1 & nnaida 2 good events still high - ~100k roughly
      No further messages from merger

17.45 DAQ continues OK
      BRIKEN reports increased neutron and gamma activity. No apparent effect on AIDA rates.

18.50 DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger errors
      Used analyser on run R34_49 - no timewarps, rates as expected

19.30 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_58 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~2 s/254 s on nnaida 2 

20.00 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_63 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~1.3 s/255 s on nnaida 2 

20.15 BigRIPS reports beam optimisation may be over soon.
      DAQ continues OK.

20.47 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_76 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~1.1 s/255 s on nnaida 2 

21.49 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_90 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~2.0 s/258 s on nnaida 2 

23.05 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_109 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~1.9 s/256 s on nnaida 2 
Entry  Thu Jun 1 00:00:25 2017, TD, PV, CB, Thusrday 1 June 08.00-16.00 17x
07.57 run continues
      BRIKEN start run 55
      BigRIPS start run 1066
      AIDA file RIBF127R1/R1_763

      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39

      84Zn BigRIPS setting, 78Ni degrader setting - 5mm (in F7) + 9mm (in F11) + 1mm (in F11) = 16 mm Al


09.01 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_779

09.03 BRIKEN start run 56
      BigRIPS start run 1067
      AIDA file RIBF127R1/R1_779

09.32 Beam dropped
      AIDA file RIBF127R1/R1_786

09.38 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_788

09.43 Beam is back      
      BRIKEN start run 57
      BigRIPS start run 1068
      AIDA file RIBF127R1/R1_789      

09.55 Beam is off
      There will be no beam for 1 hour, accelerator people want to check the ion source.
      DAQ stop BigRIPS, BRIKEN
      AIDA file R1_792

10.29 Beam is back
      BRIKEN start run 58
      BigRIPS start run 1069
      AIDA file 799
      Beam intensity reduced half, then slowly increasing


11.29 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_813

11.29 BRIKEN start run 59
      BigRIPS start run 1070
      AIDA file RIBF127R1/R1_813
      
11.32 Sorted file RIBF127R1/R1_811. DSSSD rates are as follows

*** data items:    2431382 ( 108760.05 Hz)
 *** ADC events:     260316 (  11644.40 Hz)
 *** time warps:          0 (      0.00 Hz)
 *** DSSSD # 1 count:    109746 old count:     88603 dt:      2.40 s  LEC rate:  45784.75 Hz
 *** DSSSD # 2 count:    148624 old count:    119105 dt:      2.40 s  LEC rate:  62004.19 Hz
 *** DSSSD # 3 count:     50556 old count:     40453 dt:      2.40 s  LEC rate:  21091.37 Hz
 *** DSSSD # 4 count:     43410 old count:     35484 dt:      2.40 s  LEC rate:  18110.14 Hz
 *** DSSSD # 5 count:     17933 old count:     14701 dt:      2.40 s  LEC rate:   7481.44 Hz
 *** DSSSD # 6 count:     36412 old count:     30468 dt:      2.40 s  LEC rate:  15190.66 Hz
 *** DSSSD # 1 count:        42 old count:        40 dt:      2.40 s  HEC rate:     17.52 Hz
 *** DSSSD # 2 count:        85 old count:        70 dt:      2.40 s  HEC rate:     35.46 Hz
 *** DSSSD # 3 count:        85 old count:        76 dt:      2.40 s  HEC rate:     35.46 Hz
 *** DSSSD # 4 count:        58 old count:        51 dt:      2.40 s  HEC rate:     24.20 Hz
 *** DSSSD # 5 count:        35 old count:        28 dt:      2.40 s  HEC rate:     14.60 Hz
 *** DSSSD # 6 count:         7 old count:         6 dt:      2.40 s  HEC rate:      2.92 Hz
     

11.40 Detector biases & leakage currents OK - see attachments 1
      FEE64 Temperatures OK - see attachment 2
      good events statistics OK - see attachment 3
      No new merger error messages - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)
      all SortSas spectra zero'd

11.45 Analysed RIBF127R1/R1_815 - see attachment 5
      No timewarps
      Negligible deadtime. No idle time issues.

11.49 System console log ttyUSB7 (nnaida1) attached - see attachment 6
      Current HEC hit pattern OK - see attachment 7


12.40 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_829

12.43 BRIKEN start run 60
      BigRIPS start run 1071
      AIDA file RIBF127R1/R1_830

13.45 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_844

13.46 BRIKEN start run 61
      BigRIPS start run 1072
      AIDA file RIBF127R1/R1_845
      
      all SortSas online spectra zero'd
      x-y hit pattern OK

13:56 System wide checks all ok
      Screenshots included for merger, statistics and FEE temperatures - attachments 8-10

14:04 leakages and Biases ok - attachment 11

14.46 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_858

14.47 BRIKEN start run 62
      BigRIPS start run 1073
      AIDA file RIBF127R1/R1_858
      
      all SortSas online spectra zero'd

14:56 online analysis x-y hit pattern OK - attachment 12

15.46 DAQ stop BigRIPS, BRIKEN
      AIDA file RIBF127R1/R1_872

15.47 BRIKEN start run 63
      BigRIPS start run 1074
      AIDA file RIBF127R1/R1_872

15.57 R1_873.txt checked
      No timewarps
      Negligible deadtime. No idle time issues.


16.22 Detector biases & leakage currents OK - see attachments 14
      No new merger error messages - see attachment 15
      FEE64 Temperatures OK - see attachment 16
      good events statistics OK - see attachment 17


      
Entry  Fri Aug 19 07:42:22 2016, TD, PV, Friday 19 August 6x
14.00 Both AIDA NIM bins U/S 
      Upper NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B - ac mains fuse 100V/8A/slow? blowing
      Lower NIM bin - 11x MACB - not powering up

      Replaced both NIM bins.
      Upper NIM bin - 11x MACB
      Lower NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B

15.50 Julabo FL11006 chiller ON - set point +20 deg C
      B2F/F11 ambient temp +24.0 deg C, RH 54%, DP +13.8 deg C

      Bias 6x MSL type BB18 DSSSDs - see attachments 1-2

16.10 Startup DAQ nnaida1-24 *except* nnaida19

      System-wide checks

      Check clock status OK
      Check ADC calibration - nnaida7 failed
      Check ASIC clock timestamp - multiple failures - see attachment 3
      Check SYNC errors not counted Since Baseline - see attachment 4
      Check SYNC pulses received - see attachment 5

      BNC PB-4 

      Amplitude 90,000, x5 attenuator IN
      t_r 50ns, tau_d 50us
      Polarity -
      INT ref, delay MIN, tail pulse

      FEE64 temperatures - see attachment 6

16.34 R1 starts (directory Aug2016)

      slow comparator 0x10, LEC/MEC fast comparator 0x10

      merge 555k data items/s, tapeserver 4430 kb/s

18.28 R4 starts

      Settings per R1

18.31 DAQ/merge stalls with (many) messages as follows

MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
Merge state now Paused
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
copyItems: queue 14, maybe copied in wrong buffer: sequence 1146, last 575
Merge_addItems: queue 14, maybe added wrong buffer: sequence 1146, last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in new data, found 689, last 1258
copyItems: queue 14, maybe copied in wrong buffer: sequence 689, last 1258
Merge_addItems: queue 14, maybe added wrong buffer: sequence 689, last 1258
MERGE Data Link (24254): client closed connection
MERGE Data Link (24254): listening on port 11001
MERGE Data Link (24255): client closed connection
MERGE Data Link (24255): listening on port 11002
MERGE Data Link (24257): client closed connection
MERGE Data Link (24257): listening on port 11004
MERGE Data Link (24258): client closed connection
MERGE Data Link (24258): listening on port 11005
MERGE Data Link (24261): client closed connection
MERGE Data Link (24261): listening on port 11008
MERGE Data Link (24260): client closed connection
MERGE Data Link (24260): listening on port 11007
MERGE Data Link (24259): client closed connection
MERGE Data Link (24259): listening on port 11006
MERGE Data Link (24256): client closed connection
MERGE Data Link (24256): listening on port 11003
MERGE Data Link (24262): client closed connection
MERGE Data Link (24262): listening on port 11009
MERGE Data Link (24263): client closed connection
MERGE Data Link (24263): listening on port 11010
MERGE Data Link (24265): client closed connection
MERGE Data Link (24265): listening on port 11012
MERGE Data Link (24266): client closed connection
MERGE Data Link (24266): listening on port 11013
MERGE Data Link (24267): client closed connection
MERGE Data Link (24267): listening on port 11014
MERGE Data Link (24269): client closed connection
MERGE Data Link (24269): listening on port 11016
MERGE Data Link (24268): client closed connection
MERGE Data Link (24268): listening on port 11015
MERGE Data Link (24270): client closed connection
MERGE Data Link (24270): listening on port 11017
MERGE Data Link (24264): client closed connection
MERGE Data Link (24264): listening on port 11011
MERGE Data Link (24273): client closed connection
MERGE Data Link (24273): listening on port 11020
MERGE Data Link (24271): client closed connection
MERGE Data Link (24271): listening on port 11018
MERGE Data Link (24277): client closed connection
MERGE Data Link (24277): listening on port 11024
MERGE Data Link (24275): client closed connection
MERGE Data Link (24275): listening on port 11022
MERGE Data Link (24274): client closed connection
MERGE Data Link (24274): listening on port 11021
MERGE Data Link (24276): client closed connection
MERGE Data Link (24276): listening on port 11023
Entry  Fri Jul 7 10:52:34 2023, TD, PJCS, Test of AIDA FEE64 d8:80:39:42:0d:0b ( aida03 ) - ASIC mezzanine replaced 16x
Test of AIDA FEE64 d8:80:39:42:0d:0b ( aida03 ) - ASIC mezzanine replaced

ISOL test station STFC DL, T9

Pulser BNC PB-5

Amplitude 1.0V
Attenuation x1
Polarity +
Frequency 100Hz
Tail pulse, decay time 1ms
Delay 250ns (min)

ASIC settings 2019May22-10.19.28
Slow comparator 0x80

Adaptor PCB - PJCS


FEE64 temperatures OK - attachments 1


System wide checks OK - attachments 2-5


Statistics OK - attachments 6-7

Hit/HitRate spectra OK - attachment 8

LEC spectra - attachments 9-12

Data observed for all channels of ASIC #1, 3, & 4
Data observed (double peaks) ASIC#2 channels 4-7 & 12-15, no data observed channels 0-3 & 8-11 - ASIC#2 address bit #2 set always ON? ASIC #2 fail or Mezzanine-FEE64 
connection issue?
Typical pulser peak widths 15-17 channels FWHM ASIC #1-4


Wave spectra OK - attachments 13-16

19/7/23 removed mezzanine and cleaned contacts with ipa. cleaned FEE contacts. reassembled mezzanine to FEE and tested. ASIC 2 now ok.
    Reply  Fri Mar 29 08:39:18 2024, TD, PJCS, Unable to start MIDAS DAQ with 1x FEE64 ( aida04 ) 
> 
> At AIDA test station T9, STFC DL unable to start MIDAS DAQ with 1x FEE64 ( aida04 ) - see attachments
> 
> aida04 appears to have booted OK and I can connect using telnet
> 
> MIDAS 8015 server exits - signal 1

I’ve had this problem too. The solution is to have the single FEE twice in the list in the config file. The software is then quite happy.

Alternatively a single FEE can still be accessed directly and will operate successfully.  The only thing missing is the spectra function menu. They won’t work, so FWHM is harder to determine. 
Entry  Thu Nov 10 09:56:20 2016, TD, OH, ML, NP1412 R26 R27 8x
start 18.55 10.11.16
stop 
AIDA filename R26_0
BRIKEN filename 161109_1852_767778Ni_045
BigRIPS filename 443
ASIC settings 2016Nov07_23.03.06
DSSSD bias & leakage currents OK attachments 1 and 2 

Degrader all degraders out
F11 rate 430Hz
BigRIPS setting 767778Ni  F5  out, F7 CH2 in, F11 6mm C in

From offline analysis of R26_2

 *** DSSSD # 1 count:       459 old count:       225 dt:      1.02 s  HEC rate:    229.83 Hz
 *** DSSSD # 2 count:       385 old count:       191 dt:      1.02 s  HEC rate:    190.54 Hz
 *** DSSSD # 3 count:       461 old count:       228 dt:      1.02 s  HEC rate:    228.85 Hz
 *** DSSSD # 4 count:       403 old count:       197 dt:      1.02 s  HEC rate:    202.33 Hz
 *** DSSSD # 5 count:       218 old count:       121 dt:      1.02 s  HEC rate:     95.27 Hz
 *** DSSSD # 6 count:        87 old count:        42 dt:      1.02 s  HEC rate:     44.20 Hz

19.02 FEE64 temperature OK - see attachment 3

19.16 BRIKEN DAQ problems following restart - multiple correlation scaler reset requests issued - ignore data
before R26_5
      New BRIKEN data file 161109_1852_767778Ni_046

19.20 AIDA DAQ stalled (presumably *end* of LN2 fill cycle)

      Soft reset OK
      Clock lock state status 0xd - see attachment 6

19.29 Start DAQ (R27) 

19.35 BRIKEN DAQ problems starting SyncCheck program - - multiple correlation scaler reset requests issued -
ignore data before R27_2

19.37 All degraders (5.4mm Al) in for new Takechi et al. setting
      BigRIPS setting 767778Ni  F5 in, F7 out, F11 6mm C in

BRIKEN filename 161109_1852_767778Ni_045
AIDA fileame R27_3
BigRIPS filename 444

19.40 All degraders out

DSSSD bias & leakage currents OK attachments 4 & 5 

From offline analysis of R27_4

 *** DSSSD # 1 count:       281 old count:       247 dt:      2.31 s  HEC rate:    121.43 Hz
 *** DSSSD # 2 count:       247 old count:       219 dt:      2.31 s  HEC rate:    106.74 Hz
 *** DSSSD # 3 count:       286 old count:       252 dt:      2.31 s  HEC rate:    123.59 Hz
 *** DSSSD # 4 count:       287 old count:       253 dt:      2.31 s  HEC rate:    124.02 Hz
 *** DSSSD # 5 count:       201 old count:       178 dt:      2.31 s  HEC rate:     86.86 Hz
 *** DSSSD # 6 count:        33 old count:        29 dt:      2.31 s  HEC rate:     14.26 Hz

20.36 Noted x4 increase in F11 rate to c. 2kHz
      All degraders in (5.4mm Al)

      BRIKEN DAQ shows x4-5 increase for c. 6 minutes
      DSSSD bias & leakage currents OK attachments 7 & 8
Entry  Tue Nov 8 13:37:06 2016, TD, OH, DK, NP1412 R2 6x
10:36 Run 2 Start
      Degraders (5.4mm Al) in at start and all removed once started.
      Other settings same as R1.

22.45 DAQ stop OK 

      All variable degraders (5.4mm Al) in

From analysis of R2_3

S O R T    C O M P L E T E ..... Tue Nov  8 22:59:02 2016


 *** ENTRY finish

 *** data items:  146156010 ( 541115.06 Hz)
 *** ADC events:    7815683 (  28936.09 Hz)
 *** time warps:          0 (      0.00 Hz)
 *** DSSSD # 1 count:     51901 old count:     51732 dt:     84.55 s  HEC rate:    613.87 Hz
 *** DSSSD # 2 count:     42966 old count:     42831 dt:     84.55 s  HEC rate:    508.19 Hz
 *** DSSSD # 3 count:     37092 old count:     36983 dt:     84.55 s  HEC rate:    438.71 Hz
 *** DSSSD # 4 count:     12570 old count:     12532 dt:     84.55 s  HEC rate:    148.67 Hz
 *** DSSSD # 5 count:      1927 old count:      1921 dt:     84.55 s  HEC rate:     22.79 Hz
 *** DSSSD # 6 count:       659 old count:       658 dt:     84.55 s  HEC rate:      7.79 Hz

 *** ENTRY finish ends

S O R T    S T O P P E D ..... Tue Nov  8 22:59:16 2016

s2000-s2005 m_p versus m_n DSSSD #1-6
s2010-s2015 x versus y DSSSD #1-6
s2020-s2025 E_p versus E_n DSSSD #1-6

23.05 DSSSD bias & leakage currents - see attachments 5 & 6
      FEE64 temperatures - see attachment 7
Entry  Tue Nov 8 14:22:16 2016, TD, OH, DK, NP1412 R3 
11:21 Run 3 Started
      Degrader moved upstream. AIDA moved to centre of BRIKEN. Fixed degrader moved upstream from centre of BRIKEN.
11:25 Run 3 Stopped
Entry  Tue Nov 8 14:29:34 2016, TD, OH, DK, NP1412 Run 4 
23:28 Run 4 Start
      Thick 6mm C target in beam
      
      Running with 3.2mm (variable) + 8mm (fixed) of Al degrader. Fixed degrader in same position as R3.
      Variable degrader reduced to 1mm of Al.
Entry  Sun Nov 6 01:34:15 2016, TD, OH, CG, Sun 6th November 18x
10:34 Detector Bias a leakage currents all ok. (Attachments 1 and 2)
      Temperatures all ok (Attachment 3)
      All system wide checks ok other than clock status see attachment 4
      DAQ continues to run ok (R124) (No stop at 6:30am)
      DAQ running at about 14MB/s to disc which is faster than before.
      Most likely due to the increased rate in detector 1 which is represented at FEEs 17, 20, 22 and 23 in
      attachment 5.

      Also attached are the spectra for layouts 1-4 & 7-8 as attachments 6-11
      When looking at the pulser spectrum in layouts 3 and 4 (attachments 9 and 10) we see that the pulser peaks
relating to this detector is broader shown by
      the decreased peak height.

      Comparing the waveforms in attachments 11 and 12 to those taken on Monday 31st Oct we see that the noise
in nnaida 5 and 6 has been greatly reduced.
      On the others however there appears to be an increase in the amplitude of the noise.

      11:25 The second putty window for leakage currents was found to have been frozen for the past 24 hours.
Was reloaded and updated leakage currents are in attachment 13 and 14

12.15 Analysis of R124_291 shows ADC & disc data synchronised - see attachment 15
      Analysis of data file - see attachment 16

18:26 DAQ stopped, presumably during LN2 fill cycle.
      Restarted the merger and tried "Stop" on the DAQ control page. ~50% of nnaida now show "error."
      R124 closed 18.29
      Could not regain control of the system. Jorge switched BRIKEN to its internal clock and FEEs power cycled @
19.20.

23.35 DAQ started (R126 - no storage mode)

      DSSSD bias & leakage currents OK - see attachment 
Entry  Thu Nov 10 12:02:24 2016, TD, OH, CG, Thursday 10 November 30.png31.png32.png34.png
21.17 DAQ starts (R30)

      Pulser walkthrough
      Amplitude 90,000 - 10,000 @ 10,000 step
      x5 attenuator IN
      t_r 50ns tau_d 50us
      polarity -, tail pulse, delay MIN
      frequency 250Hz range

21.30 DAQ stops OK 

21.32 DAQ starts (R31)

      Pulser walkthrough
      Amplitude 10,000 - 2,000 @ 2,000 step

21.40 DAQ stops OK

21.44 DAQ starts (R32)
     
      Background
      BNC PB-4 pulser OFF

21.59 DAQ stops OK

23.45 Started R35 so Jorge can do some tests.
      No storage mode.

00.20 Carried out manual fill on LN2 in advance of beam arriving later (~03.00).
      Took around 20mins.
      Continued writing to R35 after soft reset of DAQ.
Entry  Thu May 18 03:07:47 2017, TD, OH, CB, RasPi system console diagnostics 105.png105.png106.png
Connecting monitor to RasPi system console diagnostics

May 18 10:59:38 aidas1 dhcpd: DHCPREQUEST for 10.32.6.189 from b8:27:eb:6b:0a:a0 via p2p1: wrong network.
May 18 10:59:38 aidas1 dhcpd: DHCPNAK on 10.32.6.189 to b8:27:eb:6b:0a:a0 via p2p1
May 18 10:59:38 aidas1 dhcpd: DHCPDISCOVER from b8:27:eb:6b:0a:a0 via p2p1
May 18 10:59:39 aidas1 dhcpd: DHCPOFFER on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 10:59:39 aidas1 dhcpd: Wrote 0 deleted host decls to leases file.
May 18 10:59:39 aidas1 dhcpd: Wrote 0 new dynamic host decls to leases file.
May 18 10:59:39 aidas1 dhcpd: Wrote 44 leases to leases file.
May 18 10:59:39 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 (10.1.1.250) from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 10:59:39 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 11:05:46 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
May 18 11:05:46 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1


ssh np-monitor -> could not resolve hostname
ssh 10.1.1.117 -> OK

Connecting remotely to RasPi system console diagnostics

http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml -> Looks OK (see attached 1)
Connected 24 USB cables. 6 cables per HUB. Connected all HUBs to RasPi. Pressed Update USB terminal list. We see 24
USBs (0-23) (see attached 2)
Pressed Connect USB terminal. Parse USB Log file for details. We can only see ttyUSB0,1,2,3. Full log:

ttyUSB0 =>  last host=nnaida14, => last addr=193.62.115.227, ; #panics => 0 ; Finished OK 
ttyUSB1 =>  last host=nnlycca4, => last addr=193.62.115.240, ; #panics => 0 ; Finished OK 
ttyUSB10 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB11 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB12 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB13 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB14 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB15 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB16 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB17 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB18 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB19 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB2 =>  last host=nnaida11, => last addr=193.62.115.232, ; #panics => 0 ; Finished OK 
ttyUSB20 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB21 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB22 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB23 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB3 =>  last host=nnaida12, => last addr=193.62.115.233, ; #panics => 0 ; Finished OK 
ttyUSB4 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB5 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB6 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB7 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB8 =>  last  => last  ; #panics => 0 ; Not Finished 
ttyUSB9 =>  last  => last  ; #panics => 0 ; Not Finished 

ttyUSB0
ttyUSB1
ttyUSB10
ttyUSB11
ttyUSB12
ttyUSB13
ttyUSB14
ttyUSB15
ttyUSB16
ttyUSB17
ttyUSB18
ttyUSB19
ttyUSB2
ttyUSB20
ttyUSB21
ttyUSB22
ttyUSB23
ttyUSB3
ttyUSB4
ttyUSB5
ttyUSB6
ttyUSB7
ttyUSB8
ttyUSB9

Ssh'ed to RasPi. There are only four log files in ~/logs/

sudo lsusb -t results in

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
        |__ Port 2: Dev 4, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 5, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 6, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 7, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 8, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 9, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 10, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 11, If 0, Class=vend., Driver=ftdi_sio, 12M
        |__ Port 3: Dev 20, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 21, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 22, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 23, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 24, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 25, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 26, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 27, If 0, Class=vend., Driver=ftdi_sio, 12M
        |__ Port 4: Dev 12, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 13, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 14, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 15, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 16, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 17, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 18, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 19, If 0, Class=vend., Driver=ftdi_sio, 12M
        |__ Port 5: Dev 28, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 1: Dev 29, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 2: Dev 30, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 3: Dev 31, If 0, Class=vend., Driver=ftdi_sio, 12M
            |__ Port 4: Dev 32, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 33, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 2: Dev 34, If 0, Class=vend., Driver=ftdi_sio, 12M
                |__ Port 3: Dev 35, If 0, Class=vend., Driver=ftdi_sio, 12M
    Reply  Thu May 18 09:29:50 2017, TD, OH, CB, RasPi system console diagnostics 
> Connecting monitor to RasPi system console diagnostics
> 
> May 18 10:59:38 aidas1 dhcpd: DHCPREQUEST for 10.32.6.189 from b8:27:eb:6b:0a:a0 via p2p1: wrong network.
> May 18 10:59:38 aidas1 dhcpd: DHCPNAK on 10.32.6.189 to b8:27:eb:6b:0a:a0 via p2p1
> May 18 10:59:38 aidas1 dhcpd: DHCPDISCOVER from b8:27:eb:6b:0a:a0 via p2p1
> May 18 10:59:39 aidas1 dhcpd: DHCPOFFER on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 10:59:39 aidas1 dhcpd: Wrote 0 deleted host decls to leases file.
> May 18 10:59:39 aidas1 dhcpd: Wrote 0 new dynamic host decls to leases file.
> May 18 10:59:39 aidas1 dhcpd: Wrote 44 leases to leases file.
> May 18 10:59:39 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 (10.1.1.250) from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 10:59:39 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 11:05:46 aidas1 dhcpd: DHCPREQUEST for 10.1.1.117 from b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> May 18 11:05:46 aidas1 dhcpd: DHCPACK on 10.1.1.117 to b8:27:eb:6b:0a:a0 (npg-monitor) via p2p1
> 
> 
> ssh np-monitor -> could not resolve hostname
> ssh 10.1.1.117 -> OK
> 
> Connecting remotely to RasPi system console diagnostics
> 
> http://10.1.1.117:8015/AIDA/Pi_Monitor/Pi_Monitor.tml -> Looks OK (see attached 1)
> Connected 24 USB cables. 6 cables per HUB. Connected all HUBs to RasPi. Pressed Update USB terminal list. We see 24
> USBs (0-23) (see attached 2)
> Pressed Connect USB terminal. Parse USB Log file for details. We can only see ttyUSB0,1,2,3. Full log:
> 
> ttyUSB0 =>  last host=nnaida14, => last addr=193.62.115.227, ; #panics => 0 ; Finished OK 
> ttyUSB1 =>  last host=nnlycca4, => last addr=193.62.115.240, ; #panics => 0 ; Finished OK 
> ttyUSB10 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB11 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB12 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB13 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB14 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB15 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB16 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB17 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB18 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB19 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB2 =>  last host=nnaida11, => last addr=193.62.115.232, ; #panics => 0 ; Finished OK 
> ttyUSB20 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB21 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB22 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB23 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB3 =>  last host=nnaida12, => last addr=193.62.115.233, ; #panics => 0 ; Finished OK 
> ttyUSB4 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB5 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB6 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB7 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB8 =>  last  => last  ; #panics => 0 ; Not Finished 
> ttyUSB9 =>  last  => last  ; #panics => 0 ; Not Finished 
> 
> ttyUSB0
> ttyUSB1
> ttyUSB10
> ttyUSB11
> ttyUSB12
> ttyUSB13
> ttyUSB14
> ttyUSB15
> ttyUSB16
> ttyUSB17
> ttyUSB18
> ttyUSB19
> ttyUSB2
> ttyUSB20
> ttyUSB21
> ttyUSB22
> ttyUSB23
> ttyUSB3
> ttyUSB4
> ttyUSB5
> ttyUSB6
> ttyUSB7
> ttyUSB8
> ttyUSB9
> 
> Ssh'ed to RasPi. There are only four log files in ~/logs/
> 
> sudo lsusb -t results in
> 
> /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
>     |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/5p, 480M
>         |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
>         |__ Port 2: Dev 4, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 5, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 6, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 7, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 8, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 9, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 10, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 11, If 0, Class=vend., Driver=ftdi_sio, 12M
>         |__ Port 3: Dev 20, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 21, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 22, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 23, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 24, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 25, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 26, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 27, If 0, Class=vend., Driver=ftdi_sio, 12M
>         |__ Port 4: Dev 12, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 13, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 14, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 15, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 16, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 17, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 18, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 19, If 0, Class=vend., Driver=ftdi_sio, 12M
>         |__ Port 5: Dev 28, If 0, Class=hub, Driver=hub/4p, 480M
>             |__ Port 1: Dev 29, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 2: Dev 30, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 3: Dev 31, If 0, Class=vend., Driver=ftdi_sio, 12M
>             |__ Port 4: Dev 32, If 0, Class=hub, Driver=hub/4p, 480M
>                 |__ Port 1: Dev 33, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 2: Dev 34, If 0, Class=vend., Driver=ftdi_sio, 12M
>                 |__ Port 3: Dev 35, If 0, Class=vend., Driver=ftdi_sio, 12M

Can you let me know when you power-cycled the FEEs please ? Or maybe just do a stop/go.
If there is nothing to log from the consoles then the Pi won't have made a log file. 
The software opens and closes the log file each time a line of text comes in from the FEE64.
    Reply  Thu May 18 09:46:36 2017, TD, OH, CB, RasPi system console diagnostics 
> Can you let me know when you power-cycled the FEEs please ? Or maybe just do a stop/go.
> If there is nothing to log from the consoles then the Pi won't have made a log file. 
> The software opens and closes the log file each time a line of text comes in from the FEE64.

The most recent power cycle of the FEE64s was May 5

DAQ stop/go per your request - see Elog https://elog.ph.ed.ac.uk/AIDA/571
ELOG V3.1.3-7933898