11.00 Moved detector stack forward several cm. 1st DSSD now ~6cm from end of snout. Middle of stack now ~47cm from
base -> approx centre of BRIKEN.
      
      4 new 79cm ribbon cables (+kaptons) added to p-side of first two DSSDs.
      
      Bias and pulser re-connected to all relevant FEEs. 

      Detector order unchanged, i.e. DSSD1 has same serial number as in June 2016 expt. Same for DSSDs 2-6.
      
      DSSD-to-FEE map also unchanged.

15.45 R7 starts

      ASIC settings 2016Sep02-18.17.01

      BNC PB-4
       Amplitude 90,000, x2 attenuator IN
       Frequency c. 26Hz

      Master MACB
       mode 5
       50MHz external clock (rear panel)
       BNC PB-4 trigger output -> Ortec SCA -> LeCroy 429A -> Master MACB Correlation DAQ Accepted Trigger - not
working - firmware issue?

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change) *and* nnaida7, nnaida21
fail ADC calibration

      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, AIDA disc, nnaida6, nnaida5,
nnaida19 & nnaida20 statistics - see attachments 4-14

      Analysis of R7_0 - ADC data synchronised

16.08 Julabo FL11006 chiller set point +20 deg C actual +20.0 deg C
      Water level c. 60% M-H markers - topup to 100% with 1x jugful water
      Temperature +25.7 deg C dew point +7.7 deg C RH 31.4%

      DAQ continues OK

      ASIC settings 2019May19-17.54.03
      slow comparator 0xa

      BNC PB-4 Pulser 
      Amplitude 90,000
      x5 attenuator IN
      Frequency 25Hz range
      t_r 100ns, tau_d 50us
      - polarity
      Delay MIN, tail pulse

14.14 all system wide checks OK *except* nnaida7 fails ADC calibration

      detector biases & leakage currents OK - see attachments 1 & 2

      FEE64 temperatures OK - see attachment 3

      Check ASIC Control *all* FEE64s & ASICs

      good event statistics OK - see attachments 4
      11x <20k, 19x <50k, 21x <100k - cf. 10x <20k 21x <50k 24x <100k yesterday

      Merger OK - see attachment 5
      TapeServer OK - see attachment 6

      FEE64 Rate spectra - attachment 7
      Check ASIC Control
      FEE64 ASIC
      1     4
      10    1
      13    1
      14    1
      14    2
      22    3
      FEE64 Rate spectra - attachment 8 - some channels recovered

      1.8.L spectra - see attachments 9 & 10
      nnaida1 77 ch FWHM, nnaida3 82 ch FWHM - cf. 74 & 78 ch FWHM respectively yesterday
      p+n FEE64s 1.8.W spectra 20us FSR - attachment 11
      n+n FEE64s 1.8.W spectra 20us FSR - attachment 12
      FEE64 Rate spectra - attachment 9

      DAQ continues OK

      ASIC settings 2019May19-17.54.03
      slow comparator 0xa

      BNC PB-4 Pulser 
      Amplitude 90,000
      x5 attenuator IN
      Frequency 25Hz range
      t_r 100ns, tau_d 50us
      - polarity
      Delay MIN, tail pulse 

17.03 ADC data synchronised - attachment 13

      2x Pb blocks placed lengthways in aperture of the upstream (CH2) shield removed

      c. 8cm (1x std width) Pb thick Pb wall immediately downstream of F11 remains in position

      check light ions when beam returns and online group changes target

18.16 AA reports F11 rate c. 2.5kHz
      AIDA plastic rate c. 2.4kHz

      online sort sampling small fraction of data so write to disk and analyse all offline

18.22 zero all spectra
      Check ASIC Settings *all* FEE64s & ASICs

      DAQ start file May19/R15

      light ion background?

      ASIC settings 2019May19-17.54.03
      slow comparator 0xa

      BNC PB-4 Pulser 
      Amplitude 90,000
      x5 attenuator IN
      Frequency 25Hz range
      t_r 100ns, tau_d 50us
      - polarity
      Delay MIN, tail pulse

      good event statistics OK - see attachments 4
      11x <20k, 19x <50k, 24x <100k - cf. 11x <20k 19x <50k 21x <100k earlier

00.46 34Ne stopped for 36Ne setting (R19_37).
      DAQ kept running as I was upstairs getting some food.
      Stopped at R19_45.

00.57 BigRIPS change to 36Ne setting. Start file 0196. Still tuning.
      Temps ok.
      Leakage currents and bias ok (see attachment 1).

01.32 R20 started on 36Ne setting.
      BigRIPS file 0197 (started @ 01.30).
      BRIKEN file 

02.23 Beam stopped to F11. End of R20_12.

02.28 Beam returned to F11. Start of R20_14.
      BigRIPS file 0198.

02.48 BRIKEN file 48 started.
      Start of R20_19.
      All system checks passed except nnaida1 fails global clocks 6.
      Temps and leakage currents ok.

03:28 BigRips file 0199 (started at 03:27)
      AIDA at end file R20_28
      BRIKEN file 47 started
      All system wide checks ok.
      Temps and leakage currents ok

04:36 BigRIPS file 0200
      Coincides with end of AIDA file R20_46
      BRIKEN file 48
      All system wide checks ok
      Temps and leakage currents ok
      ASIC load check all ok 

04:59 BigRips run 0201 (Started at 4:45)
      Corresponding AIDA file for start of BigRips run is R20_49
      BRIKEN file 49
      Corresponding AIDA file for start of BRIKEn is R20_52

05:33 System wide checks all ok
      Temperatures, biases and leakage currents ok (seet attachment 2)

05:44 BigRips start run 0202
      Start of AIDA file R20_64
      BRIKEN file 50
      System wide checks passed
00.46 34Ne stopped for 36Ne setting (R19_37).
      DAQ kept running as I was upstairs getting some food.
      Stopped at R19_45.

00.57 BigRIPS change to 36Ne setting. Start file 0196. Still tuning.
      Temps ok.
      Leakage currents and bias ok (see attachment 1).

01.32 R20 started on 36Ne setting.
      BigRIPS file 0197 (started @ 01.30).
      BRIKEN file 

02.23 Beam stopped to F11. End of R20_12.

02.28 Beam returned to F11. Start of R20_14.
      BigRIPS file 0198.

02.48 BRIKEN file 48 started.
      Start of R20_19.
      All system checks passed except nnaida1 fails global clocks 6.
      Temps and leakage currents ok.

03:28 BigRips file 0199 (started at 03:27)
      AIDA at end file R20_28
      BRIKEN file 47 started
      All system wide checks ok.
      Temps and leakage currents ok

04:36 BigRIPS file 0200
      Coincides with end of AIDA file R20_46
      BRIKEN file 48
      All system wide checks ok
      Temps and leakage currents ok
      ASIC load check all ok 

04:59 BigRips run 0201 (Started at 4:45)
      Corresponding AIDA file for start of BigRips run is R20_49
      BRIKEN file 49
      Corresponding AIDA file for start of BRIKEn is R20_52

05:33 System wide checks all ok
      Temperatures, biases and leakage currents ok (seet attachment 2)

05:44 BigRips start run 0202
      Start of AIDA file R20_64
      BRIKEN file 50
      System wide checks passed

6:15 Operators call...R20_72 1.5 GB
     The BigRIPS system display PC looks like it crashed

6:22 2017-04-01 06:21:19 BigRIPS run 203 starts
      AIDA run 20_74 is about 25% started
      BRIKEN run 51

6:40 Phone rings, AIDA run R20_79 just started
      Stripper foil seems to have been changed

6:43 2017-04-01 06:42:03 BigRIPS starts a new run     
     R20_79 is 75% the way through.
     BRIKEN run 52
     System checks passed.

7:09 PPACs are discharging very frequently.

7:39 counted 8 discharges.  It's about once in four minutes.  An acceptable discharge rate is several times in
one shift.
     Firstly, each discharge damages the PPAC.  Even 10 to 20 discharges can easily cause dead regions of cm size.
     Secondly, with the frequency of discharge, we are now missing like as much as 25% of all events (at least
at a given PPAC)
     As the detector becomes more damaged, it can discharge more frequently.
     Moving the detector's position slightly can help to alleviate the problem by putting the beam on a fresh region
     Anyway, as for a parasitic run they can do whatever they want, but it seems they are mis-operating the PPACs
     Basically the voltage is too high or the detector is too old and needs to be replaced.

7:40 2017-04-01 07:40:12 BigRIPS run 205 begins
     AIDA run R20_94 is at 721 MB at 7:40
     BRIKEN run 53
     System-wide checks are okay

8:08 Bias and Temp attachments 3 & 4 (bias_1_.png and temp_1_a.png)

9:01 BigRIPS run 206 2017-04-01 09:01:06
     AIDA run R20_115 is at 600 MB at 9:01
     BRIKEN run is 54
     System-wide checks

9:39 Now that the students went home and some more experienced beam physicists are on shift, the PPAC discharge rate
goes down to ~2x per hour.

10:02 New BigRIPS run at 2017-04-01 10:01:40 number 0207
     AIDA run R20_130 was at 1.9 GB at 10:02..
     BRIKEN run 55
     System-wide checks passed

(I went upstairs for a break...it seems that the beam got turned off at some point)

10:55 Phone call, and I hear the chime for the beam coming back.

10:56 BigRIPS starts run 0208 2017-04-01 10:56:06
      AIDA run R20_145 has just started at 10:56
      BRIKEN run was not restarted and continued as run 55
      System-wide checks are okay

11:02 We checked that the synchronization looks okay.

11:58 Noticed that BigRIPS started a new run at 2017-04-01 11:56:42, run 0209
     AIDA run R20_160 should overlap about 50% with this run.
     BRIKEN run is 56
     System wide checks performed.
     Attached are Biases and Temperatures as 5 & 6 ("bias_1_b" ; "temp_1_b")

12:58 BigRIPS starts run 0210 at 2017-04-01 12:57:21
     AIDA run R20_176 was 422 MB at 12:57
     BRIKEN 57
     Pass system-wide checks

14:30 beam current has been fluctuating for awhile

14:50 BigRIPS started a new run 0212 2017-04-01 14:44:18
     R20_203 was at 1.2 GB at 14:44
     BRIKEN run 58
     System checks okay

15:18 Phone rings.  RF is down so the beam stopped.
     R20_212 is at 500 MB at 15:19

15:57 Phone rings again.

16:37 BigRIPS start run 0214
      Start of AUDA file R20_232
      BRIKEN run 59
      System wide check all ok

16:53 Biases and leakage currents ok (See attachment 7)

17:36 BigRips start run 0215
      Strt of AIDA file R20_247
      BRIKEN start run 60
      System wide checks all ok
      Temperatures all ok

18:04 BigRips stop to change gas stripper foil
      AIDA file R20_254

18:09 BigRips start run 0216
      AIDA file R20_255
      BRIKEN start file 61
      System wide checks all ok
6:15 Operators call...R20_72 1.5 GB
     The BigRIPS system display PC looks like it crashed

6:22 2017-04-01 06:21:19 BigRIPS run 203 starts
      AIDA run 20_74 is about 25% started
      BRIKEN run 51

6:40 Phone rings, AIDA run R20_79 just started
      Stripper foil seems to have been changed

6:43 2017-04-01 06:42:03 BigRIPS starts a new run     
     R20_79 is 75% the way through.
     BRIKEN run 52
     System checks passed.

7:09 PPACs are discharging very frequently.

7:39 counted 8 discharges.  It's about once in four minutes.  An acceptable discharge rate is several times in
one shift.
     Firstly, each discharge damages the PPAC.  Even 10 to 20 discharges can easily cause dead regions of cm size.
     Secondly, with the frequency of discharge, we are now missing like as much as 25% of all events (at least
at a given PPAC)
     As the detector becomes more damaged, it can discharge more frequently.
     Moving the detector's position slightly can help to alleviate the problem by putting the beam on a fresh region
     Anyway, as for a parasitic run they can do whatever they want, but it seems they are mis-operating the PPACs
     Basically the voltage is too high or the detector is too old and needs to be replaced.

7:40 2017-04-01 07:40:12 BigRIPS run 205 begins
     AIDA run R20_94 is at 721 MB at 7:40
     BRIKEN run 53
     System-wide checks are okay

8:08 Bias and Temp attachments 3 & 4 (bias_1_.png and temp_1_a.png)

9:01 BigRIPS run 206 2017-04-01 09:01:06
     AIDA run R20_115 is at 600 MB at 9:01
     BRIKEN run is 54
     System-wide checks

9:39 Now that the students went home and some more experienced beam physicists are on shift, the PPAC discharge rate
goes down to ~2x per hour.

10:02 New BigRIPS run at 2017-04-01 10:01:40 number 0207
     AIDA run R20_130 was at 1.9 GB at 10:02..
     BRIKEN run 55
     System-wide checks passed

(I went upstairs for a break...it seems that the beam got turned off at some point)

10:55 Phone call, and I hear the chime for the beam coming back.

10:56 BigRIPS starts run 0208 2017-04-01 10:56:06
      AIDA run R20_145 has just started at 10:56
      BRIKEN run was not restarted and continued as run 55
      System-wide checks are okay

11:02 We checked that the synchronization looks okay.

11:58 Noticed that BigRIPS started a new run at 2017-04-01 11:56:42, run 0209
     AIDA run R20_160 should overlap about 50% with this run.
     BRIKEN run is 56
     System wide checks performed.
     Attached are Biases and Temperatures as 5 & 6 ("bias_1_b" ; "temp_1_b")

12:58 BigRIPS starts run 0210 at 2017-04-01 12:57:21
     AIDA run R20_176 was 422 MB at 12:57
     BRIKEN 57
     Pass system-wide checks

14:30 beam current has been fluctuating for awhile

14:50 BigRIPS started a new run 0212 2017-04-01 14:44:18
     R20_203 was at 1.2 GB at 14:44
     BRIKEN run 58
     System checks okay

15:18 Phone rings.  RF is down so the beam stopped.
     R20_212 is at 500 MB at 15:19

15:57 Phone rings again.

16:37 BigRIPS start run 0214
      Start of AUDA file R20_232
      BRIKEN run 59
      System wide check all ok

16:53 Biases and leakage currents ok (See attachment 7)

17:36 BigRips start run 0215
      Strt of AIDA file R20_247
      BRIKEN start run 60
      System wide checks all ok
      Temperatures all ok

18:04 BigRips stop to change gas stripper foil
      AIDA file R20_254

18:09 BigRips start run 0216
      AIDA file R20_255
      System wide checks all ok

18:25 BRIKEN start file 61
      AIDA file R20_259
      BRIKEN had to restart the computer so a resync needed to be done

19:07 BigRips start file 0217
      AIDA on file R2-_270 (End of file)
      BigRips reducing beam intensity to calibrate the beam current

19:12 Back to normal BigRips condition
      Al degrader 1mm is removed
      BigRips run 0218 (started at 19:11)
      AIDA file R20_271
      BRIKEN start file 62
      System wide checks ok

20:06 Bias and leakage currents ok
      Temperature ok

20:12 BigRips start run 0219
      Start of AIDA file R20_287
      BRIKEN starts file 64
      System wide checks all ok
      ASIC load check ok

21:12 BigRips start run 0220
      AIDA file R20_301
      System wide checks ok
      BRIKEN starts file 65

22:12 BigRips stopped to change stripper foils
      End of AIDA file R20_315

22:20 BigRips start run 0220
      End of AIDA file R20_317
      System wide checks and temperatures ok
      BRIKEN start file 66

23:08 Biases and leakages ok

23:10 BigRips start run 0222
      End of AIDA file R20_329
      BRIKEN start file 67
      System wide checks all ok

21/02/15

1230: Developing the software to check synchronisation between DAQs. Below line runs program on aidas1 - sends data from TapeServer to BELEN.

            ./DataRelayFilter -n 10.32.6.196 -p 10307 -I 2

1500: Investigating pulser peak FWHMs as function of bias voltage.

           Constant shaping time of 0x3, slow comp thresh 0x20, fast comp thresholds of 0xff. All other setting as standard. Detector MSL BB18 2977-20.

           Results summarised in table directly below.

             At bias of -200V nnaida1+6 stopped outputting data but continued to produce SYNCs. Good event rate dropped from ~few thousand per sec to ~300 per sec.

             After returning bias to -150V, nnaida1+6 started producing data again. No effect was seen in nnaida3+8.

             nnaida8 peaks are not particularly Gaussian in shape. Look more skewed with one rolling edge and one side a relatively sharp drop.

            No obvious trends with changing bias voltage. Will return to -100V.

1545: Investigating pulser peak FWHMs as function of shaping time.

           Bias returned to -100V, with leakage current 11.000uA.

           Settings for all nnaida: slow comp thresh = 0x20, fast comp thresh = 0xff, hold time = 0x4 if shaping time <= 0x8, 0x8 if shaping time > 0x8.

           Visual 'analysis' suggested 0xb to be the optimal value so used that as shaping time in further tests.

           (Graph to follow)

1900: Investigating effect of changes to slow comparator thresholds.

           Settings: bias = -100V, hold timing = 0x8, shaping time = 0xb, fast comp thresholds = 0xff. Otherwise, standard settings.

          With threshold at 0xd, threshold position was very hard to determine for nnaida6+8, assume discirminator is running in the noise. Will ignore data for 0xd.

          Assuming central position of 32768 and energy per channel of 0.7 keV, below table gives approximate position of threshold in keV.

           Thus (very roughly), between 0x10 and 0x20, the above table shows the threshold moving by about 10 keV per increment in the threshold.

           With the threshold at 0x10, the data rate show on the TapeServer window was ~3Mb/s (uncompressed?).

           This was with 1 detector (BB18 2977-20) connected to 4 FEE64s with above settings, a 25Hz pulser and 241Am+207Bi sources (activities in ELog entry 40).

           The Good Events rate averaged around 30,000-60,000/s per FEE64 card, with only a few Pause/Resumes coming through.

           Example stats screen and hit patterns included below for final configuration.

10:40    Put lead wall in front of AIDA during beam tuning.

12.30 BNC PB-4 Pulser OFF
      AIDA file RINF128/R7_998

12.31  E17/F11 ambient temperature +24.7 deg C, RH 42.2%, d.p. +10.7 deg C
       Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 70%)

12.53 analysis RIBF128/R7_1002 <0.5% dead-time, zero ts & MBS timewarps - see attachment 1

13:12 R7_1006 is the last run with 100% no pulser.  Pulser turned on around 400 MB into R7_1007

BNC PB-5 Pulser

Fall time 1.0ms
Rate 100Hz
Delay 250ns
Ampl 5.0V
Polarity +
Pulse Top Tail
Atten 10x 
PB-5 Pulse ON
Clamp ON

- polarity via Cooknell SA1 Summing Amplifier

nnaida11 & 12 AIDA adaptor rev C (280114) LK1 jumper fitted
nnaida13 & 14 AIDA adaptor rev B (180713)

Heavy duty Cu braid and large clips connect BNC connectors of 
cabling to test inputs of both adaptor PCBs.

Heavy duty Cu braid connects Cu mezzanines of nnaida11 & 14

Standard ASIC settings
nnaida11 & 12 - input, nnaida13 & 14 + input
shaping time 8us
fast discriminators 0xff
slow comparator 0x40

***

Zero input load 

         nnaida11 nnaida12 nnaida13 nnaida14
1.4.L    14.63    14.88    15.95    13.03
1.6.L    14.35    15.01    13.47    12.62
1.8.L    14.86    14.54    12.88    13.43
1.10.L   14.67    13.76    13.03    13.43

See attachments 1-3

***

Left hand coupler (011215) nnaida11 & 14 (only)
No EMC shielding or grounding

         nnaida11 nnaida12 nnaida13 nnaida14
1.4.L    33.64    16.02    15.34    18.74 
1.6.L    23.92    14.92    12.98    14.00
1.8.L    17.09    14.36    13.07    13.63
1.10.L   16.66    14.67    13.27    13.92

Left hand coupler (011215) nnaida11 & 14
No EMC shielding or grounding

***

Left hand coupler (011215) + 1x 45cm 34-way ribbon cable nnaida13
Left hand coupler (011215) + 1x 90cm 34-way ribbon cable nnaida12
No EMC shielding or grounding

         nnaida11 nnaida12 nnaida13 nnaida14
1.4.L    30.37    19.41    23.02    21.15
1.6.L    21.64    16.99    17.73    15.82
1.8.L    16.85    16.43    14.14    14.78
1.10.L   16.31    16.70    13.07    13.75
3.4.L    16.13    650      122.61   14.76                  
3.6.L    15.98    710      140.07   14.32
3.8.L    16.53    610      111.10   14.36
3.10.L   17.05    630      121.59   14.26

See attachment 4 - are these actually 3*W waveforms?

***

Left hand coupler (011215) + 1x 45cm 34-way ribbon cable nnaida11
Left hand coupler (011215) + 1x 90cm 34-way ribbon cable nnaida12

Left hand coupler (011215) + 1x 90cm 34-way ribbon cable nnaida14
Left hand coupler (011215) + 1x 45cm 34-way ribbon cable nnaida13

No EMC shielding or grounding

         nnaida11 nnaida12 nnaida13 nnaida14
1.4.L    27.44    18.70    21.79    20.32
1.6.L    19.84    16.90    17.06    16.06
1.8.L    16.20    15.73    14.06    19.34
1.10.L   16.23    16.42    14.27    17.11
3.4.L    146.47   510      91.03    390                 
3.6.L    160.69   555      105.11   640
3.8.L    142.32   490      85.85    370
3.10.L   144.06   510      93.30    390

See attachments 5-13

***

Shaping time 2us

         nnaida11 nnaida12 nnaida13 nnaida14
1.4.L                               19.60
1.6.L                               15.67
1.8.L                               17.68
1.10.L                              15.88
3.4.L    160.95   560      90.85    550                 
3.6.L    173.91   600      108.19   865
3.8.L    155.90   535      88.35    460
3.10.L   162.21   560      95.45    540

A table summarising the run information within the lab book and ELog for Apr-May.

AIDA was relocated, and now I want to power it on to do some tests.  However, I cannot get the Rly16 service running.

Specifically, after powering up the entire AIDA system, on aidas1 PC, I cannot connect to the raspberry pi (nnrpi1) via a web browser

http://nnrpi1:8015/AIDA/Rly16/

The following elog may be relevant: https://elog.ph.ed.ac.uk/AIDA/58

Now I am ssh'd to nnrpi1 via aidas1

% dmesg | grep -A 3 USB0
[    8.970098] usb 1-1.3: pl2303 converter now attached to ttyUSB0
[    9.012479] usb 1-1.2.2: Detected FT232RL
[    9.178208] usb 1-1.2.2: FTDI USB Serial Device converter now attached to ttyUSB1
[    9.348288] ftdi_sio 1-1.2.3:1.0: FTDI USB Serial Device converter detected

So, I will try Patrick's workaround in elog #58

I tried it, though it's hard to know if it is done correctly.  Which item is which?  Or more importantly, which one runs Rly16?

[    8.107290] usb 1-1.2.3: USB disconnect, device number 8
[    8.261338] usbserial: USB Serial support registered for FTDI USB Serial Device
[    8.541012] ftdi_sio 1-1.2.2:1.0: FTDI USB Serial Device converter detected
[    8.712409] usb 1-1.2.2: Detected FT232RL
[    8.847955] usb 1-1.2.2: FTDI USB Serial Device converter now attached to ttyUSB0
[   12.252576] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   12.728966] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   23.695562] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   25.212043] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC1E1
[   32.372015] Adding 102396k swap on /var/swap.  Priority:-1 extents:2 across:2134012k SSFS
[  109.219022] usb 1-1.2.3: new full-speed USB device number 9 using dwc_otg
[  109.335538] usb 1-1.2.3: New USB device found, idVendor=0403, idProduct=6001
[  109.335579] usb 1-1.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  109.335601] usb 1-1.2.3: Product: USB <-> Serial
[  109.335619] usb 1-1.2.3: Manufacturer: FTDI
[  109.354420] ftdi_sio 1-1.2.3:1.0: FTDI USB Serial Device converter detected
[  109.354761] usb 1-1.2.3: Detected FT232BM
[  109.356174] usb 1-1.2.3: FTDI USB Serial Device converter now attached to ttyUSB1
[  163.939476] usb 1-1.3: new full-speed USB device number 10 using dwc_otg
[  164.042348] usb 1-1.3: New USB device found, idVendor=067b, idProduct=2303
[  164.042424] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  164.042446] usb 1-1.3: Product: USB-Serial Controller D
[  164.042468] usb 1-1.3: Manufacturer: Prolific Technology Inc.
[  164.096379] usbcore: registered new interface driver pl2303
[  164.098786] usbserial: USB Serial support registered for pl2303
[  164.099085] pl2303 1-1.3:1.0: pl2303 converter detected
[  164.106666] usb 1-1.3: pl2303 converter now attached to ttyUSB2

On ttyUSB# have:
FT232RL
FT232BM
pl2303


% lsusb
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 010: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 005: ID 046d:c016 Logitech, Inc. Optical Wheel Mouse
Bus 001 Device 007: ID 04f2:0402 Chicony Electronics Co., Ltd Genius LuxeMate i200 Keyboard
Bus 001 Device 006: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 009: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC

N.B. USB-serial link to FEE64 is not connected to anything on the serial side.  What is the FEE64 Console?  See Attachment 1.  It goes to nothing!  Should it go somewhere?

08/02/16

Spoke to PJC-S -> Ethernet cables from FEEs to switch are all 4m Cat6.

System powered on with bias to detector of +200V and leakage current of 6.4uA at room temp 19.0oC.

Double peaking present again in nnaida11/12 -> individual peaks @ ~100ch FWHM with separation ~150ch @ 2us shaping time.

Double peaking also present in nnaida13/14 -> indiv peaks ~100ch FWHM and separation ~90ch.

Changing grounding had no effect.

Made ribbon cable with 2 separately shielded halves plus connecting Cu tape strip at FEE and + drain wire in contact with both halves - single peaks - widths of ~110-160ch.

09/02/16

Added conductive gasket to Cu mount blocks

      - resistance between Cu and gasket ~0.2 Ohms (probe-to-probe = 0.16 Ohms)

      - LEMO connector to Cu mount block ~0.35 Ohms

      - LEMO to LEMO (between adapter PCBs) ~0.55 Ohms (connected by heavy duty Cu braid)

Powered system on. Bias of +200V and leakage current = 5.5uA @ room temp 19.5oC.

With gasket added and heavy duty Cu braid between mount blocks, pulser FWHM in nnaida11 ~ 62ch and in nnaida13 ~72ch at shaping time of 8us.

Investigated effect of shaping time on peak FWHM (avg width found from average of 1.(6,8,10,12).L for each FEE.     (see attachments)

    Shaping time (us)    |     nnaida11    |     nnaida12    |     nnaida13     |     nnaida14    

                0.5                   |          95*          |           108*       |          126*        |          99*

                1.0                   |          82            |           116**     |          130^        |         110**  

                2.0                   |           68           |            83^        |           110^       |           92^

                4.0                   |           60           |            77          |             85'        |           90

                8.0                   |            56          |             63         |              77        |            64

     * = all peaks single and Gaussian looking

     ** = not all peaks Gaussian, some flat or sloped top 

     ^ = double peaks, widths estimated from tallest peak

     ' = flat topped

01/02/16

Started up system. Biased det to +200V 

 25/01/16

On start up, received error when trying to access spectrum browser (attachment 1)   => required restart of 'Http for AIDA'

At 2us shaping time, and all other setting as last week, the situation remains the same

    - nnaida11 has width ~50ch, nnaida12 has width ~300ch, nnaida13 is quad peaked and nnaida14 is not quite Gaussian but has width of ~80ch (possibly two peaks very close together)

Change shaping time to 8us.

Swap cables to see if the cable is the source. Currently have two LH coupler + shielded ribbon cables connected to nnaida11/13 (11 good performance, 13 quad peaked)

     - swapped the ribbon cables over but left the couplers where they were to see if the good performance is a consequence of something intrinsic to a particular ribbon or coupler.

     - nnaida11 now has width ~40ch                  (attachment 2)

     - nnaida13 now back to double peaked       (attachment 3)

     - inconclusive.

Could old style kapton laid next to nnaida11 be shielding it due to the built in screen? (the kapton is laid against the ribbon cable)

     - removing the kapton makes no difference to widths, other than nnaida12 which comes down to ~15ch (as expected)

Clipping cable ends together makes no difference, and no combination of jumpers could improve the widths.

Bending cables through 90 degrees makes no difference either (if they were acting as aerials).

Swapped complete PCBs with cables still in place over. Test first thing tomorrow.

26/01/16

Swapping PCBs produced no difference

     - nnaida11 has widths of ~50-60ch, although some structure visible in some peaks

     - nnaida14 sees double peaks.

Clipping ribbon ends together now makes things worse.

Swapped PSU cables to nnaida11/14

     - nnaida11 now marginally worse @ ~70ch width

     - nnaida14 sees no obvious change

Connected PSU chasis to NIM crate - no change.

Added two more ribbon + HL/RH couplers to system. Now:

      - nnaida11 = LH + ribbon

      - nnaida12 = LH + ribbon + RH

      - nnaida13 = LH + ribbon + RH

      - nnaida14 = LH + ribon

Clipped drain wire to LEMO connectors on PCB instead of pulser BNC cable.

See marked improvement in performance. All nnaida now have single Gaussian peaks with widths now:   (attachments 4-6)

       - nnaiada11 at ~35ch

       - nnaida12 at ~70ch

       - nnaida13 at ~80-120ch

       - nnaida14 at ~45-55ch (3 at ~80-150ch)

Added RH coupler to nnaida11/14, with thin braid taped over joint in shielding to reduced resistance

       - nnaida11 widths increase slightly to ~40-60ch

       - nnaida14 widths increase slightly to ~60-80ch, few round ~100-120

This is consistent with the RH coupler increasing widths by ~25ch, as seen in previous tests.

See occasional ripple through waveforms in nnaida11/14    (attachments 7-11).

      - expanded view shows period of 34 samples  ==> 680ns   ==> frequency ~1.5MHz

Variation with shaping time (avg peak width over a several channels):

* = some non-Gaussian peaks

Connected detector, biased to +200V (+ve core to nnaida12, -ve braid to nnaida14) with leakage current = 9uA.

At 8us shaping time, and with Lk1 fitted on nnaida13/14 RevC adapter, peaks across all nnaida have widths ~70ch and are all single Gaussian peaks.   (attachments 12-15)

 18/01/16

Set up as last week but with 8us shaping time.

With detector connected, still seeing rail-to-rail noise on waveforms in nnaida13 and double peaking in all nnaida.

Measured various resistances around the system to gauge quality of grounding connections (all measured in Ohms).

     - NIM crate to PCB LEMO connector:                                        4.8

     - NIM crate to mezzanine Cu block:                                           5.3    -> Plastic guide pins now installed.

     - Between LEMO connectors on the adapter boards:           0.19

     - nnaida11/12 drain wire to nnaida13/14 drain wire:            0.25

     - RH coupler nnaida13 to RH coupler nnaida11:                  1.4

     - LEMO to ribbon shielding nnaida13:                                     0.2

     - LEMO to RH coupler shielding nnaida13:                            0.4

Double peaking not seen in RIKEN or by PCS during his tests -> most likely pickup of external noise.

Connected multiple different cable types to see effect they have on the sinusoidal wave seen in the waveforms and on the double peaking.

       - nnaida11: LH coupler (not shielded) + 45cm ribbon (shielded) + RH coupler (shielded)

       - nnaida12: LH coupler (not shielded) + 45cm ribbon (shielded)

       - nnaida13: old style 90cm kapton (19/02/14 revision)

       - nnaida14: 45cm thin kapton (shielded) with foil over loose end

Setting as last week, with 2us shaping time.

Peak -to-peak differences in all nnaida for several channels:

               |   nnaida11    |                       nnaida12                    |   nnaida13   |   nnaida14    

    1.4.L  |       157          |                           123                           |         160        |   single peak of width ~400ch

    1.6.L  |      119           |                            111                          |         170        |       "                             "

    1.8.L  |        72           |  Flat top single peak, width ~35    |         155        |      "                             "

   1.10.L |       90            |    "                                             " 42    |         177        |       "                             "                 -> Noticed after cable gnd jumper was not connected on nnaida14 and needs to be for these kaptons. Explains v. large peak width. (Attachments 3-5)

19/01/16

Swapped cables on nnaida11 and nnaida13. Now have:

       - nnaida11: old style 90cm kapton (19/02/14 revision)

       - nnaida12: LH coupler (not shielded) + 45cm ribbon (shielded)

       - nnaida13: LH coupler (not shielded) + 45cm ribbon (shielded) + RH coupler (shielded)

       - nnaida14: 45cm thin kapton (shielded) with foil over loose end

In nnaida13 see ripple on baseline of waveform with period ~30ch = 600ns (@20ns per ch) => frequency of 1.67 MHz.  (Attachment 6+7)
 

Resistance between drain wire end and ribbon shielding on nnaida13 cabele:   ~0.8 Ohms.

        "                                                          "     RH coupler shielding on nnaida13:    ~1.1 Ohms.

For shielded ribbon cable on nnaida13, see fairly flat waveforms. High frequency transients pass through, but no sign of sinusoidal wave over time window of 6us.   (Attachment 9)

After swapping cables on nnaida11 and nnaida13 (only change to system), now see single, Gaussian peaks in nnaida12 with widths ~40ch.

For nnaida12:

           Shaping time (us)     |     Avg peak width (ch)        (avg of 1.8/9/10/11.L peak widths)

                      0.5                     |                   35

                      1.0                     |                   39      (1.11 width ~50ch, all others ~37-42)

                      2.0                     |                   39      (   "                                                      "     )

                      4.0                     |                   35      (1.11 width ~45ch, all others ~32-38)

                      8.0                     |                   37

At 8us shaping time: in nnaida11 (old style cable), see single pulser peak of width ~150 ch.

                                           nnaida13, still see double peaking with peak separation of ~90ch and an extra structure, like a small shoulder, on the outside of the peaks.    (Attachments 8+10)

                                           nnaida14, not fully double peaked, more a peak with a shoulder ~40ch apart.

At 2us shaping time: in nnaida11, pusler peak width back to ~300ch.

                                           nnaida13, retains extra structure on outside of peaks, but peak searation increased to ~150ch.

                                           nnaida14, double peaked, with peak separation of ~65ch.

(Attachments 11-15)

Cannot see waveforms on nnaida12, so swapped cables with nnaida11. Also acts as test of whether the good performance seen in nnaida12 follows the cables or is something to do with the FEE.

Now have:

       - nnaida11: LH coupler (not shielded) + 45cm ribbon (shielded)

       - nnaida12:  old style 90cm kapton (19/02/14 revision)

Now see some Gaussian peaks in nnaida11 with width of ~60ch @2us shaping time, however some peaks seem to have a slight shoulder.

      > suggests issue causing double-peaking is still present (and likely was in nnaida12), but is more/less apparent in some channels than others.

Questions:

     ** Is this just a very well shielded cable, or something to do with the intrinsic properties of this specific cable?

     ** What happens if a RH coupler is added?

     ** What happens if the RH coupler is removed from the cable connected to nnaida13?

Added RH coupler to cable on nnaida11.

Measured some resistances of the shielding on the nnaida11 (good performance - single peaks) cable and the nnaida13 (double peaks) cable:

       - drain wire to far end of shielding:      nnaida11 = 0.2 Ohms           nnaida13 = 0.2 Ohms

       - drain wire to RH coupler:                    nnaida11 = 0.4 Ohms            nnaida13 = 0.5 Ohms

       - far end of nnaida11 cable shielding to far end of nnaida13 cable shielding:     0.05 Ohms   (this seems wrong considering pushing the multimeter probes together gets 0.14 Ohms, but was repeated several times and was always 0.05 Ohms)

With addition of RH coupler to nnaida11 (and shaping time still at 2us):       (Attachments 16+17)

     - nnaida11 peaks show slight shoulder and increased width of ~110ch.

     - nnaida12 now sees noise over full-scale range

     - nnaida13 now quad peaked.

     - nnaida14 still double peaked, but peak separation increased to ~150ch.

Removed RH coupler from nnaida11, all peaks go back to how they were previously, before its addition (Something is repeatable - excellent!)

     - some peaks in nnaida11 now non-Gaussian nad with widths ~130ch. Most Gaussian and width ~50ch.

Removed RH coupler from nnaida13:

    - no real change in the spectra of any nnaida    -> what is the difference between the cables on nnaida11/13???

Stock take of ribbon cables currently at DL:

     - 90cm                   = 16

     - 45cm                   = 16   (4 with me for testing in Edinburgh this week. Will be returned to DL next week)

     - LH couplers       = 10   (2 with me for testing in Edinburgh this week. Will be returned to DL next week)

     - RH couplers       = 10  (2 with me for testing in Edinburgh this week. Will be returned to DL next week) 

 12/01/16      (Attachments 1-8)

FEEs set up as per ELOG entry 136 (15/12/15)

Connected cables as follows:

     - nnaida11 > LH coupler + 45cm ribbon (unshielded)

     - nnaida12 > no load (no waveforms from this card)

     - nnaida13 > LH coupler + 45cm ribbon (unshielded) + RH coupler

     - nnaida14 > LH coupler

Peak widths (ch):         <all with shaping time of 8us and heavy duty Cu braid between test inputs on each adapter card>

                     nnaida11  |  nnaida12  | nnaida13  |  nnaida14

     1.4.L   |        392       |       14.7       |      250       |      34.3 

     1.6.L   |       300        |       13.6       |      150       |      23.1

     1.8.L   |       290        |       13.0       |       123      |      16.8

    1.10.L  |       309        |       13.2       |       129      |  no output

     > widths greater than expected

     > thought it could possibly be due to pickup of external noise in the room so added Cu tape as shielding to cable connected to nnaida13 with drain wire connected to CU braid on pulser input

     > reduced peak with in nnaida13 to ~160ch

     > connecting far end of Cu shielding via a Cu braid to main Cu braid reduces this to ~130ch

     > connecting Cu shielding on RH coupler to main ribbon cable shielding with thin braid reduced this further to ~110ch

For nnaida13, see double peaking of pulser peak with all shaping times up to 2us. At 4us, see single peak of width ~160ch. At 8us, see single peak of width ~110ch.

13/01/16      (Attachments 9-20)

Set up system as yesterday.

With shaping time of 2us, nnaida13 does not show Gaussian peaks. Some peaks are double, others just single non-Gaussian peaks.

Yesterday peak-to-peak difference was ~150ch

    - added tin foil to loose coupler end and over connection between ribbon cable and RH coupler, as it was suspected it could be pickup of external noise

    - @ 2us shaping time, double peaks still present with p-2-p difference of ~130ch

    - clipped foil firmly to Cu shielding tape  >  no difference.

In nnaida14, see single roughly Gaussian peak of width ~15ch.

In nnaida11, see single peaks of widths ~250ch. At 8us this reduces to ~180ch.

Connecting the Lk7 jumper for nnaida13 produces no difference.

Added full LH coupler + 45cm ribbon + RH coupler to all nnaida, with full shielding > no improvement.

Connected all to detector and saw no improvements, in fact increased noise in nnaida13 dramatically (see waveforms)

See low frequency wave in waveforms, will work next week to try remove this as this could be what is causing the double peaking and hiding the true capabilities of the cables.

Original system configuration before my arrival: nnaida11 - 2b:22:6e
                                                 nnaida12 - 2b:33:15
                                                 nnaida13 - 2b:09:ce
All on firmware version 0x1900c1f.

With 8us shaping time and pulser amplitude @ 5V, FWHM [ch] were:

   Ch   | nnaida11 | nnaida12 | nnaida13 | nnaida14
  1.4.L |   38     |    33    |   38     |   34
  1.6.L |   24     |    25    |   28     |   23
  1.8.L |   20     |    17    |   17     |   16
 1.10.L |   21     |    17    |   17     |   16

Slightly higher than expected for a few channels, but FWHM fall within ~16-40ch for all channels in all FEEs.
All system checks passed.

nnaida13/14 kept constant throughout as control.

(1) Swapped nnaida11/12: nnaida11 -> 2b:33:0c     nnaida12 -> 2a:d0:1a
    Initially failed SYNC error test - consequence of old firmware version.
    Updated firmware from 0x10300c12 to 0x1900c1f.
    All system tests passed.
    FWHM on 8us shaping time and same pulser settings.

     Ch   | nnaida11 | nnaida12 | nnaida13 | nnaida14
    1.4.L |   37     |    40    |   42     |   35
    1.6.L |   31     |    28    |   29     |   23
    1.8.L |   22     |    18    |   15     |   15
   1.10.L |   23     |    20    |   16     |   15

(2) Swapped nnaida11/12: nnaida11 -> 2a:b8:90     nnaida12 -> 2a:b6:45
    Updated firmware from 0x10300c12 to 0x1900c1f.
    All system checks passed.
    No ASIC temp in nnaida11.
    FWHM on 8us shaping time and same pulser settings.

     Ch   | nnaida11 | nnaida12 | nnaida13 | nnaida14
    1.4.L |   40     |    32    |   40     |   34
    1.6.L |   29     |    25    |   29     |   23
    1.8.L |   21     |    18    |   17     |   16
   1.10.L |   21     |    18    |   17     |   16

(3) Swapped nnaida11/12: nnaida11 -> 2a:ed:9a     nnaida12 -> 2b:11:c5
    Updated firmware from 0x10300c12 to 0x1900c1f.
    All system checks passed.
    No ASIC temp in nnaida12.
    FWHM on 8us shaping time and same pulser settings.

     Ch   | nnaida11 | nnaida12 | nnaida13 | nnaida14
    1.4.L |   44     |    36    |   40     |   35
    1.6.L |   30     |    27    |   29     |   24
    1.8.L |   22     |    18    |   16     |   17
   1.10.L |   23     |    18    |   17     |   16


The above were all the repaired modules and seem to be operating normally.
Following are the 2 modules for which no fault report was provided. I only had time to complete testing on one, the other remains unknown.

(4) Swapped nnaida11/12: nnaida11 -> 2a:ed:8f     nnaida12 -> 2b:09:da
    Updated firmware from 0x10300c12 to 0x1900c1f.
    nnaida11 failed clock status tests. Tried 'Perform resync' from Master Timestamp window with no effect.
    No SYNC errors reported. All other system tests passed.
    nnaida12 missing lots of channels but all system checks passed.
    FWHM on 8us shaping time and same pulser settings.

     Ch   | nnaida11 | nnaida12 | nnaida13 | nnaida14
    1.4.L |   46     |    46    |   41     |   36
    1.6.L |   31     |    26    |   28     |   24
    1.8.L |   22     |    19    |   17     |   16
   1.10.L |   22     |    20    |   17     |   16


**

Overall, the first three modules which have been repaired seem to be operating normally. The fourth seems not be and the last module remains unknown.
Ch 1.4.L displays greater than expected FWHM throughout in all nnaida11/12, but FWHM were largely constant (to within a few channels) throughout and all FWHM fell in the range ~15-40ch apart from 1.1.L in *all* nnaida.

During R30 ( see https://elog.ph.ed.ac.uk/AIDA/154 ) obtain the following merger error messages
(rate ~ 10 per minute) with merger rate ~100-200k data items/s 

Merge: Timestamp (611b2080877)[Q 2, Stream 2] is older than previous (611c207f4ef) [Q 2, Stream 2]
Merge: Attempting resynchronisation to timestamp 611c207f4f0 or greater
Merge: Last sync pulse time: 611c20400a0
Merge: Last timestamp seen: 611c207f4ef
Merge: Restarting at timestamp 611c72800a0
Merge: Timestamp (611f6f0101f)[Q 2, Stream 2] is older than previous (61206effe27) [Q 2, Stream 2]
Merge: Attempting resynchronisation to timestamp 61206effe28 or greater
Merge: Last sync pulse time: 61206ec00a0
Merge: Last timestamp seen: 61206effe27
Merge: Restarting at timestamp 612088000a0
Merge: Timestamp (61213ac1e1f)[Q 2, Stream 2] is older than previous (61223abffa7) [Q 2, Stream 2]
Merge: Attempting resynchronisation to timestamp 61223abffa8 or greater
Merge: Last sync pulse time: 61223a800a0
Merge: Last timestamp seen: 61223abffa7
Merge: Restarting at timestamp 6122ab400a0

611c72800a0-611c207f4ef = 5200BB1 = 85986225 * 1e-8 = 0.86s

612088000a0-61206effe27 = 1900279 = 0.26s 

6122ab400a0-61223abffa7 = 70800F9 = 1.2s

Resynchronisation appears to require significant periods of time (~s) - why?

I assume this implies significant dead time (data loss) too?

Current nnaida12 (MAC address 2b:09:e8) has several noisy channels in ASICs 1-3, whilst ASIC4 does not work at all.

Frequently, but not always, ASIC4 fails to load/check.

Additionally, fast discriminators are turned off in settings but show activity in the Hit spectrum.

 The case has been changed for one that doesn't allow the power cable to be easily extracted.

The RPi power cable cannot sit fully in the connector, and the lock pins can't engage, because the case gets in the way of the plastic around the end.

As a result we have knocked it out a couple of times and lost power to the system.

Can this be modified or can we get a new case/cable that resolves this issue?

Currently running with a source, and everything seems to be running fine in terms of histograms doing what we expect, no error messages etc.

Tried to start the merger and tape server, but the merger reports as waiting for the first SYNC.

Checked the stats page and sure enough, no SYCs are being produced.

Performed a re-sync from the MasterTS page but nothing changed.

When trying to calibrate FADCs in FADC Align and Control page, ticked 'Act on ALL FEE64 modules'.

Error message displayed (attachment 1) and, despite 'Act on ALL' box being ticked, was only able to act on one FEE64 at a time.