Have you identified a specific noise issue with the Julabo recirculating chiller?

The STFC DL T9 test system has the recirculating chiller and AIDA PSUs connected to the same ac mains extender.

List of the type and quantity of cables we have at RIKEN that are not currently in use.

R1120 stalled shortly after DAQ GO
      'unable to connect' etc

      power cycle & reboot FEE64s

R1121 currently running 
      AIDA SYNCs currently but rates <100Hz in many FEE64s - see attachment 2
      link64 processes consuming large fraction of available CPU - see attachment 1

R1118 ended OK

R1119 DAQ GO, check AIDA SYNCs OK, toggle merge input etc 

Merge then halted with the usual unable to connect to COUNTER, nnaida1, nnaida1 etc error messages
so unable to reset, setup etc

Power-cycling and rebooting

22.14 R1118 starts

      aidas1 has been rebooted


      Changed /MIDAS/linux-ppc_4xx/startup/aidacommon from

     netint DACQ_PushEnable 60
     netint DACQ_FlushEnable 60

     to

     netint DACQ_PushEnable 10
     netint DACQ_FlushEnable 10

      AIDA FEE64s then power-cycled and rebooted  

      nnaida19 data link to merger disabled

      see attachments 1-8

22.30 R1118 stops

      AIDA SYNCs & good events at end of R1118 - see attachments 9-10

11:32 Pulser walkthrough, 90k down to 10k, 10k steps, 1 min per step
      R26
    
11:41 stop run R26


11:44 Pulser walkthrough, 10k to 2k in 2k steps, 90 seconds per step
      R27
11:53 stop run R27

11:56 Pulser walkthrough, 2k to 10k in 2k steps (90 sec each) & 10k to 40k in 10k steps (60 sec per step)
      R28

12:05 Attachment 1 shows biases ("bias_2_a.png")

12:07 stop run R28

12:15 background run, R29

12:39 stop run 29.

Needed to run startx and start the TcLHttpd server on rpi

Following along here: https://elog.ph.ed.ac.uk/AIDA/242

Can now access http://nnrpi1:8015/AIDA/Rly16/

Window at first connection reads:

Using /dev/ttyUSB0 firmware version 0

Using /dev/ttyUSB1 firmware version 0901

Looks good.  Sequence ALL On was selected

(By mistake, I then clicked Relay # to turn off...misread instructions...Switch all OFF, then sequence ALL on again)

7:51 they are turned on.  Waiting several minutes now.

Add 4 L of water to the chiller

8:12

Save/Control Settings from Hardware Control tab:

Selected: 2016Jun13-00.53.55 and Restore. (missing from elog 242)

From Run Control, Enable Histogramming, Data Transfer: Enable #1 All (missing from elog 242)

TapeServer / Merger / Httpd started on Workspace 2

TapeServer reports

Cannot issue more Access Tokens: Claim TapeServer returned:- 3 Capability table is full

Make a new directory for storing test data: /data10/TapeData/July2016

Merger: Setup and configure

Data Statistics shows about ~5 x 10^5 events in many nnaida.  Should we use a different setting file?

Now we try 2016Jun12-10.10.08

Perform ReSync

Probably event rate can be high because the SSD bias is off?  We set up Putty and turn on all biases to -100 V.

Attachments 1 and 2 show the bias settings after they are turned on.

Good Events rate improves, factor 2 to 4 (now about 100k roughly)

Pulser settings: around ~900 on dial, attenuator x10, x10, x2 switches down, 5x up.  Frequency ~250 Hz

21:19

Now we change the Control ASICs (select act on all ASICs, act on all FEEs)

Slow Comparator from 0x8 to 0x20

LEC/MEC Fast Comparator 0x4 to 0x80

Now the Good Events go down to about 10k to 25k roughly

TapeServer starts a run.

Merger started / paused / unpaused

Merger rate is 500k / sec.  TapeServer is 4 k / sec.

AIDA Syncs shown in Attachment 3

Temps in Attachment 4

Good Events Attachment 5

Run Number 1

Setting file: 2016Jul05-21.37.20 (Note: I wrote the wrong setting file name last night...this is the right one!)

Show sample pulser peak in Attachment 6, 7, 8

July 6 - 12:29: Stop the run.  114 run files, 2.0 GB each.

Attachments 9, 10 show biases at the end of the run

 Regarding the P side problem.

What is the result if you change one of the N-Side FEEs to use the P-side pulser and vice-versa ?

Regarding the temperature problem.

Which position are the problem modules in ? 

 The MACB system of clock and SYNC pulse distribution does have to follow some rules. 

The Master needs to be in a very particular place. The rest are less important as long as they are all at the same level in the tree.

In the tree you report above and using the numbering you report. The Master should be in MACB3:out1. 

The settings you have shown are correct if the Master is placed correctly ...

I will input a separate Elog entry detailing the settings for the MACB and the rules for the tree.

Regarding nnaida7+8. I agree the issue to tackle first is the cooling plate contact to the module. Where do they feature in the layout ?

Is the plate tightened up correctly ?

Regarding the P side FEE64s ... sorry about this ... but is the P pulser working ... well i did  have to ask ;-)

20/04/15

1200: Complete all points on first part of check list, except checking the PSU voltages.

          System now in powered and has all network/HDMI cables attached.

1300: Connect RPi to power switch but when powered on it didn't find the switch. A reliable procedure to start seems to be...

• connect USB power switch to RPI, make sure serial cable is disconnected from hub
• power up RPi and power switch
• check connection using dmesg, should see ...FT232RL for USB power switch on USB0
• check connection from aidas1 by pinging RPi inet address 10.1.1.251
• if good, Rly16 page should work from aidas1
• connect serial cable, open PuTTY session, select AIDA console settings, open
• should assign serial cable as USB1 an allow use of both the power switch and FEE console.

       Currently this all works, we can control the power switch both from the RPi and aidas1, and see the FEE console via PuTTY session.

1400: Powered on system and tried to set up DAQ but received error (attachment 1) when doing the initial system reset in Run Control.

          DAQ still usable for spectrum/temperature viewing, and all FEEs can be accessed to check ASIC control, but it seems timestamping will not be correct.

          Tried to perform manual ReSync from master timestamp window and received similar error (attachment 2).

1430: nnaida8 running quite hot (ASIC temp ~65^oC) and ASIC temp on nnaida7+15 read as 0.00^oC. Otherwise, from pulser/stats data, both seem to be working correctly.

          Unplugged power cable for nnaida7+8 at PSU end.

1600: Could timestamping issue originate from MACBs?

          In rebuilding AIDA MACB -> FEE cabling redone. Should it follow a particular pattern/tree?

          Current tree structure looks like (with MACB numbering going 1->11 L->R as you look at it):

                  MACB1(setting: 0) ------- out 1: MACB2 (setting: 2) ---------- out 1: MACB3 (setting: 2) -> nnaida3+4, 11+12

                                                                                                    ---------- out 2: MACB4 (setting: 3) -> nnaida19+20, 27+28

                                                                                                    ---------- out 3: MACB5 (setting: 3) -> nniada1+2, 9+10

                                                                                                    ---------- out 4: MACB6 (setting: 3) -> nnaida17+18, 25+26

                                              --------- out 2: MACB7 (setting: 3) ---------- out 1: MACB8 (setting: 3) -> nnaida21+22, 29+30

                                                                                                    ---------- out 2: MACB9 (setting: 3) -> nnaida7+8, 15+16

                                                                                                    ---------- out 3: MACB10 (setting: 3) -> nnaida5+6, 13+14   -----> nnaida5 is the master and this comes from output2.

                                                                                                    ---------- out 4: MACB11 (setting: 3) -> nnaida23+24, 31+32

          Patrick - Is this set up OK? Do certain FEEs need to receive the MACB signal from certain MACBs or just all at the same point in the tree?

                        Also, MACBs 2+3 are on setting 2 on the dial, whereas the others (except MACB1, the 'grandparent') are set to 3. It seems like a hard thing to change accidentally so are they supposed to be like this or should they all be on the same setting?

1630: Voltages to nnaida7+8 OK at PSU output (+6V, +6V, +6V, -6.5V, +8V). Issue with contact to cooling rack?

          With pulser input to all FEEs, only see pulser data in spectra for n-side (nnaida3+4, 7+8, 11+12, 15+16) FEEs. Other FEEs all show zero statistics.

I have been trying to compile the sort program I have been working on in Edinburgh so I can work on it locally
while at RIKEN.

When compiling i receive the error shown in make.log (attached) saying that there are undefined reference in
DataSource.cpp.
Also attached is the make output when compiled in Edinburgh - basically just 'unused variable' warnings.

This is exactly the same DataSource.cpp I am using in Edinburgh. I have carried out 'diff' on the DataSpyLib and
DataXferLib directories which are included
in the Makefile and no differences are present. I have also done this for DataSource directory and again, there
are no differences between what I am running here and in Edinburgh.

The only difference I have been able to observe so far is that in Edinburgh I am using g++ version:
 Using built-in specs.
 COLLECT_GCC=g++
 COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
 Target: x86_64-redhat-linux
 Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array
--disable-libgcj --with-isl=/builddir/build/BUILD /gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install
--enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
 Thread model: posix
 gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)

whereas on aidas1:
 Using built-in specs.
 Target: x86_64-redhat-linux
 Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile
--enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
 Thread model: posix
 gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)

The libraries used are all found in LD_LIBRARY_PATH.
On aidas1 this is: /MIDAS/Linux/lib64:/homes/npg/root/lib:/homes/npg/root/lib:/MIDAS/Linux/lib64:/homes/npg/root/lib
and in Edinburgh:
/Disk/ds-sopa-group/np/RIKEN/AIDAsort/MIDAS/Linux/lib64:/Disk/ds-sopa-group/np/thehubdata/thehub6/jlab_software/Linux_Scientific6-x86_64-gcc4.4.6/root/5.34.23/lib:


*********UPDATE 03/10/16 *************

The methods are defined in /MIDAS/DataPackage/DataXferLib/V4_TCP/transfer.c

This is in the expected INCLUDES path of: INCLUDES=  -I/MIDAS/DataPackage/DataXferLib/V4_TCP -I/MIDAS/DataPackage/DataSpyLib

Preamplifier waveforms sampled by 50MSPS, 14 bit ADC


attachment 1 - nnaida13 ASIC settings (+ input)
attachment 2 - nnaida16 ASIC settings (- input)

beam on
-------

attachments 3-5 - three successive waveform updates nnaida13 asic #1 - 1000ch = 20us
attachments 6-8 - three successive waveform updates nnaida13 asic #1 - 1000ch = 2ms


attachments 9-11 - three successive waveform updates nnaida16 asic #1 - 1000ch = 20us
attachments 12-14 - three successive waveform updates nnaida16 asic #1 - 1000ch = 2ms

beam off
--------

attachments 15-18 - three successive waveform updates nnaida13 asic #1 - 1000ch = 2ms
attachments 19-21 - three successive waveform updates nnaida16 asic #1 - 1000ch = 2ms


attachments 22-23 - three successive waveform updates nnaida16 asic #1 - 1000ch = 2ms, y-scale expanded
attachments 24-25 - three successive waveform updates nnaida13 asic #1 - 1000ch = 2ms, y-scale expanded

 Measuring the signal at the FEE64 end.

+5v , -6v , +7v, +7v 

The two other 'scope channels are connected by FET probes to the FEE64. I now see there is a ground connection.... so i will remove it and re-measure the signals in the filter.

 No real difference.

Measuring at the FEE64 and the spikes when the Mux ADC is reading out have disappeared. 

The new pcb for the power supply filter arrived.

I assembled the components ( after finding the whoops ) and connected it to the bench power supply for nnaida2.

nnaida2 functions just about the same as before but the power supply filter results seem amazing.

I have measured the signals on the power supply before and after the filter.

The 'scope is referenced to earth on the filter board and this earth is connected to earth through the +5v@10A supply.

Before the filter the average noise on the rail ( AC coupled ) is 33mV and after is ....... 5mV pretty exciting stuff.

 I have attached the two 'scope pictures.

The same measurements on the +7v, -6v and 0v rails give similar improvements.

The choke on the +5v rail measures 34 degrees C so care will need to be taken.

Next I will measure the voltage drops across the filter and decide on the correct voltage settings for the AIDA switch mode power supply.

Then install the filter into the supply and check ..... and check ......

2.0.L FWHM:14.26

2.15.L ( 270pF )  FWHM : 119.04

(ooops found the 'scope channel was on 20Mhz BW filter ) 

DC measurements.

At FEE64 :- +7.822 , -6.36, +4.937

at Filter output ( AIDA Cable input )  :- +7.96, -6.46, 5.37

at filter input:- +7.97, -6.476, +5.488

Set supply with these voltages and add filter.

Measured earth point to mains earth pin as 0.095ohms.

Power-up with nnaida2. ADCs Calibrate fine....successful boot.

Voltages at FEE64:- +7.802, -6.324, +4.887. I will adjust the power supply voltages to match the FEE64 previous setup.

2.0.L FWHM :13.83

2.15.L FWHM:116.6

So not really a difference in performance at this simple level.

Attached are some oscilloscope traces of the six voltage rails after the filter on the filter board. +5v & 0v , -6v & 0v, +7v & 0v relative to the earth on the filter board. 

Attached are some oscilloscope traces of the six voltage rails before the filter on the filter board. +5v & 0v , -6v & 0v, +7v & 0v relative to the earth on the filter board.

There is a nice sine wave at 235KHz that needs to be finished off perhaps. Otherwise the filter seems to be doing well.

Next set of 'scope traces are across the three supplies. +7v, -6v, +5v.

Sine wave is present ( 220KHz )  at 80mv, 80mv, 105mv.

Next I'll try this supply on nnaida12 ( detector negative ) ...... next entry .... 

 Reading about the power supply changes and the requirement to replace fuses leads me to be concerned.

I think the power being changed is the AC input to the FEE64 power supplies. 

Which Fuses are failing ? 

Stack of Pb bricks upstream of AIDA+BRIKEN to protect from incoming beam.

Carried out some further work on failed mezzanine modules.

Inspected them externally and removed the copper block, internally with a bench microscope.

Found two obvious faults ..... broken components with scrambled pcb tracks.

To help determine problems with loading control registers I've added a new menu item in the ASIC control window to allow all four ASIC return values to be checked. This is only useful at the mezzanine test system.

Tested 7 that were labelled as faulty. Failed all of them. 2 with broken components and 1 with some broken bond wires. 

Tested one fresh from the cupboard. It failed with over current on the +5v psu ( > 4A! ).

I propose to create a method of testing these mezzanines which allows better access to the power supply regulators ( LT3080 ) and the associated control voltage.

PID plot produced from 2 hours of merged data - attachment 1

Z on y-axis, A/Q on x-axis

numbers estimated by gating on each isotope and integrating

94Ag - 10584 events in 2hrs

97In - 42 events in 2 hrs

100Sn - 56 events in 2 hrs

94Pd - 383030 events in 2 hrs