Attached is the specification for the MACB with the settings currently available.

11.45 AIDA discriminator rates - see attachment 1
      Today's rates similar to yesterday's rates 

14.00 Commence positioning of AIDA. Adjust stand height using laser scanner and aligned to wall
      references. Determine that the most forward position AIDA DSSSD#1 is ~3cm downstream of
      EURICA centre. Need to move DSSSDs ~6cm upstream and lengthen DSSSD - AIDA adpptor PCB 
      cabling. In progress.

20.50 Update /etc/dhcp/dhcpd.conf 

      nnaida19 is faulty, nnaida28 is not used - see https://elog.ph.ed.ac.uk/AIDA/193
      We have swapped FEE modules for nnaida19/20 for nnaida27/28 such that nnaida19 -> 28, nnaida20 -> 27,
      nnaida27 -> 19, nnaida28 -> 20. 
      The power cables for nnaida19 and nnaida28 have also been swapped - the power cable to nnaida19
      *might* be faulty as we have experienced problems at this crate position before. Also swapped
      AIDA FEE PSU outlets to ensure pairs of FEE64 cards are powered from the same PSU outlet.
 
[root@aidas1 dhcp]# diff dhcpd.conf dhcpd.conf.BAK-260516
205,206c205,206
<    host nnaida19 {
<      hardware ethernet 00:04:a3:2a:ed:9a;
---
>    host nnaida19 {
>      hardware ethernet 00:04:a3:2b:09:e7;
210,211c210,211
<    host nnaida20 {
<      hardware ethernet 00:04:a3:2b:11:c5;
---
>    host nnaida20 {
>      hardware ethernet 00:04:a3:2b:22:55;
225,226c225,227
<    host nnaida27 {
<      hardware ethernet 00:04:a3:2b:22:55;
---
> # 2 of 2 scratch FEE modules assembled by PJCS April 2016
>    host nnaida27 {
>      hardware ethernet 00:04:a3:2a:ed:9a;
230,232c231,232
< # 2 of 2 scratch FEE modules assembled by PJCS April 2016
<    host nnaida28 {
<      hardware ethernet 00:04:a3:2b:09:e7;
---
>    host nnaida28 {
>      hardware ethernet 00:04:a3:2b:11:c5;
[root@aidas1 dhcp]# cat dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#

authoritative;

ddns-update-style none; ddns-updates off;

max-lease-time 691200;

option domain-search code 119 = string;

subnet 10.1.1.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 10.1.1.255;
   option routers 10.1.1.250;
   pool {
      range 10.1.1.100 10.1.1.249;
   }
}

subnet 10.1.2.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 10.1.2.255;
   option routers 10.1.2.250;
   pool {
      range 10.1.2.100 10.1.2.249;
   }
}

subnet 10.1.3.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 10.1.3.255;
   option routers 10.1.3.250;
   pool {
      range 10.1.3.100 10.1.3.249;
   }
}

subnet 10.1.4.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 10.1.4.255;
   option routers 10.1.4.250;
   pool {
      range 10.1.4.100 10.1.4.249;
   }
}

subnet 10.1.5.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 10.1.5.255;
   option routers 10.1.5.250;
   pool {
      range 10.1.5.100 10.1.5.249;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "10.1.1.250";
   next-server 10.1.1.250;

   host nnrpi1 {
     hardware ethernet b8:27:eb:78:ee:44;
     fixed-address 10.1.1.251;
   }
   host nnaida1 {
     hardware ethernet 00:04:a3:2a:d0:1b;
     fixed-address 10.1.1.1;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida1,tcp";
   }
   host nnaida2 {
     hardware ethernet 00:04:a3:2a:d9:e1;
     fixed-address 10.1.1.2;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida2,tcp";
   } 
   host nnaida5 {
     hardware ethernet 00:04:a3:2a:d0:10;
     fixed-address 10.1.1.5;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida5,tcp";
   }
   host nnaida6 {
     hardware ethernet 00:04:a3:2a:d4:b7;
     fixed-address 10.1.1.6;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida6,tcp";
   }
   host nnaida9 {
     hardware ethernet 00:04:a3:2a:d9:ad;
     fixed-address 10.1.1.9;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida9,tcp";
   }
   host nnaida10 {
     hardware ethernet 00:04:a3:2a:f7:07;
     fixed-address 10.1.1.10;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida10,tcp";
   }
   host nnaida13 {
     hardware ethernet 00:04:a3:2b:11:b9;
     fixed-address 10.1.1.13;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida13,tcp";
   }
   host nnaida14 {
     hardware ethernet 00:04:a3:2b:11:b8;
     fixed-address 10.1.1.14;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida14,tcp";
   }
   host nnaida17 {
     hardware ethernet 00:04:a3:2a:d9:cc;
     fixed-address 10.1.1.17;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida17,tcp";
   }
   host nnaida18 {
     hardware ethernet 00:04:a3:2a:b2:b2;
     fixed-address 10.1.1.18;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida18,tcp";
   }
   host nnaida21 {
     hardware ethernet 00:04:a3:2b:27:23;
     fixed-address 10.1.1.21;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida21,tcp";
   }
   host nnaida22 {
     hardware ethernet 00:04:a3:2a:f6:e9;
     fixed-address 10.1.1.22;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida22,tcp";
   }
   host nnaida25 {
     hardware ethernet 00:04:a3:2a:d0:11;
     fixed-address 10.1.1.25;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida25,tcp";
   }
   host nnaida26 {
     hardware ethernet 00:04:a3:2b:27:54;
     fixed-address 10.1.1.26;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida26,tcp";
   }
   host nnaida29 {
     hardware ethernet 00:04:a3:2a:d0:06;
     fixed-address 10.1.1.29;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida29,tcp";
   }
   host nnaida30 {
     hardware ethernet 00:04:a3:2a:d9:b6;
     fixed-address 10.1.1.30;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida30,tcp";
   }


}


group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "10.1.2.250";
   next-server 10.1.2.250;

   host nnaida3 {
     hardware ethernet 00:04:a3:2a:f7:06;
     fixed-address 10.1.2.3;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida3,tcp";
   }
   host nnaida4 {
     hardware ethernet 00:04:a3:2a:ed:b8;
     fixed-address 10.1.2.4;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida4,tcp";
   }
   host nnaida7 {
     hardware ethernet 00:04:a3:2a:b8:8d;
     fixed-address 10.1.2.7;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida7,tcp";
   }
   host nnaida8 {
     hardware ethernet 00:04:a3:2a:9c:07;
     fixed-address 10.1.2.8;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida8,tcp";
   }
   host nnaida11 {
     hardware ethernet 00:04:a3:2b:11:af;
     fixed-address 10.1.2.11;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida11,tcp";
   }
   host nnaida12 {
     hardware ethernet 00:04:a3:2a:d0:2f;
     fixed-address 10.1.2.12;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida12,tcp";
   }
   host nnaida15 {
     hardware ethernet 00:04:a3:2a:f3:ac;
     fixed-address 10.1.2.15;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida15,tcp";
   }
   host nnaida16 {[root@aidas1 aidas1]# diff startup.tcl startup.tcl.BAK-260516
31,32c31,32
<     variable PACQSERVERS; set PACQSERVERS { nnaida1 nnaida2 nnaida5 nnaida6  nnaida9 nnaida10 nnaida13
nnaida14 nnaida17 nnaida18 nnaida21 nnaida22 nnaida25 nnaida26 nnaida29 nnaida30 }
<     variable NACQSERVERS; set NACQSERVERS { nnaida3 nnaida4 nnaida7 nnaida8 nnaida11 nnaida12 nnaida15
nnaida16 nnaida19 nnaida20 nnaida23 nnaida24 nnaida27 nnaida28 nnaida31 nnaida32 }
---
>     variable PACQSERVERS; set PACQSERVERS { nnaida1 nnaida2 nnaida5 nnaida6  nnaida9 nnaida10 nnaida13
nnaida14 nnaida17 nnaida18 nnaida21 nnaida22 nnaida25 nnaida29 nnaida30 }
>     variable NACQSERVERS; set NACQSERVERS { nnaida3 nnaida4 nnaida7 nnaida8 nnaida11 nnaida12 nnaida15
nnaida16 nnaida20 nnaida23 nnaida24 nnaida27 nnaida28 nnaida31 nnaida32 }
33a34
> # startup.tcl.BAK-130516 has all 32 NNAIDA modules included
52c53
<     lappend ACQSERVERS nnaida19
---
> #    lappend ACQSERVERS nnaida19
59c60
<     lappend ACQSERVERS nnaida26
---
> #    lappend ACQSERVERS nnaida26

     hardware ethernet 00:04:a3:2a:9c:0e;
     fixed-address 10.1.2.16;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida16,tcp";
   }
# 1 of 2 scratch FEE modules assembled by PJCS April 2016
   host nnaida19 {
     hardware ethernet 00:04:a3:2a:ed:9a;
     fixed-address 10.1.2.19;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida19,tcp";
   }
   host nnaida20 {
     hardware ethernet 00:04:a3:2b:11:c5;
     fixed-address 10.1.2.20;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida20,tcp";
   }
   host nnaida23 {
     hardware ethernet 00:04:a3:2a:b8:90;
     fixed-address 10.1.2.23;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida23,tcp";
   }
   host nnaida24 {
     hardware ethernet 00:04:a3:2a:b6:45;
     fixed-address 10.1.2.24;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida24,tcp";
   }
   host nnaida27 {
     hardware ethernet 00:04:a3:2b:22:55;
     fixed-address 10.1.2.27;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida27,tcp";
   }
# 2 of 2 scratch FEE modules assembled by PJCS April 2016
   host nnaida28 {
     hardware ethernet 00:04:a3:2b:09:e7;
     fixed-address 10.1.2.28;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida28,tcp";
   }
   host nnaida31 {
     hardware ethernet 00:04:a3:2b:11:ae;
     fixed-address 10.1.2.31;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida31,tcp";
   }
   host nnaida32 {
     hardware ethernet 00:04:a3:2a:ed:b9;
     fixed-address 10.1.2.32;
     option root-path "/MIDAS@aidas/XilinxLinux/ppc_4xx/rfs/nnaida32,tcp";
   }
}


group {
# 8 days
   default-lease-time 691200;

#    option dhcp-client-identifier "\0foo";

   host nndhcp052 {
# Analysis server
     hardware ethernet 00:22:19:50:02:40;
     fixed-address 10.1.1.252;
   }

}


DAQ server configuration file /MIDAS/config/TclHttpd/aidas1/startup.tcl update - see attachment 2

[root@aidas1 aidas1]# diff startup.tcl startup.tcl.BAK-260516
31,32c31,32
<     variable PACQSERVERS; set PACQSERVERS { nnaida1 nnaida2 nnaida5 nnaida6  nnaida9 nnaida10 nnaida13
nnaida14 nnaida17 nnaida18 nnaida21 nnaida22 nnaida25 nnaida26 nnaida29 nnaida30 }
<     variable NACQSERVERS; set NACQSERVERS { nnaida3 nnaida4 nnaida7 nnaida8 nnaida11 nnaida12 nnaida15
nnaida16 nnaida19 nnaida20 nnaida23 nnaida24 nnaida27 nnaida28 nnaida31 nnaida32 }
---
>     variable PACQSERVERS; set PACQSERVERS { nnaida1 nnaida2 nnaida5 nnaida6  nnaida9 nnaida10 nnaida13
nnaida14 nnaida17 nnaida18 nnaida21 nnaida22 nnaida25 nnaida29 nnaida30 }
>     variable NACQSERVERS; set NACQSERVERS { nnaida3 nnaida4 nnaida7 nnaida8 nnaida11 nnaida12 nnaida15
nnaida16 nnaida20 nnaida23 nnaida24 nnaida27 nnaida28 nnaida31 nnaida32 }
33a34
> # startup.tcl.BAK-130516 has all 32 NNAIDA modules included
52c53
<     lappend ACQSERVERS nnaida19
---
> #    lappend ACQSERVERS nnaida19
59c60
<     lappend ACQSERVERS nnaida26
---
> #    lappend ACQSERVERS nnaida26

Need to update nnaida19 and nnaida26 to latest firmware version - see https://elog.ph.ed.ac.uk/AIDA/237

Done. Awaiting test. See attachments 1-2.

> The attached firmware file, FEE_GF_Feb16_18.bin, should be saved to /MIDAS/Aida and the FlashPgm.csh file edited
> to load "FEE_GF_Feb16_18.bin"
> 
> Use the Run Control Expert command to update the firmware on all the FEE64s.
> 
> Power cycle the FEE64s to load the new firmware.
> 
> Make a backup copy of the files in the directory /MIDAS/TclHttpd/Html/AIDA/DISC.
> 
> Unzip the files from DISC_25May16.zip and copy them into /MIDAS/TclHttpd/Html/AIDA/DISC to replace the existing
> ones.
> 
> Make a backup copy of the file /MIDAS/TclHttpd/Html/AIDA/LOCAL.tml
> 
> Unzip the file from LOCAL_tml_25May16.zip and copy into /MIDAS/TclHttpd/Html/AIDA/LOCAL to replace the existing
> file.
> 
> 'RESET' and browser refresh the two updated control windows to load the new software.
> -----------------------------------------------------
> 
> To use the Multiplicity Trigger:-
> In the Local Controls select option 5 for the Trigger.
> 
> In the Discriminator window.
> 
> Set the 'Upper Limit'
> Set the 'Lower Limit' ( always greater than 0 ..... or the Trigger will always be present )
> Set the 'Time window'. The number of 10nS clocks. So 4 => 40nS. Register is 0 to 255.
> 
> -----------------------------------------------------
> 
> These registers are not in the Save/Restore set yet.
> 
> Tested on the T9 system using the pulser and setting/clearing Mask bits.
> 
> Please let me know how this works for you.

The attached firmware file, FEE_GF_Feb16_18.bin, should be saved to /MIDAS/Aida and the FlashPgm.csh file edited
to load "FEE_GF_Feb16_18.bin"

Use the Run Control Expert command to update the firmware on all the FEE64s.

Power cycle the FEE64s to load the new firmware.

Make a backup copy of the files in the directory /MIDAS/TclHttpd/Html/AIDA/DISC.

Unzip the files from DISC_25May16.zip and copy them into /MIDAS/TclHttpd/Html/AIDA/DISC to replace the existing
ones.

Make a backup copy of the file /MIDAS/TclHttpd/Html/AIDA/LOCAL.tml

Unzip the file from LOCAL_tml_25May16.zip and copy into /MIDAS/TclHttpd/Html/AIDA/LOCAL to replace the existing
file.

'RESET' and browser refresh the two updated control windows to load the new software.
-----------------------------------------------------

To use the Multiplicity Trigger:-
In the Local Controls select option 5 for the Trigger.

In the Discriminator window.

Set the 'Upper Limit'
Set the 'Lower Limit' ( always greater than 0 ..... or the Trigger will always be present )
Set the 'Time window'. The number of 10nS clocks. So 4 => 40nS. Register is 0 to 255.

-----------------------------------------------------

These registers are not in the Save/Restore set yet.

Tested on the T9 system using the pulser and setting/clearing Mask bits.

Please let me know how this works for you.

10.05 Recirculating chiller on
      FEE power on
      Detector bias on - ambient temp +23 deg  C - see attachments 1-2

20.30 Noisy fast discriminator channels masked in all nnaida to bring 'AIDA discriminator' rate to <~5kHz.
      Slow and fast comparator thresholds at 16 (decimal).
      Settings saved to 2016May14-13.57.38 DB key.

The USB controlled power Relays are near their AC fuse operating limit which has caused some Fuse failures.
To reduce the power consumption of each FEE64 it is possible to power down the Waveforms ADCs saving about 20W per FEE64. 

To Powerdown the ADCs in a FEE64 use the 'Local Data' browser window from the 'Control' browser window. Change the setting of the ADC Control 
Register at offset 2 from 0 to 0xFF. All the FEE64s can be thus controlled by using the 'Act on all' tick box.
Clearly, the System Check for ADC calibration will now fail for all those that are powered down.

To Power up the ADCs change the setting from 0xFF to 0. Then using the FADC Calibration browser window calibrate the FADCs. 

The SYSTEM should be in the 'STOP' state during these procedures.

Should the SYSTEM be power cycled then the ADCs will be powered up.

Saving at the 16A AC fuse is estimated to be 3A as there are 16 FEE64 being powered through each.

The 3x WDC WD30EFRX ATA 3Tb disks installed in aidas1 in April 2015 (formerly /data20, /data30 and
/data40) have now been added to the aidas1 logical volume management (LVM) configuration to form a
single c. 8Tb filesystem (/data10) which will be automatically mounted at boot.

N.B. Although the disks are 3Tb capacity, Linux OS/BIOS can only access the first 2.2Tb of each disk.

Data previously stored on /data20, /data30 and /data40 has been copied to the local Seagate USB 8Tb
disk (/ntfs) and the University of Edinburgh ECDF multi-Pb disk array.

aidas1> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidas1-lv_root
                       50G   27G   20G  58% /
tmpfs                 3.9G  1.1M  3.9G   1% /dev/shm
/dev/sdb1             477M   77M  376M  17% /boot
/dev/mapper/vg_aidas1-lv_home
                      860G  106M  816G   1% /home
/dev/mapper/Data1-data10
                      7.7T  1.6T  5.8T  22% /data10
/dev/sda2             7.3T  3.4T  4.0T  46% /ntfs

[root@aidas1 /]# ln -s /data10/TapeData /TapeData
[root@aidas1 /]# ls -l /
total 210
dr-xr-xr-x.   2 root root      4096 Apr  7 03:49 bin
dr-xr-xr-x.   5 root root      1024 Feb 17  2015 boot
drwxr-xr-x.  10 root root      4096 Jul 23  2015 cgroup
drwxr-xr-x.   8 root root      4096 Dec 12 19:29 data10
drwxr-xr-x.   2 root root      4096 Apr 28  2015 data20
drwxr-xr-x.   2 root root      4096 Apr 28  2015 data30
drwxr-xr-x.   2 root root      4096 Apr 28  2015 data40
drwxr-xr-x.  21 root root      4500 May 24 14:40 dev
drwxr-xr-x. 135 root root     12288 May 24 17:55 etc
drwxr-xr-x.   4 root root      4096 Apr 28  2015 home
drwxr-xr-x.   4 root root      4096 Jan  8  2014 homes
dr-xr-xr-x.  13 root root      4096 Feb 18 03:19 lib
dr-xr-xr-x.  11 root root     12288 Apr 14 06:02 lib64
drwx------.   2 root root     16384 Jan  7  2014 lost+found
drwxr-xr-x.   2 root root      4096 May 24 11:00 media
lrwxrwxrwx.   1 root root        12 Apr 22  2014 MIDAS -> /MIDAS@aidas
drwxrwxr-x.  29 npg  npgstaff  4096 May  3 23:47 MIDAS@aidas
drwxr-xr-x.  22 root root      4096 Dec 12 12:10 MIDAS@aidas.121215
drwxr-xr-x.  28 root root      4096 Feb 24  2015 MIDAS@aidas.240215
drwxr-xr-x.   2 root root         0 May 24 11:00 misc
drwxr-xr-x.   2 root root      4096 Jul 21  2011 mnt
drwxr-xr-x.   2 root root         0 May 24 11:00 net
drwxrwxrwx.   1 root root      4096 Apr 28 10:55 ntfs
drwxr-xr-x.   4 root root      4096 Feb 27  2014 opt
dr-xr-xr-x. 411 root root         0 May 24 10:58 proc
dr-xr-x---.  28 root root      4096 May 24 18:01 root
drwxr-xr-x.   2 root root     12288 Feb 18 03:19 sbin
dr-xr-xr-x.   2 root root      4096 Feb 27  2014 scratch
drwxr-xr-x.   7 root root         0 May 24 10:58 selinux
drwxr-xr-x.   2 root root      4096 Jul 21  2011 srv
drwxr-xr-x.  13 root root         0 May 24 10:58 sys
lrwxrwxrwx.   1 root root        16 May 24 18:02 TapeData -> /data10/TapeData
drwxrwxrwt.  41 root root     65536 May 24 18:01 tmp
drwxr-xr-x.  13 root root      4096 Jan  7  2014 usr
drwxr-xr-x.  23 root root      4096 Jan  7  2014 var
dr-xr-xr-x.   2 root root      4096 Feb 27  2014 WWW

Attachments 1-30 per FEE64 ASIC #1 waveforms

Note - nnaida19 and nnaida26 disabled
       nnaida11, nnaida17 and nnaida29 ADC calibration failed => no waveforms

Overnight. DAQ stopped. Detector bias off. FEE power and recirculating chiller on.

08.00 Detector bias on OK - see attachments 1-2
      DAQ start OK - see attachments 3-4
      All system-wide tests OK *except* nnaida9 and nnaida29 failed ADC calibration
      Temperatures OK 

18.00 Saved corrected ASIC settings to DB Key 2016May14-13.57.38
      Changed options - nnaida5 master timestamp sync phase from 0x1 to 0x3 

When starting Aida some FEE64 fail to provide SYNC data , or any data for that matter, on the ASIC data stream.
The problem is that the ASIC Readout is not starting properly and is becoming stuck waiting for an internal
event which will never arrive.
The chief source has been found to be the Correlation part of the VHDL. The reason is not yet clear but the work
around of disabling the Correlation part of all of the FEE64 at 'GO' except the Master has allowed the System to
start with all FEE64 producing SYNCs. 

This change is now incorporated in the sys.tcl file which is used when 'GO' is activated in the RunControl window.

The Master is required as it provides the 25Mhz clock and the Reset pulse. The Correlation Trigger will cause a
correlation data item to be put in the Master ASIC data stream.
Only one Correlation data item is required in the data stream, so having just the Master should be fine.

If it is required to enable another FEE64 then do this after 'GO' by opening the Correlation Browser window from
the Control window and set the Control ( at offset 0 ) to 0x0001.

Should one of the FEE64s still fail to provide SYNCs first try to 'STOP' , 'GO' the system two times and see
if that works.

A further means of correcting the problem is to directly reset the ASIC Readout using the Data Readout Window
available through the Control window. In the ASIC Readout - buffer status & control (0x300) section enter 0x4000
into the Control register to reset the state machines. Then enter 0x0001 to re-enable them. This may solve the
problem.

This is all by no means an ideal situation and the source will be found and the problem solved.

Please read and comment on the attached specification.

12.45 Rly16 'Sequence ALL On' operational

15.20 Startup remote control of CAEN N1419 4-ch HV modules
      DSSSD #1-7 leakage currents OK - see attachments 1-2
      Ambient temperature +24.8 deg C

15.35 Plot Temperature appears to plot the first 15,000 (or so) temperature readings
      and not the *most recent* 15,000 (or so) readings. See attachment 3.

      There is no way to change the time window for the temperature plot

      As a workaround, empty file using the command

      cp /dev/null /MIDAS/log/temperatures.txt

17.10 For all FEEs (*except nnaida19 & nnaida26) which are currently disconnected)
      reprogram FEE flash memory using FEE_GF_Feb16_17.bin
      Power cycle

      attempt #1 - wait 4'

      STATE for nnaida9 returned with an error
      connect failed
      NONE
      connect failed

      attempt #2 - wait 6'

      STATE for nnaida4 returned with an error
      connect failed
      NONE
      connect failed

      attempt #3 - wait 8' - OK

      Installed updated Check software - Check.tml .tcl .js and Common.js

      Started DAQ - histogramming enabled - data transfer disabled 

      Large fraction of FEEs not producing SYNCs, SYNC rate low (~20-30Hz) for other FEEs - see attachments 4 & 5
      Note slow comparator 0xc8 (i.e. high) and fast discriminators (LEC/MEC) disabled

      Firmware version - attachment 6

      System wide checks ASIC clock sync (30/30 fail) and ADC calibration (7/30 fail) work as before but
      new checks produce errors - see attachments 7-11

      Additional screenshots from Master Timestamp Control - see attachments 12-17

00:40 PJCS identifies workaround to above problems - correlation controls now enabled for master timestamp
      FEE (nnaida5) only. May also explain issues with nnaida19 & nnaida26.
      PJCS will produce detailed, separate Elog entries

      Currently, all (30) FEE64s running OK with sensible AIDA SYNC rates. 

      But. No spare fuses (16A/250V slow blow) for  USB-controlled ac mains relay. Until Chris arrives in
      c. 9h. We will explore options to use *second* USB-controlled ac mains relay.

The program for online monitoring the correlation of the DAQ systems (AIDA-EURICA-BigRIPS) through a common scaler is SyncCheck (J. 
Agramunt). 

The code runs in the d05 server, receiving data transferred to a DataSink from each of the individual DAQs. The data transfer is 
done with the DataRelayFilter code for AIDA (AE), and the euricatssender and ribftssender (Baba-san) codes for EURICA and RIBFdaq, 
respectively.  

The attached screenshots show how to setup these programs. Follow these steps:

1) Log in to d05 as user 'aida'. Start an instance of the DataSink code with option -i 3 (3 data stream, which will use ports 10305 
to 10307). The code is saved in /home/aida/DataPackage/DataSink/Linux64

2) In the d05 server start the following codes: euricatssender (in /home/aida/ribfts/euricatssender/ directory), and ribftssender 
(in /home/aida/ribfts/ribftssender/ directory). If you give the codes dummy command line option (e.g. -X) the value of transferred 
timestamps will be printed to screen as diagnostics.

3) In the AIDA computer (aidas1), start the DataRelayFilter program (will work from any directory). As command line arguments 
provide the IP address of d05, and the TCP port and id of the data stream for the DataSink program: DataRelayFilter -n 10.32.0.12 -
p 10307 -I 2

Only the master FEE will produce correlation scalar items (?). The code has as default NNAIDA5 for the module where to search for the Corr Scaler
items; if this has to be changed use command line option -m X (where X is NNAIDAX number for new master FEE)

4) Back in the d05 server start the SyncCheck program, in the directory /home/aida/SyncCheck/. You can give as an argument the ID 
of the Master data stream for the synchronization check, through "-M ID" (should be done if one of the three data streams is down). 
For example "SynCheck -M 2" will use AIDA as the master timestamp.

5) Perform a manual correlation reset request to synchronize the value of the correlation scaler in all three DAQs. This is done with the 
button 'Force Correlation Reset' of the AIDA "Correlation Status & Control" webpage (attachment 2), which can be accessed from the MIDAS web-
based control of AIDA.

The SyncCheck code will display a few spectra for onine monitoring of DAQ correlation (attachment 3, shown only with AIDA and EURICA running). 
A peak will appear showing the offset in the correlation scaler between the different DAQs. The counts in the peak should increase 
continuously, and its position should not change. If it doesn't, try sending a Correlation Reset signal (step 5). If this does not fix the 
monitor, report a correlation error! 

Further information on the scheme for data synchronization is here: 
https://elog.ph.ed.ac.uk/BRIKEN/6
https://elog.ph.ed.ac.uk/BRIKEN/5

 Reading about the power supply changes and the requirement to replace fuses leads me to be concerned.

I think the power being changed is the AC input to the FEE64 power supplies. 

Which Fuses are failing ? 

When using the T9 system and running with too much rate in one FEE64 so the system is unbalanced.

I found a good way to understand this and to be able to operate the controls of the FEE64s was to place the Merger in "Pause" mode.

Then the data is extracted from the busy FEE64 but it can still be controlled.

When there is unbalance in the system it is possible for one or more queue in the Merger to become full and so the communication with that FEE64 is not possible.

Placing the Merger in Pause mode will mean all the data is discarded, the links run at full speed and action can be taken on the problem FEE64(s) to help balance the system out better.

I hope this helps .... it is highly likely that this is already being done.....

Perhaps a wise sequence when starting the whole DAQ is to operate with the Merger in Pause mode and  only switch it to merging when the user is happy with the rates.

In the statistics window of the FEE64 there is an indication of the data being transferred over the link.

When deciding how to run the experiment thought needs to be given to the various maximum rates. The rate from the FEE64 is 500k events/sec. If running with 26 FEE64 then this goes up to 13M events/sec. An event is 64 bits.

I'm not sure what the maximum rates are in the Merger and the TapeServer.

11 am: Changed configuration of power source for AIDA PSU (see paper logbook). Now both cables with power for
USB relay are connected to the 'C' unit of the F11 power supply. They use different socket pads, and share them
with power cables for raspberry pi and NIM crates. Situation intermediate to what we have on Tuesday.

Resolution of pulser peaks improved slightly w.r.t. yesterday, but still we have the high frequency noise
(500kHz) present (attachments).

12 pm: We found the FEEs crash (loose connection/synchronization) when we enable the fast discriminators (see
attach. 1). This could be in the case when some of the discriminator channels are triggering at a fast rate
(noise above threshold). We observed the same pattern a few times.

5 pm: During all afternoon we've not been able to put AIDA in an operational state. There is always one or more
FEEs that do not synchronize or loose connection. Also the fuse of the PSU broke and we had to replace them.
Source for state of the system could be related to current configuration of PSU power, or the presence of noise
we observed in last couple of days.


6 pm: Back to configuration of PSU from yesterday:

F11 power supply 'C' ---->----> PSU relays 1-4
                          |
                          |---> NIM crates, RPI, etc


F11 power supply 'B' ----> PSU relays 5-8



Started by checking the options. Loaded Options 2016May03-20.44.33. It looks like settings of ASICs are not the
standard ones (e.g. threshold 47). Tried restore settings in Save/Restore option. It worked for all but NNAIDA25
(fri22.png), but other have proper settings (but values of #25 also look ok!).

Performed 'Check ASIC clock timestime' in systemwide checks: all failed (fri23.png)
Performed 'Check ADC calibration' in systemwide checks: all passed, except for nnaida26 (fri24.png)
Master Timestamp Control -> Perform ReSYNC: worked for all modules except nnaida26


we lost connection to 26..... (fri26.png)

Attempted ADC calibration of NNAIDA26 (FADC align and control) failed (fri27.png)

Disabled NNAIDA from Run Control, and Reset all modules. But now after reset the setup button does not appear,
just Reset one as if the action had no effect.

Tryed restore settings again. Again nnaida25 returned problem, but yet again nnaida26 is the one that actually
doesn't have proper ASIC settings. Confirmed this with 'Check ASIC control' (fri29.png). Is there an error in the
error report message of 'Restore Settings'?

We checked cable connections (HDMI, ethernet), and afterwards NNAIDA26 keeps giving problems. We give up on it,
remove it from DAQ configuration (https://elog.ph.ed.ac.uk/AIDA/189), and system nos seems to be runing ok
without it.


8 pm: disconnected all remaining cables from MACB->other crates (only daq correlation in macb1), and went back to
yesterdays configuration of PSU power with all power from 'C'

F11 power supply 'C' ---->----> PSU relays 1-4
                     |    |
                     |    |---> PSU relays 5-8
                     |
                     |--->---> NIM crates, RPI, etc

Continued work on beta-trigger for EURICA. However the system seems to have different noise condition than before, which 
we think appeared after we changed power source for FEE's PSU (https://elog.ph.ed.ac.uk/AIDA/220). This is high frequency 
noise of low amplitude, with a period of 10 usec.

The effect on the resolution measured with the pulser is not too large - thus our conclusion in elog entry 220 that noise 
conditions had not changed much - but the noise does create a large trigger rate that would not allow us to set a low 
trigger threshold (for MAC-B outputs from fast discriminators). The rate of singles from front or back, as well as coincidence of 
front-back strips, increased by more than a factor of ten compared to previous conditions (e.g. ~200 coincidences for one ASIC 
enables in each side of DSSD1, and fast threshold of 10).

Saved some data in this conditions, with a threshold of 16 for the fast discriminators (enabled in one ASIC for 8 of the 
FEEs). When fast discriminator lowering threshold to 9, the system got upset and lost synchronization. Tomorrow we'll 
attempt to revert to previous configuration of PSUs.

R16*gz:

DSSD HV: +100 for all.
Leakage current (uA): 4.2, 4.9, 3.7, 3.9, 4.2, 3.8, 10.7 (from Det#1 to veto)

Enabled fast discriminators in ASIC2 of DSSD1 and DSSD5 (NNAIDA#: 25, 30, 27, 32, 9, 13, 11, 16)

DSSD#1 is next to 207Bi source

Please note that the Force Trigger and Reset buttons  in the Correlation Browser only have meaning if the FEE64 
selected is the Master.

*2:30 pm*: changed power source of USB-controlled power supply for FEEs. Now one of the power cables is connected
to the 'B' power supply of F11, and the second to the 'C' power supply (these are in the unit on top of WASBI
crates). The reason for change is that before both were connected to 'C', and drained current was very close to
limit  of 10 A (and power cord getting noticeable hot). We'll check resolution after this switch.

Detectors 1, 2, 3, and 4 biased to 100V, detectors 5, 6, and veto biased to 70V.

n-side is more noisy than p-side, but resolution comparable to previous configuration:

NNAIDA#   FWHM [ch]
    1     91
    2     61
    3     84
    4     -
    5    160
    6     -
    7     79
    8     84
    9     71
   10     76
   11    101
   12     81
   13     69
   14     78
   15    199
   16    107
   17     82
   18     62
   19     -
   20     89
   21    168
   22     70
   23     80
   24     73
   25     81
   26     13
   27    109
   28     13
   29     15
   30    126
   31     13
   32    100



*5 pm*: Set-up correlation scalers for AIDA<->EURICA<->BigRIPS (detail diagram coming soon, now only in paper
logbook). 


*6:30 pm*: Started EURICA DAQ with Pieter and Gabor.

A folder is created to record data of June campaign: /raid/11/mbs/2016_ucampaign/

Tested correlation of AIDA and EURICA with a pulser run, triggering EURICA and AIDA correlation scaler with 250
Hz. The corresponding files are

/raid/11/mbs/2016_ucampaign/test0003.lmd
/TapeData/may2016/R12_0.gz

Phong did a quick analysis and the pattern of correlation scalers in both files is similar (attached figures). It
is also consistent with the number of reset requests we sent during the data taking (three requests closely spaced
in the middle of run, and two more in the end of the run).