If you obtain the 'unable to connect' from one, or more, of the FEE64s try the
following procedures *before* power-cycling/rebooting all of the FEE64s:

1) 'unable to connect' whilst DAQ is going using the Merger
   
   https://elog.ph.ed.ac.uk/AIDA/303

2) determine whether multiple FEE64s are unable to connect

aidas1> ping nnaida1
PING nnaida1 (10.1.1.1) 56(84) bytes of data.
64 bytes from nnaida1 (10.1.1.1): icmp_seq=1 ttl=64 time=2.81 ms
64 bytes from nnaida1 (10.1.1.1): icmp_seq=2 ttl=64 time=2.81 ms
64 bytes from nnaida1 (10.1.1.1): icmp_seq=3 ttl=64 time=2.80 ms
^C
--- nnaida1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2029ms
rtt min/avg/max/mdev = 2.807/2.812/2.817/0.061 ms
aidas1> ping nnaida2
PING nnaida2 (10.1.1.2) 56(84) bytes of data.
64 bytes from nnaida2 (10.1.1.2): icmp_seq=1 ttl=64 time=2.87 ms
64 bytes from nnaida2 (10.1.1.2): icmp_seq=2 ttl=64 time=2.81 ms
64 bytes from nnaida2 (10.1.1.2): icmp_seq=3 ttl=64 time=2.83 ms
64 bytes from nnaida2 (10.1.1.2): icmp_seq=4 ttl=64 time=2.81 ms
^C
--- nnaida2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3350ms
rtt min/avg/max/mdev = 2.810/2.833/2.874/0.069 ms

  :
  :
   
  etc

If you find that you are unable to ping a group (or groups) of 8x FEE64s
it is probable that the issue is a fuse failure(s) in the USB-controlled
ac mains relay. It will be necessary to replace the fuse(s) and perform a
cold start of the FEE64s

 https://elog.ph.ed.ac.uk/AIDA/418

3) If you are able to ping all FEE64s telnet to the FEE64 you are unable to
connect to, login as root, and issue a reboot command

aidas1> telnet nnaida2
Trying 10.1.1.2...
Connected to nnaida2.
Escape character is '^]'.

Linux 2.6.31 (localhost) (08:23 on Thursday, 01 November 2018)

login: root
Password: 
Last login: Mon May 23 16:02:30 from myserver
-bash-3.2# ls
a.out  ld_aidamem.csh  xaida
-bash-3.2# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Oct28 ?        00:00:07 init [3]  
root         2     0  0 Oct28 ?        00:00:00 [kthreadd]
root         3     2  0 Oct28 ?        00:00:00 [ksoftirqd/0]
root         4     2  0 Oct28 ?        00:00:00 [watchdog/0]
root         5     2  0 Oct28 ?        00:00:00 [events/0]
root         6     2  0 Oct28 ?        00:00:00 [khelper]
root         7     2  0 Oct28 ?        00:00:00 [async/mgr]
root         8     2  0 Oct28 ?        00:00:00 [kblockd/0]
root         9     2  0 Oct28 ?        00:00:00 [kseriod]
root        10     2  0 Oct28 ?        00:00:00 [khungtaskd]
root        11     2  0 Oct28 ?        00:00:00 [pdflush]
root        12     2  0 Oct28 ?        00:00:00 [pdflush]
root        13     2  0 Oct28 ?        00:00:00 [kswapd0]
root        14     2  0 Oct28 ?        00:00:00 [aio/0]
root        15     2  0 Oct28 ?        00:00:00 [nfsiod]
root        20     2  0 Oct28 ?        00:00:00 [81400400.hd-xps]
root        21     2  0 Oct28 ?        00:00:00 [81400000.xps-sp]
root        22     2  0 Oct28 ?        00:00:00 [kpsmoused]
root        25     2  0 Oct28 ?        00:00:00 [rpciod/0]
root        54     1  0 Oct28 ?        00:00:00 /sbin/udevd -d
root       226     1  0 Oct28 ?        00:00:00 syslogd -m 0
root       229     1  0 Oct28 ?        00:00:00 klogd -x
root       259     1  0 Oct28 ?        00:00:00 rpcbind
root       275     1  0 Oct28 ?        00:00:00 xinetd -stayalive -pidfile /var/
root       377     1 98 Oct28 ?        3-22:18:16 ./AidaExecV8
root       392     1  0 Oct28 ttyS0    00:00:00 /sbin/mingetty --noclear console
root       404   275  0 08:23 ?        00:00:00 in.telnetd: myserver
root       405   404  0 08:23 ?        00:00:00 login -- root            
root       406   405  1 08:23 ttyp0    00:00:00 -bash
root       427   406  6 08:23 ttyp0    00:00:00 ps -ef
-bash-3.2# reboot

Broadcast message from root (ttyp0) (Thu Nov  1 08:25:20 2018):

The system is going down for reboot NOW!
-bash-3.2# Connection closed by foreign host.

Wait 5 minutes for the filesystem to be mounted and the FEE64 boot sequence
to complete. 

Switch to Desktop 1 and re-select 'Data Acquisition Run Control' tab and select 
Update. The status of the FEE64 you were unable to connect to should now 'undefined'.

Follow cold start sequence steps 7-10 and 15

 https://elog.ph.ed.ac.uk/AIDA/418

@ CAEN N1419

Set programmable parameters - see CAEN N1419 manual p.16 (attachment 1)

INTERLOCK = CLOSED
CONTROL = REMOTE

Set front panel switches HV on/off/kill = ON
No interlock jumpers




@ AIDA server

From a command terminal, login to nnrpi1 using ssh, set device permissions, and launch putty.

ssh -X nnrpi1 -l npg
password:
    :
    :
    :
% sudo chmod 777 /dev/ttyUSB2
% putty &

From the putty window sequentially select 'USB2', 'load' and 'open' - this will select
and load a saved serial session for device /dev/ttyUSB2 and launch a command terminal
for the serial session.

Enter the command 'caen' followed by return. Note the command is not echoed.

The CAEN HV interface should appear. Enter the command 'd' to display/modify parameters.
See CAEN N1419 manual pp.27-29 for further commands. See attachment 2.

N.B. The above procedure assumes:

/dev/ttyUSB0 = USB-controlled ac mains switch      (device pl2303)
/dev/ttyUSB1 = serial link to FEE64 system console (device FT232RL)
/dev/ttyUSB2 = CAEN N1419                          (device FT232BM)

However, these assignments may vary. You can use the command dmesg to check assignments.
For example,

% dmesg | grep usb
[    1.178926] usbcore: registered new interface driver usbfs
[    1.184722] usbcore: registered new interface driver hub
[    1.190218] usbcore: registered new device driver usb
[    1.568604] usbcore: registered new interface driver smsc95xx
[    2.065233] dwc_otg bcm2708_usb: DWC OTG Controller
[    2.075294] dwc_otg bcm2708_usb: new USB bus registered, assigned bus number 1
[    2.087646] dwc_otg bcm2708_usb: irq 32, io mem 0x00000000
[    2.115566] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.127464] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.139731] usb usb1: Product: DWC OTG Controller
[    2.149477] usb usb1: Manufacturer: Linux 3.18.7+ dwc_otg_hcd
[    2.160291] usb usb1: SerialNumber: bcm2708_usb
[    2.190521] usbcore: registered new interface driver usb-storage
[    2.348492] usbcore: registered new interface driver usbhid
[    2.360241] usbhid: USB HID core driver
[    2.677666] usb 1-1: new high-speed USB device number 2 using dwc_otg
[    2.897985] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[    2.910008] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.217656] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    3.327968] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[    3.340138] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.422390] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet,
b8:27:eb:0b:f6:73
[    3.517666] usb 1-1.3: new full-speed USB device number 4 using dwc_otg
[    3.630124] usb 1-1.3: New USB device found, idVendor=067b, idProduct=2303
[    3.642534] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.655514] usb 1-1.3: Product: USB-Serial Controller D
[    3.666257] usb 1-1.3: Manufacturer: Prolific Technology Inc. 
[    3.761138] usb 1-1.4: new low-speed USB device number 5 using dwc_otg
[    3.894746] usb 1-1.4: New USB device found, idVendor=046d, idProduct=c016
[    3.908414] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.922059] usb 1-1.4: Product: Optical USB Mouse
[    3.933904] usb 1-1.4: Manufacturer: Logitech
[    3.955374] input: Logitech Optical USB Mouse as
/devices/platform/bcm2708_usb/usb1/1-1/1-1.4/1-1.4:1.0/0003:046D:C016.0001/input/input0
[    3.976035] hid-generic 0003:046D:C016.0001: input,hidraw0: USB HID v1.10 Mouse [Logitech Optical USB Mouse]
on usb-bcm2708_usb-1.4/input0
[    4.087698] usb 1-1.5: new low-speed USB device number 6 using dwc_otg
[    4.343360] usb 1-1.5: New USB device found, idVendor=04f2, idProduct=0402
[    4.359726] usb 1-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    4.377627] usb 1-1.5: Product: USB Keyboard
[    4.377641] usb 1-1.5: Manufacturer: Chicony
[    4.456521] input: Chicony USB Keyboard as
/devices/platform/bcm2708_usb/usb1/1-1/1-1.5/1-1.5:1.0/0003:04F2:0402.0002/input/input1
[    4.475410] hid-generic 0003:04F2:0402.0002: input,hidraw1: USB HID v1.11 Keyboard [Chicony USB Keyboard] on
usb-bcm2708_usb-1.5/input0
[    4.634626] input: Chicony USB Keyboard as
/devices/platform/bcm2708_usb/usb1/1-1/1-1.5/1-1.5:1.1/0003:04F2:0402.0003/input/input2
[    4.657391] hid-generic 0003:04F2:0402.0003: input,hiddev0,hidraw2: USB HID v1.11 Device [Chicony USB
Keyboard] on usb-bcm2708_usb-1.5/input1
[    7.431307] usbcore: registered new interface driver usbserial
[    7.659127] usbcore: registered new interface driver usbserial_generic
[    7.806673] usbserial: USB Serial support registered for generic
[    8.078137] usbcore: registered new interface driver pl2303
[    8.408967] usbserial: USB Serial support registered for pl2303
[    9.088988] usb 1-1.3: pl2303 converter now attached to ttyUSB0                          <- FEE64 console
[   13.617763] usb 1-1.2: new high-speed USB device number 7 using dwc_otg
[   13.749727] usb 1-1.2: New USB device found, idVendor=05e3, idProduct=0608
[   13.775541] usb 1-1.2: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[   13.807792] usb 1-1.2: Product: USB2.0 Hub
[   20.127979] usb 1-1.2.2: new full-speed USB device number 8 using dwc_otg
[   20.294212] usb 1-1.2.2: New USB device found, idVendor=0403, idProduct=6001
[   20.323567] usb 1-1.2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   20.357627] usb 1-1.2.2: Product: FT232R USB UART
[   20.397680] usb 1-1.2.2: Manufacturer: FTDI
[   20.415415] usb 1-1.2.2: SerialNumber: A801LOA8
[   20.542107] usbcore: registered new interface driver ftdi_sio
[   20.569826] usbserial: USB Serial support registered for FTDI USB Serial Device
[   20.628458] usb 1-1.2.2: Detected FT232RL
[   20.651619] usb 1-1.2.2: FTDI USB Serial Device converter now attached to ttyUSB1         <- ac mains switch
[   23.708086] usb 1-1.2.3: new full-speed USB device number 9 using dwc_otg
[   23.824265] usb 1-1.2.3: New USB device found, idVendor=0403, idProduct=6001
[   23.824306] usb 1-1.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   23.824326] usb 1-1.2.3: Product: USB <-> Serial
[   23.824346] usb 1-1.2.3: Manufacturer: FTDI
[   23.833176] usb 1-1.2.3: Detected FT232BM
[   23.834515] usb 1-1.2.3: FTDI USB Serial Device converter now attached to ttyUSB2         <- CAEN N1419
[  775.531187] ftdi_sio ttyUSB2: usb_serial_generic_read_bulk_callback - urb stopped: -32
[  775.531296] ftdi_sio ttyUSB2: usb_serial_generic_read_bulk_callback - urb stopped: -32
[  775.705443] usb 1-1.2.3: USB disconnect, device number 9
[  780.036554] usb 1-1.2.3: new full-speed USB device number 10 using dwc_otg
[  780.153082] usb 1-1.2.3: New USB device found, idVendor=0403, idProduct=6001
[  780.153150] usb 1-1.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  780.153174] usb 1-1.2.3: Product: USB <-> Serial
[  780.153193] usb 1-1.2.3: Manufacturer: FTDI
[  780.167981] usb 1-1.2.3: Detected FT232BM
[  780.169356] usb 1-1.2.3: FTDI USB Serial Device converter now attached to ttyUSB2         <- CAEN N1419
[ 2364.526816] ftdi_sio ttyUSB2: usb_serial_generic_read_bulk_callback - urb stopped: -32
[ 2364.526982] ftdi_sio ttyUSB2: usb_serial_generic_read_bulk_callback - urb stopped: -32
[ 2364.657165] usb 1-1.2.3: USB disconnect, device number 10
[ 2365.657499] usb 1-1.2.3: new full-speed USB device number 11 using dwc_otg
[ 2365.774075] usb 1-1.2.3: New USB device found, idVendor=0403, idProduct=6001
[ 2365.774116] usb 1-1.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 2365.774142] usb 1-1.2.3: Product: USB <-> Serial
[ 2365.774268] usb 1-1.2.3: Manufacturer: FTDI
[ 2365.788695] usb 1-1.2.3: Detected FT232BM
[ 2365.790604] usb 1-1.2.3: FTDI USB Serial Device converter now attached to ttyUSB2         <- CAEN N1419

If using >1 CAEN N1419 check the output of the command 'demsg usb' for FT232BM devices and identify
the USB device numbers. 

You should then set the device permissions for *each* /dev/ttyUSB[n] and start a putty session for *each*
CAEN N1419 module. You can distinguish CAEN N1419 modules by the address (Addr) label shown in the CAEN
display. 

From 'Control' tab select 'Save/Restore Settings'

From 'Save/Restore Module Settings'

     select 'Reload'

     select 'Module Names' whose settings should be saved (usually all)


     select 'Generate new DB key' and make a note of the DB key generated
          the DB key will have the format YYYYMMMDD-HH.MM.SS

          The DB key will be used to name a directory to store AIDA settings
          and is created uid: npg, gid: npgstaff, perms 755. However, its
          contents are written uid: nfsnobody, gid nfsnobody. The default
          directory (DB Key) permissions do not work - change the permissions
          to 777 - before attempting to save data to this directory (DB Key).


     change the permissions of the DB key directory using the command

     chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/YYYYMMMDD-HH.MM.SS


     select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
     the directory (DB key)

Example

aidas1> ls /MIDAS/DB/EXPERIMENTS/2016May03-20.44.33
CONTENTS  nnaida13  nnaida18  nnaida22  nnaida27  nnaida31  nnaida7
nnaida1   nnaida14  nnaida19  nnaida23  nnaida28  nnaida32  nnaida8
nnaida10  nnaida15  nnaida2   nnaida24  nnaida29  nnaida4   nnaida9
nnaida11  nnaida16  nnaida20  nnaida25  nnaida3   nnaida5
nnaida12  nnaida17  nnaida21  nnaida26  nnaida30  nnaida6
aidas1> ls /MIDAS/DB/EXPERIMENTS/2016May03-20.44.33/nnaida1
CONTENTS
aidas1> more /MIDAS/DB/EXPERIMENTS/2016May03-20.44.33/nnaida1/CONTENTS
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.2.20 string 0x68
ASIC.1.4 string 0x07
ASIC.2.21 string 0x08
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x00
ASIC.2.26 string 0xbf
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x0d
ASIC.2.3 string 0x02
ASIC.3.12 string 0x10
ASIC.2.4 string 0x07

etc


To ensure that the new settings continue to be used through system power cycles
you define the DB key directory to use in AIDA Options as follows:

From 'Experiment Control' tab select 'Save/Restore Options'


From 'Options' tab

     select 'Reload'

     select 'Act on ALL Data Acquisition Servers?'

     enter directory (DB Key) in the 'ASIC.settings' field

     select 'Save Options'

Example

aidas1> ls /MIDAS/DB/EXPERIMENTS/AIDA/Options
nnaida1   nnaida13  nnaida17  nnaida20  nnaida24  nnaida28  nnaida31  nnaida6
nnaida10  nnaida14  nnaida18  nnaida21  nnaida25  nnaida29  nnaida32  nnaida7
nnaida11  nnaida15  nnaida19  nnaida22  nnaida26  nnaida3   nnaida4   nnaida8
nnaida12  nnaida16  nnaida2   nnaida23  nnaida27  nnaida30  nnaida5   nnaida9
aidas1> ls /MIDAS/DB/EXPERIMENTS/AIDA/Options/nnaida1
CONTENTS
aidas1> cat /MIDAS/DB/EXPERIMENTS/AIDA/Options/nnaida1/CONTENTS
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift
&&MACB_TRIG_MODE&&Aida.offset&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Ai
da.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&Stat.shift&&Aida
.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_H
ist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 1
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 14
Rate.channels string 150
Aida_GroupBase string 1
Stat.channels string 150
Aida_Hist_D_Enable string 1
ASIC.settings string 2016May03-20.44.33
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
Stat.shift string 6
Aida.Wchannels string 1020
Aida_Hist_V_Enable string 0
Include.Aida string 1
Aida.channels string 65536
Aida_Hist_H_Enable string 1
DataFormat string AIDA
DataAcqPgm string AidaExecV8
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1

From the 'AIDA Control' tab select 'Local Controls'

From the 'LOCAL controls' tab 

     select 'Reload'
     select FEE64 required
     select (or deselect) 'Act on ALL FEE64 modules?' as required
     enter 'Trigger O/P control register' value required

The current default value (set by AIDA Options) is 0xe (logic 0)

When logged in and before opening PuTTY open a cmdtool window.

cd /dev

sudo chmod 777 /dev/ttyUSB0

then you should be able to run PuTTY.

This is based on notes in my log-book.

Tried this when we got in this morning and successfully changed the permissions for /dev/ttyUSB0, but made no difference to the error received when opening the PuTTY session, which was:

            PuTTy Fatal Error

            Unable to open connection to:

            Unable to open serial port.

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!



How To Start Run 

1) Workspace 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check DAQ stopped (green 'GO' button displayed)

   Check state of *all* data acquisition servers is 'stopped'
   Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags

   See attachment 1

2) Workspace 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check TapeServer stopped and disk file closed (green GO button displayed)

   See attachment 2

3) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check Merge Program paused - see banner 

   Merge State = GOing     :    paused     :    xfer enabled     :    output paused

   If not paused select 'Toggle Merge Program Pause State'

   See attachment 3

4) Workspace 1 - Firefox web browser - ASIC Control tab

   Select 'Reload'

   Check 'Act on ALL FEE64 modules?' selected

   Check 'Act on ALL ASICs' selected

   Select 'Select required function' -> 'Check ASIC Control'

   After ~60s a pop-up will appear showing the response from each ASIC of each data acquisition server.
   Check all ASICs are OK and status = 0x001F0000.
   Dismiss pop-up by selecting OK

   See attachments 4-5

5) Workspace 1 - Firefox web browser - Run Control tab 

   Start DAQ - select green GO button

   Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates 

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags

   See attachment 6

6) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check *all* enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are inactive call for expert help and do NOT proceed!

   Observe active spinner in MERGER server terminal window

   See attachment 7

7) Workspace 2 - Firefox web browser - TapeServer tab

   Check 'Volume Name' and 'File Name'

   Start TapeServer and open disk file - select green 'GO' button

   See attachment 8

8) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check all enabled data links (0-23) are always active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are *always* inactive call for expert help and do NOT proceed!

   Select 'Toggle Merge Program Pause State' to enable data merging 

   Select 'Reload' to observe merge rate (~ 9000 data items/s)

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

   See attachment 9

9) Workspace 1 - Firefox web browser - Run Control tab

   Select 'Start ASIC Data Readout'

   See attachment 10

10) Workspace 2 - Firefox web browser - MERGE tab

    Select Reload several times and check all enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

    N.B. If some data links are *never* active call for expert help and do NOT proceed!

    Select 'Reload' to observe merge rate (~ 0.5-2M data items/s)

    See attachment 11

11) Workspace 2 - Firefox web browser - TapeServer tab

    Select 'Reload' and observe number of blocks written to disk increasing

    See attachment 12

12) Workspace 1 - Firefox web browser - Control tab

    Select 'Save/Restore Settings'

    From 'Save/Restore Module Settings' tab

     select 'Reload'

     select 'Module Names' whose settings should be saved (usually all)

     select 'Generate new DB key' and make a note of the DB key generated
          the DB key will have the format YYYYMMMDD-HH.MM.SS

          The DB key will be used to name a directory to store AIDA settings
          and is created uid: npg, gid: npgstaff, perms 755. However, its
          contents are written uid: nfsnobody, gid nfsnobody. The default
          directory (DB Key) permissions do not work - change the permissions
          to 777 - before attempting to save data to this directory (DB Key).

     launch a terminal and change the permissions of the DB key directory using the command

     chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/[YYYYMMMDD-HH.MM.SS]

     select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
     the directory (DB key)

11) Update BRIKEN Elog with the following information

    AIDA
    run filename (e.g. R99)
    [# data blocks]
    start date & time
    [stop date & time]
    ASIC settings DB key 
    additional comments
    your name

    Complete run sheet for written logbook

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!



How To Start Run 

1) Workspace 1 - Firefox web browser - Run Control tab

   Select Update

   Check DAQ stopped (green GO button displayed)

   Check state of *all* data acquisition servers is 'stopped' and histogramming and data transfer 
   is enabled indicated by 'H X'

   See attachment 1

2) Workspace 2 - Firefox web browser - TapeServer tab

   Check Reload

   Check TapeServer stopped and disk file closed (green GO button displayed)

   See attachment 2

3) Workspace 2 - Firefox web browser - MERGE tab

   Check Reload

   Check Merge Program paused - see banner 

   Merge State = GOing     :    paused     :    xfer enabled     :    output paused

   If not paused select Toggle Merge Program Pause State

   See attachment 3

4) Workspace 1 - Firefox web browser - ASIC Control tab

   Check Reload

   Check 'Act on ALL FEE64 modules?' and wait for request to be completed

   Check 'Act on ALL ASICs' and wait for request to be completed

   From 'Select required function' select 'Check ASIC Control'
   After ~60s a pop-up will appear showing the response from each ASIC of each data acquisition server.
   Check all ASICs are OK and status = 0x001F0000.
   Dismiss pop-up by selecting OK

   See attachments 4-5

5) Workspace 1 - Firefox web browser - Run Control tab 

   Start DAQ - select green GO button

   Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates 

   Select 'Data Acquisition Statistics'

6) Workspace 1 - Firefox web browser - Statistics tab 

   Check Reload

   Check 'Show ALL Data Acquisition Servers' 

   From 'Select counter to Monitor' select 'AIDA SYNC'

   Select 'Update Once'

   Check *all* data acquisition servers have a non-zero rate (typically ~300-500Hz)

   See attachment 6

   N.B. If one, or more, data acquisition servers have zero rate

        return to Run Control tab

           stop DAQ (select red STOP button)

           check state of *all* data acquisition servers is 'stopped' with histogramming and
           data transfer are enabled indicated by 'H X'

        start DAQ (select green GO button)

           check state of *all* data acquisition servers is 'going' with histogramming and
           data transfer are enabled indicated by 'H X'

        return to Statistics tab

           check AIDA SYNC rates again and if zero rate persists for one, or more, data acquisition
           servers call for expert help and do *NOT* proceed!

7) Workspace 2 - Firefox web browser - TapeServer tab

   Start TapeServer and open disk file - select green GO button

8) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check all enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are *always* inactive call for expert help and do NOT proceed!

   Select Toggle Merge Program Pause State to enable data merging 

   Select Reload to observe merge rate (data items/s)

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

   See attachment 7

9) Workspace 2 - Firefox web browser - TapeServer tab

   Select Reload and observe number of blocks written to disk

10) Workspace 1 - Firefox web browser - Control tab

    Select 'Save/Restore Settings'

    From 'Save/Restore Module Settings' tab

     select 'Reload'

     select 'Module Names' whose settings should be saved (usually all)

     select 'Generate new DB key' and make a note of the DB key generated
          the DB key will have the format YYYYMMMDD-HH.MM.SS

          The DB key will be used to name a directory to store AIDA settings
          and is created uid: npg, gid: npgstaff, perms 755. However, its
          contents are written uid: nfsnobody, gid nfsnobody. The default
          directory (DB Key) permissions do not work - change the permissions
          to 777 - before attempting to save data to this directory (DB Key).

     launch a terminal and change the permissions of the DB key directory using the command

     chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/[YYYYMMMDD-HH.MM.SS]

     select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
     the directory (DB key)

11) Update EURICA 2016 Elog with the following information

    AIDA
    run filename (e.g. R99)
    [# data blocks]
    start date & time
    [stop date & time]
    ASIC settings DB key
    AIDA trigger (e.g. OR DSSSDs #1 - #6) 
    additional comments
    your name

    Complete run sheet for written logbook



How to Stop Run

1) Workspace 2 - Firefox web browser - MERGE tab

   Pause MERGE input - select Toggle Merge Program Pause State

2) Workspace 1 - Firefox web browser - Run Control tab

   Select Update

   Stop DAQ - select red STOP button

   Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates 

   Check state of *all* data acquisition servers is 'stopped'

3) Workspace 2 - Firefox web browser - TapeServer tab

   Stop TapeServer and close disk file - select red STOP button

4) Complete EURICA 2016 Elog with the following information

    AIDA
    run filename (e.g. R99)
    [# data blocks]
    start date & time
    [stop date & time]
    ASIC settings DB key
    AIDA trigger (e.g. OR DSSSDs #1 - #6) 
    additional comments
    your name

    Complete run sheet for written logbook




   

1) Start Julabo FL11006 Recirculating Chiller

   See manual (page 19)  http://www2.ph.ed.ac.uk/~td/AIDA/Information/Julabo%20FL11006.pdf 
    
   Set point +20 deg C

   Check flow of coolant by inspecting the two flow indicators the left and right hand sides
   on top of the AIDA stand

   Do NOT proceed if the Julabo FL11006 Recirculating Chiller is not operating, or if there is
   little, or no, flow in one, or both, flow indicators.

2) Start interface to USB-controlled ac mains relay to AIDA data acquisition servers

   On the AIDA Dell T410 Workstation aidas1 switch to Desktop 7 and atart a Firefox web browser
   
   Enter URL   http://nnrpi1:8015/AIDA/Rly16/

   See attachment 1

   Select 'Sequence ALL On' to powerup the AIDA data acquisition servers

   Check all relays Relay 1 to Relay 8 move from the 'Switch ON' to 'Switch OFF' row

   See attachment 2

   Wait 8 minutes (yes - really!) before proceeding

3) Start Httpd Data Acquisition server

   See attachment 3

   Switch to Desktop 1 and select MIDAS yellow icon 'Httpd for DataAcquisition' which will start a terminal
   to display server messages as follows

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-504.8.1.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS@aidas and MIDAS_LIBRARY = /MIDAS@aidas/TclHttpd/Linux64
PATH =
/MIDAS@aidas/bin_Linux64:/MIDAS@aidas/TclHttpd/Linux64:/homes/npg/root/bin:/homes/npg/root/bin:/homes/npg/root/lib:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "jhi+u8jzdbpl"
httpd started on port 8015

Custom startup from /MIDAS@aidas/config/TclHttpd/aidas1@8015/startup.tcl
DefineMessage defined with server at http://elog.ph.ed.ac.uk:8015/MessageLoggerServer
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
loading tcl/AIDARunControl.tcl for namespace ::
RunControlClient provided
RunControlClient loaded
Completed custom startup from /MIDAS@aidas/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl

   See attachment 4

4) Start web browser for interface to AIDA

   Switch to Desktop 1 and start Firefox web browser

   Enter URL localhost:8015

   See attachments 5-6

5) Commence startup of AIDA interface

   Select AIDA Experiment Control

6) From AIDA Experiment Control tab 

   See attachment 7

   Select AIDA Hardware Control

   See attachment 8

   Re-select AIDA Experiment Control tab

   Select Data Acquisition Run Control

7) From Data Acquisition Run Control tab

   See attachment 9

   Select Act On ALL Data Acquisition Servers

   See attachment 10

   Select Reset

   Wait for update and check state of *all* data acquisition servers is 'reset'

   Select Setup

   See attachment 11

   Wait for timestamp re-synchronisation pop-up

   See attachment 12

   Usually all data acquisition servers (nnaida1, nnaida2 ... etc) will ReSYNC OK
   Sometimes one, or more, data acquisition servers report ReSYNC not complete

   Re-select AIDA Hardware Control tab and select Master Timestamp Control

   From the Master Timestamp Control tab

   Select Reload

   Select FEE64 -> nnaida6

   Select Perform ReSYNC

   See attachment 13

   Wait for timestamp re-synchronisation pop-up

   See attachments 14-15


   
8) AIDA: Master Timestamp Control

Ditto ReSync

7) Data Acquisition Run Control

8) Spectrum Browser

Assumes 3-phase ac power (to Julabo FL11006 recirculating chiller) and single-phase ac power
(to AIDA NIM bins, PSUs, network switches, USB-controlled ac mains relay and Raspberry Pi)
is connected and switched on.


All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


1) Start Julabo FL11006 Recirculating Chiller

   See manual (page 19)  http://www2.ph.ed.ac.uk/~td/AIDA/Information/Julabo%20FL11006.pdf 
    
   Set point +20 deg C

   Check flow of coolant by inspecting the two flow indicators which are located on top of
   the AIDA stand

   Do NOT proceed if the Julabo FL11006 Recirculating Chiller is not operating, or if there
   is little, or no, flow in one, or both, flow indicators.

2) Start interface to USB-controlled ac mains relay to AIDA data acquisition servers

   On the AIDA Dell T410 Workstation aidas1 switch to Desktop 7 and start a Firefox web browser
   
   Enter URL   http://nnrpi1:8015/AIDA/Rly16/

   See attachment 1

   Select 'Sequence ALL On' to powerup the AIDA data acquisition servers

   Check all relays Relay 1 to Relay 8 move from the 'Switch ON' to 'Switch OFF' row

   See attachment 2

   Wait 10 minutes (yes - really!) before proceeding

3) Start Httpd Data Acquisition server

   See attachment 3

   Switch to Desktop 1 and select the yellow MIDAS icon 'Httpd for DataAcquisition' which will start a terminal
   to display server messages as follows

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-504.8.1.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS@aidas and MIDAS_LIBRARY = /MIDAS@aidas/TclHttpd/Linux64
PATH =
/MIDAS@aidas/bin_Linux64:/MIDAS@aidas/TclHttpd/Linux64:/homes/npg/root/bin:/homes/npg/root/bin:/homes/npg/root/lib:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "jhi+u8jzdbpl"
httpd started on port 8015

Custom startup from /MIDAS@aidas/config/TclHttpd/aidas1@8015/startup.tcl
DefineMessage defined with server at http://elog.ph.ed.ac.uk:8015/MessageLoggerServer
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
loading tcl/AIDARunControl.tcl for namespace ::
RunControlClient provided
RunControlClient loaded
Completed custom startup from /MIDAS@aidas/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl

   See attachment 4

4) Start web browser for interface to AIDA

   Switch to Desktop 1 and start Firefox web browser

   Enter URL     localhost:8015

   See attachment 5

5) Commence startup of AIDA interface

   Select 'AIDA Experiment Control'

   See attachment 6

6) From AIDA Experiment Control tab 

   Select 'AIDA Hardware Control'

   See attachment 7

   Select 'Data Acquisition Run Control'

7) Reset and setup all data acquisition servers

   From 'Data Acquisition Run Control' tab

   See attachment 8

   Select 'Act On ALL Data Acquisition Servers'

   The web page will update (~20s) to show the status of all of the configured AIDA FEE64
   data acquisition servers which are named nnaida1, nnaida2 ... etc

   See attachment 9

   Select 'Reset'

   Wait for update and check that the state of *all* data acquisition servers is 'reset'

   Select 'Setup'

   Wait for timestamp re-synchronisation pop-up

   See attachments 10-11

   Usually all data acquisition servers (nnaida1, nnaida2 ... etc) will ReSYNC OK and you may proceed to step 8.

   Sometimes one, or more, data acquisition servers report ReSYNC not complete.

   Re-select the 'AIDA Experiment Control' tab and select 'AIDA Hardware Control'

   See attachments 12-13

   From the 'AIDA Hardware Control' tab select 'Master Timestamp Control'

   From the 'Master Timestamp Control' tab

   Select 'Reload'

   Select FEE64 -> nnaida6

   Select 'Perform ReSYNC'

   Wait for timestamp re-synchronisation pop-up.
   All data acquisition servers 9Nnaida1, nnaoida2 .. etc) should ReSYNC OK - if they do not
   please seek expert help

8) Start ASIC controls

   Re-select the 'AIDA Experiment Control' tab and select 'AIDA Hardware Control'

   See attachments 12-13

   Select 'ASIC Control'

   See attachment 14

   From 'ASIC Control' tab
     select 'Act on ALL FEE64 modules' and await page update
     select 'Act on ALL ASICs' and await page update

   See attachment 15

9) Check FEE64 operating temperatures

   Re-select the 'AIDA Hardware Control' tab and select 'FEE Temperatures'

   See attachment 16

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C

   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0
   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows
   
   Desktop 6 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off' and *now* seek expert help

10) Perform system-wide diagnostic checks

    Re-select the 'AIDA Hardware Control' tab and select 'System Wide Checks'

    See attachments 17-18

11) Start servers for data merging and output to disk

    Switch to Desktop 2 and start the 'Httpd for Merge/TapeServer', 'Merger' and 'TapeServer' servers
    by selecting the three adjacent MIDAS in sequence

    See attachment 19

    Three terminal sessions will pop-up to display server messages. These terminals should be moved to
    left of the desktop and arranged as shown 

    See attachment 20

    This will enable you to simultaneously observe the latest messages from each server and the activity
    spinners of the Merger and TapeServer servers 

12) Start web interface to Merger and TapeServer

    Start Firefox web browser

    Enter URL     localhost:8115

    See attachment 21

13) Open data file

    Select 'Tape Service'

    See attachment 22

    In the 'TapeServer Service' tab complete the Volume Name, Run Name and Run Number fields.
    Typically the volume name will the experiment ID (e.g. NP1234) and Run Name (R). 

    See attachment 23

    Select 'Go' 

    See attachment 24

    This will create an experiment directory if it does not already exist (e.g. /TapeData/NP1234)
    and will open a data file within this directory with a filename concatenating the Run Name
    and Run Number fields - if the selected file already exists a new filename (Run Number) will
    be suggested

14) Setup merger

    Re-select the 'AIDA' tab and select 'MERGE Control'

    See attachment 25

    Select 'Setup & Configure'

    See attachment 26

    Select 'Go'

    See attachment 27

    Select 'Toggle Merge Program Pause State' 

    See attachment 28

15) Enable histogramming and data transfer #1 links

    Switch to Desktop 1 and re-select 'Data Acquisition Run Control' tab

    Select Histogramming 'Enable All' and Data Transfer #1 'Enable All'

16) AIDA is now ready to commence collecting data 
    - you can proceed from step 5 of https://elog.ph.ed.ac.uk/AIDA/412

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


How to Stop Run

1) Workspace 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check number of blocks written to disk is increasing

   See attachment 1

   If TapeServer data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303

2) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~0.5-2M data items/s)

   See attachment 2

   If MERGER data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303

3) Workspace 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags

   Stop DAQ - select red 'STOP' button

   Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates 

   Check state of *all* data acquisition servers is 'stopped'
   Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags

   See attachment 3

   N.B. Occasionally the state of all data acquisition servers does not change to 'stopped'
        and if this happens simply re-try by re-selecting 'STOP' 

4) Workspace 2 - Firefox web browser - MERGE tab

   Pause MERGE input - select 'Toggle Merge Program Pause State'

   See attachment 4

5) Workspace 2 - Firefox web browser - TapeServer tab

   Stop TapeServer and close disk file - select red 'STOP' button

   See attachment 5

6) Complete BRIKEN Elog with the following information

    AIDA
    run filename (e.g. R99)
    [# data blocks]
    start date & time
    [stop date & time]
    ASIC settings DB key
    additional comments
    your name

    Complete run sheet for written logbook

The function and setting of the AIDA discriminators is as follows:

                  LEC (20MeV FSR)    MEC (1GeV FSR)    HEC (20GeV FSR)    Function

slow comparator   10keV/unit         500keV/unit       n/a                controls LEC/MEC ADC readout

fast comparator   100keV/unit        5MeV/unit         100MeV/unit        controls LEC/MEC disc readout
                                                                          controls HEC ADC readout


For example, a slow comparator setting of 0xa corresponds to a threshold of 100keV (LEC)
or 5MeV (MEC), and the default HEC fast comparator setting of 0x2 corresponds to 200MeV. 

If you obtain the 'unable to connect' from one, or more, of the FEE64s try the
following procedure *before* power-cycling/rebooting all of the FEE64s. 

1) From the Firefox web browser (Workspace 2) close any tab related to the Merge program
   typically 'MERGE: Control', 'Merger Link Data Rates' and 'MRGE Statistics'

2) Restart the Merge program by selecting the 'MergeServer' icon on the task bar at the top
   of the display which the run the script /MIDAS/Merger/AIDA/Startup

   Restarting the Merge program in this way will ensure that all merge program components
   are removed and that the merge program starts with a timestamp of zero.

   [Attempts to just restart manually which fail to remove ALL of the link tasks will cause
    the data collection for those FEE64 links to an existing link task to fail (port already
    in use).

    It is also important that the Merge restarts with the time stamp at zero and with the link
    queues all empty  (which happens when the software is reloaded)]

3) From the Firefox web browser (Workspace 2) 'MERGE: Control' tab

   Select Reload

   The Merge state should now be 'Unknown'

   Select Configure & Setup

   Select GO

   This should immediately commence reading data from the links and sending merged data to the
   TapeServer if enabled.

   Select Toggle Merge Data Transfer State to toggle data transfer to the TapeServer 

   Select Toggle Merge Program Pause State to toggle input data to the Merge program

Based upon notes from VFEP (originally written for the IDS LyrTech system at CERN)

   Type: Application in Terminal
   Name: Httpd for Data Acquisition
Command: /MIDAS@aidas/TclHttpd/Linux64/TclHttpd-server -port 8015
Comment: TapeServer

   Type: Application in Terminal
   Name: Httpd for Merge/TapeServer
Command: /MIDAS/TclHttpd/Linux64/TclHttpd-server -port 8115
Comment:

   Type: Application in Terminal
   Name: Merger
Command: /MIDAS/Merger/AIDA/Startup
Comment: Merger

   Type: Application in Terminal
   Name: TapeServer
Command: /MIDAS/TapeServer/startup/AidaTapeServer
Comment: TapeServer

aidas1> ls -l /
total 230
dr-xr-xr-x.   2 root root      4096 Nov 23 03:44 bin
dr-xr-xr-x.   5 root root      1024 Feb 17  2015 boot
drwxr-xr-x.  10 root root      4096 Mar 23  2016 cgroup
drwxr-xr-x.  11 root root      4096 Jun 13 17:38 data10
drwxr-xr-x.   2 root root      4096 Apr 28  2015 data20
drwxr-xr-x.   2 root root      4096 Apr 28  2015 data30
drwxr-xr-x.   2 root root      4096 Apr 28  2015 data40
drwxr-xr-x.  21 root root      4560 Nov 26 05:22 dev
drwxr-xr-x. 140 root root     12288 Nov 26 05:25 etc
drwxr-xr-x.   5 root root      4096 Nov  1 17:41 home
drwxr-xr-x.   4 root root      4096 Jan  8  2014 homes
dr-xr-xr-x.  13 root root      4096 Jul 20 03:16 lib
dr-xr-xr-x.  11 root root     12288 Nov 23 03:44 lib64
drwx------.   2 root root     16384 Jan  7  2014 lost+found
drwxr-xr-x.   5 root root      4096 Nov 26 05:22 media
lrwxrwxrwx.   1 root root        12 Apr 22  2014 MIDAS -> /MIDAS@aidas
drwxrwxr-x.  29 npg  npgstaff  4096 Jun  9 11:32 MIDAS@aidas
drwxr-xr-x.  22 root root      4096 Dec 12  2015 MIDAS@aidas.121215
drwxr-xr-x.  29 root root      4096 Jun  8 02:35 MIDAS@aidas.240215
drwxr-xr-x.   2 root root         0 Nov 26 05:22 misc
drwxr-xr-x.   2 root root      4096 Jul 21  2011 mnt
drwxr-xr-x.   2 root root         0 Nov 26 05:22 net
drwxrwxrwx.   1 root root      4096 Nov 12 11:44 ntfs
drwxr-xr-x.   4 root root      4096 Feb 27  2014 opt
dr-xr-xr-x. 501 root root         0 Nov 26 05:20 proc
dr-xr-x---.  29 root root      4096 Nov 26 06:33 root
drwxr-xr-x.   2 root root     12288 Nov 22 03:51 sbin
dr-xr-xr-x.   2 root root      4096 Feb 27  2014 scratch
drwxr-xr-x.   7 root root         0 Nov 26 05:20 selinux
drwxr-xr-x.   2 root root      4096 Jul 21  2011 srv
drwxr-xr-x   13 root root         0 Nov 26 05:20 sys
lrwxrwxrwx.   1 root root        16 May 24  2016 TapeData -> /data10/TapeData
drwxrwxrwt.  80 root root     86016 Nov 26 16:27 tmp
drwxr-xr-x.  13 root root      4096 Jan  7  2014 usr
drwxr-xr-x.  23 root root      4096 Jan  7  2014 var
dr-xr-xr-x.   2 root root      4096 Feb 27  2014 WWW

Every 4h

1) Record DSSSD #1 = #6 detector bias voltages and leakage currents in written logbook

   Desktop 6 - 2x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 3 = DSSSD # 4

   Addr 01

   Ch 0 = DSSSD # 5
   Ch 1 = DSSSD # 6

   Compare with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select Reload

   Check temperatures

       Virtex < 75 deg C
       PSU    < 45 deg C
       ASIC   < 65 deg C

   If not call for expert help. 

   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Statistics tab

   Check Reload

   Check 'Show ALL Data Acquisition Servers' 

   From 'Select counter to Monitor' select 'AIDA Discriminator'

   Select 'Update Once'

   Check *all* data acquisition servers have a non-zero rate

   N.B. If one, or more, data acquisition servers have zero rate call for expert help!

   From keyboard select Alt-Print Screen and save screenshot to /tmp directory as a .png file
   Upload the .png file to the EURICA 2016 Elog for the current run

   Compare rates with previous observations at same settings 

4) Desktop 1 - Firefox web browser - Statistics tab

   Check Reload

   Check 'Show ALL Data Acquisition Servers' 

   From 'Select counter to Monitor' select 'AIDA SYNC'

   Select 'Update Once'

   Check *all* data acquisition servers have a non-zero rate (typically ~300-500Hz)

   N.B. If one, or more, data acquisition servers have zero rate call for expert help!

5) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check all enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are *always* inactive call for expert help!

   Select Reload to observe merge rate (date items/s)

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

6) Desktop 2 - Firefox web browser - TapeServer tab

   Select Reload and observe number of blocks written to disk

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


Every 4h

1) Desktop 7 - 2x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 3 = DSSSD # 4

   Addr 01

   Ch 0 = DSSSD # 5
   Ch 1 = DSSSD # 6

   Sequentially select each terminal and obtain a screenshot (Alt-PrtScn) of DSSSD #1 = #6 detector
   bias voltages and leakage currents
   Upload screenshots to BRIKEN Elog

   If B2F/F11 ambient temperature information is available please add this information to the Elog

   Compare DSSSD leakage currents (and temperatures) with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select 'Reload'

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C


   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0

   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows

   Desktop 6 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off'

   and *now* seek expert help


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags

5) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~0.5-2M data items/s) consistent with value observed at start of run

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

6) Desktop 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check data rate consistent with value observed at start of run

aidas1> setenv MIDASBASE /MIDAS@aidas.240215
aidas1> setenv PATH $MIDASBASE/bin_Linux64:$PATH
aidas1> MIDAS64-session

When starting Aida some FEE64 fail to provide SYNC data , or any data for that matter, on the ASIC data stream.
The problem is that the ASIC Readout is not starting properly and is becoming stuck waiting for an internal
event which will never arrive.
The chief source has been found to be the Correlation part of the VHDL. The reason is not yet clear but the work
around of disabling the Correlation part of all of the FEE64 at 'GO' except the Master has allowed the System to
start with all FEE64 producing SYNCs. 

This change is now incorporated in the sys.tcl file which is used when 'GO' is activated in the RunControl window.

The Master is required as it provides the 25Mhz clock and the Reset pulse. The Correlation Trigger will cause a
correlation data item to be put in the Master ASIC data stream.
Only one Correlation data item is required in the data stream, so having just the Master should be fine.

If it is required to enable another FEE64 then do this after 'GO' by opening the Correlation Browser window from
the Control window and set the Control ( at offset 0 ) to 0x0001.

Should one of the FEE64s still fail to provide SYNCs first try to 'STOP' , 'GO' the system two times and see
if that works.

A further means of correcting the problem is to directly reset the ASIC Readout using the Data Readout Window
available through the Control window. In the ASIC Readout - buffer status & control (0x300) section enter 0x4000
into the Control register to reset the state machines. Then enter 0x0001 to re-enable them. This may solve the
problem.

This is all by no means an ideal situation and the source will be found and the problem solved.

The latest version of the FEE64 software Interface document ( May 2016 ) is available as a .pdf from
http://npg.dl.ac.uk/documents/edoc000/#AIDA. EDOC955

The version available via the pink "documentation" button in the Control browser window is now out of date. This
will be updated at some future time.

Other documents that are of use are there also.
MACB specification.
GREAT Data Format.