> 
> 14.00 Both AIDA NIM bins U/S 
>       Upper NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B - ac mains fuse 100V/8A/slow? blowing
>       Lower NIM bin - 11x MACB - not powering up
> 
>       Replaced both NIM bins.
>       Upper NIM bin - 11x MACB
>       Lower NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B
> 
> 15.50 Julabo FL11006 chiller ON - set point +20 deg C
>       B2F/F11 ambient temp +24.0 deg C, RH 54%, DP +13.8 deg C
> 
>       Bias 6x MSL type BB18 DSSSDs - see attachments 1-2
> 
> 16.10 Startup DAQ nnaida1-24 *except* nnaida19
> 
>       System-wide checks
> 
>       Check clock status OK
>       Check ADC calibration - nnaida7 failed
>       Check ASIC clock timestamp - multiple failures - see attachment 3
>       Check SYNC errors not counted Since Baseline - see attachment 4
>       Check SYNC pulses received - see attachment 5
> 
>       BNC PB-4 
> 
>       Amplitude 90,000, x5 attenuator IN
>       t_r 50ns, tau_d 50us
>       Polarity -
>       INT ref, delay MIN, tail pulse
> 
>       FEE64 temperatures - see attachment 6
> 
> 16.34 R1 starts (directory Aug2016)
> 
>       slow comparator 0x10, LEC/MEC fast comparator 0x10
> 
>       merge 555k data items/s, tapeserver 4430 kb/s
Looks like a problem with the inter-MACB wiring.
The screen shots show groups of 4 with the problems.
I suggest re-seating the short HDMI cables and reset the system.

14.00 Both AIDA NIM bins U/S 
      Upper NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B - ac mains fuse 100V/8A/slow? blowing
      Lower NIM bin - 11x MACB - not powering up

      Replaced both NIM bins.
      Upper NIM bin - 11x MACB
      Lower NIM bin - BNC PB-4, Ortec 433, 2x CAEN N1419, 1x CAEN N1419B

15.50 Julabo FL11006 chiller ON - set point +20 deg C
      B2F/F11 ambient temp +24.0 deg C, RH 54%, DP +13.8 deg C

      Bias 6x MSL type BB18 DSSSDs - see attachments 1-2

16.10 Startup DAQ nnaida1-24 *except* nnaida19

      System-wide checks

      Check clock status OK
      Check ADC calibration - nnaida7 failed
      Check ASIC clock timestamp - multiple failures - see attachment 3
      Check SYNC errors not counted Since Baseline - see attachment 4
      Check SYNC pulses received - see attachment 5

      BNC PB-4 

      Amplitude 90,000, x5 attenuator IN
      t_r 50ns, tau_d 50us
      Polarity -
      INT ref, delay MIN, tail pulse

      FEE64 temperatures - see attachment 6

16.34 R1 starts (directory Aug2016)

      slow comparator 0x10, LEC/MEC fast comparator 0x10

      merge 555k data items/s, tapeserver 4430 kb/s

18.28 R4 starts

      Settings per R1

18.31 DAQ/merge stalls with (many) messages as follows

MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
Merge state now Paused
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in pending data, found 1146,
last 575
copyItems: queue 14, maybe copied in wrong buffer: sequence 1146, last 575
Merge_addItems: queue 14, maybe added wrong buffer: sequence 1146, last 575
MERGE Program (24316): check_for_data: index 14, unexpected header sequence number in new data, found 689, last 1258
copyItems: queue 14, maybe copied in wrong buffer: sequence 689, last 1258
Merge_addItems: queue 14, maybe added wrong buffer: sequence 689, last 1258
MERGE Data Link (24254): client closed connection
MERGE Data Link (24254): listening on port 11001
MERGE Data Link (24255): client closed connection
MERGE Data Link (24255): listening on port 11002
MERGE Data Link (24257): client closed connection
MERGE Data Link (24257): listening on port 11004
MERGE Data Link (24258): client closed connection
MERGE Data Link (24258): listening on port 11005
MERGE Data Link (24261): client closed connection
MERGE Data Link (24261): listening on port 11008
MERGE Data Link (24260): client closed connection
MERGE Data Link (24260): listening on port 11007
MERGE Data Link (24259): client closed connection
MERGE Data Link (24259): listening on port 11006
MERGE Data Link (24256): client closed connection
MERGE Data Link (24256): listening on port 11003
MERGE Data Link (24262): client closed connection
MERGE Data Link (24262): listening on port 11009
MERGE Data Link (24263): client closed connection
MERGE Data Link (24263): listening on port 11010
MERGE Data Link (24265): client closed connection
MERGE Data Link (24265): listening on port 11012
MERGE Data Link (24266): client closed connection
MERGE Data Link (24266): listening on port 11013
MERGE Data Link (24267): client closed connection
MERGE Data Link (24267): listening on port 11014
MERGE Data Link (24269): client closed connection
MERGE Data Link (24269): listening on port 11016
MERGE Data Link (24268): client closed connection
MERGE Data Link (24268): listening on port 11015
MERGE Data Link (24270): client closed connection
MERGE Data Link (24270): listening on port 11017
MERGE Data Link (24264): client closed connection
MERGE Data Link (24264): listening on port 11011
MERGE Data Link (24273): client closed connection
MERGE Data Link (24273): listening on port 11020
MERGE Data Link (24271): client closed connection
MERGE Data Link (24271): listening on port 11018
MERGE Data Link (24277): client closed connection
MERGE Data Link (24277): listening on port 11024
MERGE Data Link (24275): client closed connection
MERGE Data Link (24275): listening on port 11022
MERGE Data Link (24274): client closed connection
MERGE Data Link (24274): listening on port 11021
MERGE Data Link (24276): client closed connection
MERGE Data Link (24276): listening on port 11023

Due to sometimes operating with very noisy systems, both at Daresbury and RIKEN, I have installed a change to the startup of the system.

The Runcontrol GO button now only starts the output of SYNC pulses. A further button labelled "Start ASIC Readout" ( see attached photo ) needs to be pressed to start the ASIC ADC, Discriminator and Correlation readout.

After carrying out the normal operations to start the system and pressing the GO button the Merger can be checked to be certain it is merging the events. Since there are only SYNC pulses it should soon be apparent if this is not happening and appropriate actions can be taken. ( Kill/Reload the Link and Merger processes using the "Merger for AIDA" icon at the top of the screen is one way. )

Once it is certain the merging is in progress then the new button labelled "Start ASIC Readout" ( see attached photo ) needs to be pressed to start the ASIC ADC, Discriminator and Correlation readout.

This must be done after every GO.

The problem indicated by "page allocation failure" and the kernal subsequently "killing" the AidaExecV8 process is due ( it appears ) to memory availability.

Solution 1:- Reduce the amount of memory used for histogramming. That would be by disabling the .V histograms or changing the size of the .H and/or .L histograms from 65536 to 32768 and using the shift attribute in the Options. This will reduce the memory required.

Solution 2:- Change a kernal setting in the /etc/sysctl.conf to request there is always a minimum amount of free memory. Insert the line "vm.min_free_kbytes=4096" in the file.

Solution 2 has been tried on the system in T9 while the flag enabling raw data to be written to disc is set. Previous to the solution, with the same operating conditions,  the fault would occur after about 15 minutes. Subsequent to the solution the system has run for 46 hours.

I propose that the .V histograms are disabled and the kernal setting is used. ( both solutions :-) The .V histograms are a late addition and not required at present.

The problem :-

The DAQ stops working and can only be fixed by a reboot of the FEE64s.

The Candidate:-

The DAQ program running in the FEE64s which handles the transfer of the data items from the FPGA to the Merger, AidaExecV8, uses a device driver, aidamem, to copy the data from DMA memory into Linux memory.

Sometimes AidaExecV8 can be killed by the Kernal due to a "page allocation failure" which occurs when the Kernal memory space has become fragmented. So when a block of contiguous memory is requested to receive the copy of the data from the FPGA DMA memory the Kernal can't allocate memory and kills the process.

The effect of this would be seen at the Merger where it would be waiting for data from the FEE.

Since the AidaExecV8 has been killed then there will be no response to status requests.

A recent change to the operation of the Aida system , flushing buffers regularly, is the most likely cuplrit. The flush of the buffers on a slow FEE will use small memory blocks while faster FEE data will always require full size buffers and mean that the Kernal memory will have large contiguous areas of memory in constant use. This will explain why this failure type is relatively recent .

A solution :- Change the request for memory copy to always use the maximum size. This way there shouldn't be the fragmentation of the memory. Also the fragmentation of memory will be investigated to see if there is a away to monitor it.

Some corroboration:-

The error messages from the Kernal that indicate this has fault has occured can be read from the FEE64 root file system. At the text file /var/log/messages. A grep of these in RIKEN using the phrase "page allocation failure" shows they have occured. It remains to see if the date and times of the failures align with the date and times of the system failure.

All comments and ideas gratefuly received.

###### # ##  # # # # #   Tested this solution on T9 system and it doesn't work ! The failure still occurs when the DAQ is set to write all input to disc.

Logged into aidas1 and opened the nnrpi1 relay control window ... powered up the FEE64s.

seperatly connected to ribfdaq and opened a firefox to aidas1:8015.

By the time the firefox had appeared on my desktop the FEE64s had started up.

Ran through the standard startup. Set thresholds to 255 for all 3 sets of comparators. Disabled all discriminators.

Enable histograms.

Started DAQ.

Using ELOG 74 as a guide logged into nnrpi1 and ran two putty sessions ( USB2 and USB3 ).

Based on the instructions in the HV unit manual attached to ELOG75 powered ion all 6 detectors ( current when settled at 100v = 3 to 5uA )

Using spectrum broswer at nnaida6 and ASIC4 to control the slow comparator threshold ..... needed to reach 32 before the pulser became visible.

Using nnaida6:1.3.L found the pulser peak at channel 32136 with a peak width of 67.73 channels.

Then poweroff the HV on all 6 detectors.

Power off the FEE64s.

Logout of all RIKEN machines and consider the best way to take advantage of the situation which allows the system to be operated completely remotely.

Attachment 2 - analysis from *end* of file AIDA75_1

2D spectrum - FEE # versus timestamp (100us/channel)

1D spectrum - projection of timestamp data for FEE # = 6

Conclude data synchronised at beginning and end of AIDA75_1. 
Period of pulser data ~36ch = 3.6ms => pulser frequency ~ 280Hz

Attachment 3 - analysis around ts=0x18bacd75c4 +/- 20000000 (dec)

Conclude FEE # 6 stops producing ADC data c. ch 353

Attachment 4 - analysis around ts=0x18cac6144b +/- 20000000 (dec)

Conclude FEE # 6 starts producing ADC data c. ch 28

Spectrum s4100 y-channel calculated as follows

C        IF ( ts.GE.z'18bacd75c4' - 20000000
C----67---
C     +       .AND.
C     +       ts.LE.z'18bacd75c4' + 20000000 ) THEN
        IF ( ts.GE.z'18cac6144b' - 20000000
C----67---
     +       .AND.
     +       ts.LE.z'18cac6144b' + 20000000 ) THEN
C
C
C       Increment ts spectra per FEE64 module

        j = MOD( INT( ts*10.0D-09*1.0D+04 ), 4096 )
        IF ( j.EQ.0 ) THEN
         DO k = 0, 4095
          DO l = 1, 31
           CALL set2d( 4100, l, k, 0 )
          ENDDO
         ENDDO
        ENDIF
        CALL inc2d( 4100, module, j )

        ENDIF

 I actually topped off the chiller with water tonight as I am returning to Scotland (and then off to CERN) within hours.

I added 4 liters on around July 5th, to move the marker from a little under the halfway mark to over the halfway mark.  After 2.x weeks, it was still sitting a bit shy of under the halfway mark.  This means that in a humid and hot Japanese month like July, AIDA doesn't evaporate more than something like 2 liters in one week. 

As it's now topped off, I think it's good to go for something like several months.

I also note that I use a faucet in the old Nishina Center, rather than passing radiation control, to find a water tap.  It's maybe more of a hike but it avoids going through radiation control.

Anyway, the point of my elog post is that the AIDA chiller will not be running out of water any time soon, probably not before October, though local members at RIKEN might check it each month etc, or on request.  I left the bucket and the large-volume container right sitting by it.  I still don't know where the funnel is, so I robbed (borrowed) a kind of 1 liter handled jug with a beaker from a place, and returned it.

11^th July ---------------------------------------------------------------------------------------------------------

09:32 UK time, logged in and power-up all FEEs OK. Setup fine, Temperature Max 12@64

System Checks all fine SYNC count 2243584.

Start Merger and Toggle Pause for no merging.

DAQ start with output enabled.

Toggle Pause in Merger -> “Want First SYNC” with no change.

Tried this several times with the same result.

Checked DAQ statistics => Buffers/sec = 50 to 65, SYNC/sec = 40 to 80  ( should be 384 )

Disabled all ASIC readout and disable all discriminators. DAQ still in GO state.

Fetched new version of ASIC4 with Enable/Disable ASIC readout commands in the “Expert” menu.

SYNC statics now all FEEs are 384/sec.

Merger … Toggle Pause and its fine. 334 items/sec.

DAQ : ASIC4 : Set all slow comparator Thresholds to 250 : Enable ASICs for readout.

Merger now 532811 items/sec.

DAQ SYNC statistics range 307 to 390/sec

LT 22:36 System Checks => Sync counts 6518784 all ok, Sync errors all passed.

LT 23:30 Merger 539826 items/sec. Temperatures => 12@68.12 ; SYNC statistics 361 to 385 ; SYNC counts 8099840 all ok ; Sync errors all ok ; ASIC clock timestamp all ok.

STOP then Power-down all fine.

13^th July ----------------------------------------------------------------------------------------------------------------------

Log in and power-up fine.

Backup Merger file ( merge.AD )  and copy latest version from DL. Won’t copy as file already open.

So instead …. Disable ASIC readout, disable Discriminators and run the system.

When Merger is operating , Enable ASIC readout => Merger rate is 8,058,412 items/sec.

LT 19:44 => Temperatures max 12@68.81 , SYNC received 845824 all ok; Sync errors all ok.

LT 20:11 => Merge rate 7,991,198 items/sec, SYNC received 1551359 all ok; Sync errors all ok.

LT 21:07 => Temperatures Max 12@69.00 ; SYNC received 2742272 all.

LT 23:20 => Stopped the whole system.

Kill the Merge process and the Startup process. Copy new Merge.AD successful. Edit Merger startup script to include netint OutputAll 1.

Start up Merger using /MIDAS/Merger/AIDA/Startup in terminal connected to aidas1.

Merger : Setup and configure ; Go ; Toggle Pause All Ok.

Go DAQ   - disable all ASIC readout.

Merger : Toggle Pause ; Merging 8832 items/sec ( = 384 x 23 …. OK )

Connect to TapeServer setup for R3 into /TapeData/July2016 , Go Tape ok.

Merger : Toggle Transfer.

LT 00:08 => enable ASICS for readout. Run for a short while. Merge spinner hesitates for 2 seconds at a time. Merge rate 5,951,471 items/sec. Tape server rate 3,5148 Kbytes/sec.

Stop system, SYNC received 8085504 all ok ; sync error all ok. Power down. Sftp tape data files to nndhcp052 in Daresbury.

14^th July -------------------------------------------------------------------------------------------------------------------------

Startup and run the system to tape again R4, ASICs with no discriminators. All fine.

LT 20:19 Enable Discriminators. Merger rate 3.9Mi/sec ( Mega items/second)

LT 20:57 Temperature Max 12@69 , Merge rate 4.2Mi/sec

UK 14:49 Network failure between Daresbury and Edinburgh. Lost contact repeatedly 4 times in a row. Contacted DL network support. No local networking faults.

LT 23:05 Re-connected to Aida in RIKEN. Merger 3.3Mi/sec.

Decided to stop due to concerns about flaky network and leaving the system powered up over the weekend.

SYNC received 7222272 all ok; sync error ok.

Power down. R4_244 is the last file. Written.

An updated copy of the merger has been downloaded ( merger64.AD ) with a copy of /MIDAS/Merger/AIDA/Startup.

These two enable the output of all of the SYNC data items to the TapeServer.

The System Wide Checks browser window has been updated and includes a report of the number SYNC pulses sent by the Master timestamp module and the number of SYNC pulses received by the Slave modules. These should all be the same value.

The ASIC4 browser window has been updated to add two actions to the Experts only menu. "Disable all ASICs For Readout"  and "Enable all ASICs for Readout". These two are useful when starting a noisy system. The sequence would be to start the Merger, toggle Pause. Then start the DAQ ( FEE64s ). "Disable all ASICs For Readout" then in the Merger toggle Pause. I found this made starting the merging much more successful. Then "Enable all ASICs for Readout" to run. I shall add similar actions to the Discriminators but will ensure the Mask Patterns set by the Restore are maintained and restored by the "Enable" action.

Before adding this capability it was quite hard to tune the system to guarantee the Merger a full set of SYNC data items at any one time. I have been running the FEE64s with low thresholds, detectors and no HV.

See https://elog.ph.ed.ac.uk/AIDA/324

Analysis of first and last data file - see attachments 1-2
Rates per FEE64 ~20k data items/s

analyser2.f program - see attachment 3

To compile the analyser2.f program (SL6)

gfortran analyser2.f -o analyser2

analyser2 <filename> to produce summary statistics
analyser2 v <filename> to produce verbose analysis

ADC multiplicity as a function of time (1 channel = 0.1ms) for file R1_114 - see attachment 4
Pulser running at ~270Hz clearly identifiable

Conclusion
DAQ ran overnight without stalling
DAQ did not lose synchronisation

Decrease comparator thresholds to increase rates to experiment-realistic levels ~50k data items/s/FEE64

Connected to aidas from daresbury.
Started Firefox with windows for Rpi, Merge, Runcontrol.
Power on FEEs from Pi.
All power-up/RESET/SETUP ok.

Change Firmware from 0x17500C15 to 0x16600C1B which adds SYNC counters to the Master and Slave Timestamp logic.
Power cycle using Pi.
nnaida20 failed to start properly. Logged in as Root ok. TclHttpd Server not in place. dmesg showed no start of
XAIDA or XAIDAMEM. Instigated "reboot" by command line not power cycle. All fine after this completed.

Copied /MIDAS/TclHttpd/Html/AIDA to a backup copy.
Changed LOCAL.*, Aida.*, Check.*

System wide checks now has check SYNC counters.

Checked temperatures. Max is nnaida12 with 68.62

SYNC check :- All read 693248.
The Master counts how many SYNC pulses are issued. All Slaves count SYNC pulses received.
The Check uses a Re-Sync pulse generated without telling the Slaves a Re-Sync is due. The Master and Slaves copy
their SYNC counter to a register when the Re-SYNC occurs. These are read back and displayed.

Started the DAQ as setup with no transfer.
Good event rates range from 502Kitems/sec to 594K.

SYNC counter check : All 839680

Stop DAQ/ Enable Transfer/Merge in Pause/Go DAQ.
All the following times are DAQ local.

19:15 Good Events 414k to 484k across all 23 FEEs.

Stop DAQ/change all thresholds to 200/Go DAQ

Good Events 4.2K to 348K

19:38 Sync counter check :- 1653760 all
Temperatures Max nnaida12 69.12

20:01 Temperatures Max nnaida12 69.19
Sync counter check :- 2181120 all

20:19 Temperatures Max nnaida12 69.25
Sync counter check :- 2590720 all

21:09 Temperatures Max nnaida12 69.19
Sync counter check :- 3729408 all

21:23 Toggled Merge Pause. Doesn't SYNC.
Check statistics. SYNC rate ok, SYNC counts uneven. PAUSE range from 0 to 1158.
DAQ stop/zero statistics/restart Merge/setup and configure/Go Merge/Pause Merge/Go DAQ/Toggle Pause
21:43 Merging now at 4943691 items/sec.

Temperatures Max nnaida12 69.25

DL to RIKEN communication link failed.

23:02 Reconnect by firefox in ribfdaq.
Temperatures Max nnaida12 69.31
Merging still active.

23:20 Sync counter check :- 6722560 all
no sysc errors in system check.

23:45 Merger running 5.4M items/sec
Temperatures max nnaida12 69.44

Stop DAQ/Toggle Pause on Merger All ok.
23:51 Sync counter check 7439360 all except nnaida22 which returns -1
Retry Sync counter check 7478272 all except nnaida22 which returns -1

Log in to nnaida22, Servers look ok. AIDAExec running ok. dmesg has nothing extra.
Runcontrol shows nnaida22 as stopped.
Access nnaida22 alone and shows undefined.

Using diagnostics checked able to read registers correctly and read the sync counter register as 7478272.

Retry Sync counter check all 8006655 except nnaida22 which shows 8006388.

Check SYNC error counters and all ok except nnaida22 which has 6363 errors.

Finishing work ..... power off FEE64s using Pi, check a couple with ping from aidas1 .... fine.

Write up ELOG.

Repeat tests on Monday.

Needed to run startx and start the TcLHttpd server on rpi

Following along here: https://elog.ph.ed.ac.uk/AIDA/242

Can now access http://nnrpi1:8015/AIDA/Rly16/

Window at first connection reads:

Using /dev/ttyUSB0 firmware version 0

Using /dev/ttyUSB1 firmware version 0901

Looks good.  Sequence ALL On was selected

(By mistake, I then clicked Relay # to turn off...misread instructions...Switch all OFF, then sequence ALL on again)

7:51 they are turned on.  Waiting several minutes now.

Add 4 L of water to the chiller

8:12

Save/Control Settings from Hardware Control tab:

Selected: 2016Jun13-00.53.55 and Restore. (missing from elog 242)

From Run Control, Enable Histogramming, Data Transfer: Enable #1 All (missing from elog 242)

TapeServer / Merger / Httpd started on Workspace 2

TapeServer reports

Cannot issue more Access Tokens: Claim TapeServer returned:- 3 Capability table is full

Make a new directory for storing test data: /data10/TapeData/July2016

Merger: Setup and configure

Data Statistics shows about ~5 x 10^5 events in many nnaida.  Should we use a different setting file?

Now we try 2016Jun12-10.10.08

Perform ReSync

Probably event rate can be high because the SSD bias is off?  We set up Putty and turn on all biases to -100 V.

Attachments 1 and 2 show the bias settings after they are turned on.

Good Events rate improves, factor 2 to 4 (now about 100k roughly)

Pulser settings: around ~900 on dial, attenuator x10, x10, x2 switches down, 5x up.  Frequency ~250 Hz

21:19

Now we change the Control ASICs (select act on all ASICs, act on all FEEs)

Slow Comparator from 0x8 to 0x20

LEC/MEC Fast Comparator 0x4 to 0x80

Now the Good Events go down to about 10k to 25k roughly

TapeServer starts a run.

Merger started / paused / unpaused

Merger rate is 500k / sec.  TapeServer is 4 k / sec.

AIDA Syncs shown in Attachment 3

Temps in Attachment 4

Good Events Attachment 5

Run Number 1

Setting file: 2016Jul05-21.37.20 (Note: I wrote the wrong setting file name last night...this is the right one!)

Show sample pulser peak in Attachment 6, 7, 8

July 6 - 12:29: Stop the run.  114 run files, 2.0 GB each.

Attachments 9, 10 show biases at the end of the run

Here are some pictures after AIDA's relocation.

In principle, it can stay here, but probably after the tests we will push it further against the wall.  However, power and ethernet cabling cannot reach that far.

Attachment 1: Shows EURICA (original position) B-RIKEN moderator (near old AIDA spot) and AIDA (pushed downstream, rotated 90 degrees counterclockwise)

Attachment 2: Zoom in on AIDA from above.  Nominally the beam left downstream perspective.  Power supply is also seen on BRIKEN moderator table.

Attachment 3: Zoom in further of above, namely near FEE power supply rack

Attachment 4: More upstream view, still on beam left, showing tube with SSDs as well as water cooler in background

Attachment 5: Beam right view

Attachment 6: Beam right view from more upstream perspective, also showing the full tower of electronics.

AIDA was relocated, and now I want to power it on to do some tests.  However, I cannot get the Rly16 service running.

Specifically, after powering up the entire AIDA system, on aidas1 PC, I cannot connect to the raspberry pi (nnrpi1) via a web browser

http://nnrpi1:8015/AIDA/Rly16/

The following elog may be relevant: https://elog.ph.ed.ac.uk/AIDA/58

Now I am ssh'd to nnrpi1 via aidas1

% dmesg | grep -A 3 USB0
[    8.970098] usb 1-1.3: pl2303 converter now attached to ttyUSB0
[    9.012479] usb 1-1.2.2: Detected FT232RL
[    9.178208] usb 1-1.2.2: FTDI USB Serial Device converter now attached to ttyUSB1
[    9.348288] ftdi_sio 1-1.2.3:1.0: FTDI USB Serial Device converter detected

So, I will try Patrick's workaround in elog #58

I tried it, though it's hard to know if it is done correctly.  Which item is which?  Or more importantly, which one runs Rly16?

[    8.107290] usb 1-1.2.3: USB disconnect, device number 8
[    8.261338] usbserial: USB Serial support registered for FTDI USB Serial Device
[    8.541012] ftdi_sio 1-1.2.2:1.0: FTDI USB Serial Device converter detected
[    8.712409] usb 1-1.2.2: Detected FT232RL
[    8.847955] usb 1-1.2.2: FTDI USB Serial Device converter now attached to ttyUSB0
[   12.252576] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   12.728966] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   23.695562] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   25.212043] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC1E1
[   32.372015] Adding 102396k swap on /var/swap.  Priority:-1 extents:2 across:2134012k SSFS
[  109.219022] usb 1-1.2.3: new full-speed USB device number 9 using dwc_otg
[  109.335538] usb 1-1.2.3: New USB device found, idVendor=0403, idProduct=6001
[  109.335579] usb 1-1.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  109.335601] usb 1-1.2.3: Product: USB <-> Serial
[  109.335619] usb 1-1.2.3: Manufacturer: FTDI
[  109.354420] ftdi_sio 1-1.2.3:1.0: FTDI USB Serial Device converter detected
[  109.354761] usb 1-1.2.3: Detected FT232BM
[  109.356174] usb 1-1.2.3: FTDI USB Serial Device converter now attached to ttyUSB1
[  163.939476] usb 1-1.3: new full-speed USB device number 10 using dwc_otg
[  164.042348] usb 1-1.3: New USB device found, idVendor=067b, idProduct=2303
[  164.042424] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  164.042446] usb 1-1.3: Product: USB-Serial Controller D
[  164.042468] usb 1-1.3: Manufacturer: Prolific Technology Inc.
[  164.096379] usbcore: registered new interface driver pl2303
[  164.098786] usbserial: USB Serial support registered for pl2303
[  164.099085] pl2303 1-1.3:1.0: pl2303 converter detected
[  164.106666] usb 1-1.3: pl2303 converter now attached to ttyUSB2

On ttyUSB# have:
FT232RL
FT232BM
pl2303


% lsusb
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 010: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 005: ID 046d:c016 Logitech, Inc. Optical Wheel Mouse
Bus 001 Device 007: ID 04f2:0402 Chicony Electronics Co., Ltd Genius LuxeMate i200 Keyboard
Bus 001 Device 006: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 009: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC

N.B. USB-serial link to FEE64 is not connected to anything on the serial side.  What is the FEE64 Console?  See Attachment 1.  It goes to nothing!  Should it go somewhere?