I have attached a document describing the new format of the Discriminator Information data item used in Version 8

When using the T9 system and running with too much rate in one FEE64 so the system is unbalanced.

I found a good way to understand this and to be able to operate the controls of the FEE64s was to place the Merger in "Pause" mode.

Then the data is extracted from the busy FEE64 but it can still be controlled.

When there is unbalance in the system it is possible for one or more queue in the Merger to become full and so the communication with that FEE64 is not possible.

Placing the Merger in Pause mode will mean all the data is discarded, the links run at full speed and action can be taken on the problem FEE64(s) to help balance the system out better.

I hope this helps .... it is highly likely that this is already being done.....

Perhaps a wise sequence when starting the whole DAQ is to operate with the Merger in Pause mode and  only switch it to merging when the user is happy with the rates.

In the statistics window of the FEE64 there is an indication of the data being transferred over the link.

When deciding how to run the experiment thought needs to be given to the various maximum rates. The rate from the FEE64 is 500k events/sec. If running with 26 FEE64 then this goes up to 13M events/sec. An event is 64 bits.

I'm not sure what the maximum rates are in the Merger and the TapeServer.

Please read and comment on the attached specification.

When starting Aida some FEE64 fail to provide SYNC data , or any data for that matter, on the ASIC data stream.
The problem is that the ASIC Readout is not starting properly and is becoming stuck waiting for an internal
event which will never arrive.
The chief source has been found to be the Correlation part of the VHDL. The reason is not yet clear but the work
around of disabling the Correlation part of all of the FEE64 at 'GO' except the Master has allowed the System to
start with all FEE64 producing SYNCs. 

This change is now incorporated in the sys.tcl file which is used when 'GO' is activated in the RunControl window.

The Master is required as it provides the 25Mhz clock and the Reset pulse. The Correlation Trigger will cause a
correlation data item to be put in the Master ASIC data stream.
Only one Correlation data item is required in the data stream, so having just the Master should be fine.

If it is required to enable another FEE64 then do this after 'GO' by opening the Correlation Browser window from
the Control window and set the Control ( at offset 0 ) to 0x0001.

Should one of the FEE64s still fail to provide SYNCs first try to 'STOP' , 'GO' the system two times and see
if that works.

A further means of correcting the problem is to directly reset the ASIC Readout using the Data Readout Window
available through the Control window. In the ASIC Readout - buffer status & control (0x300) section enter 0x4000
into the Control register to reset the state machines. Then enter 0x0001 to re-enable them. This may solve the
problem.

This is all by no means an ideal situation and the source will be found and the problem solved.

The USB controlled power Relays are near their AC fuse operating limit which has caused some Fuse failures.
To reduce the power consumption of each FEE64 it is possible to power down the Waveforms ADCs saving about 20W per FEE64. 

To Powerdown the ADCs in a FEE64 use the 'Local Data' browser window from the 'Control' browser window. Change the setting of the ADC Control 
Register at offset 2 from 0 to 0xFF. All the FEE64s can be thus controlled by using the 'Act on all' tick box.
Clearly, the System Check for ADC calibration will now fail for all those that are powered down.

To Power up the ADCs change the setting from 0xFF to 0. Then using the FADC Calibration browser window calibrate the FADCs. 

The SYSTEM should be in the 'STOP' state during these procedures.

Should the SYSTEM be power cycled then the ADCs will be powered up.

Saving at the 16A AC fuse is estimated to be 3A as there are 16 FEE64 being powered through each.

The attached firmware file, FEE_GF_Feb16_18.bin, should be saved to /MIDAS/Aida and the FlashPgm.csh file edited
to load "FEE_GF_Feb16_18.bin"

Use the Run Control Expert command to update the firmware on all the FEE64s.

Power cycle the FEE64s to load the new firmware.

Make a backup copy of the files in the directory /MIDAS/TclHttpd/Html/AIDA/DISC.

Unzip the files from DISC_25May16.zip and copy them into /MIDAS/TclHttpd/Html/AIDA/DISC to replace the existing
ones.

Make a backup copy of the file /MIDAS/TclHttpd/Html/AIDA/LOCAL.tml

Unzip the file from LOCAL_tml_25May16.zip and copy into /MIDAS/TclHttpd/Html/AIDA/LOCAL to replace the existing
file.

'RESET' and browser refresh the two updated control windows to load the new software.
-----------------------------------------------------

To use the Multiplicity Trigger:-
In the Local Controls select option 5 for the Trigger.

In the Discriminator window.

Set the 'Upper Limit'
Set the 'Lower Limit' ( always greater than 0 ..... or the Trigger will always be present )
Set the 'Time window'. The number of 10nS clocks. So 4 => 40nS. Register is 0 to 255.

-----------------------------------------------------

These registers are not in the Save/Restore set yet.

Tested on the T9 system using the pulser and setting/clearing Mask bits.

Please let me know how this works for you.

Done. Awaiting test. See attachments 1-2.

> The attached firmware file, FEE_GF_Feb16_18.bin, should be saved to /MIDAS/Aida and the FlashPgm.csh file edited
> to load "FEE_GF_Feb16_18.bin"
> 
> Use the Run Control Expert command to update the firmware on all the FEE64s.
> 
> Power cycle the FEE64s to load the new firmware.
> 
> Make a backup copy of the files in the directory /MIDAS/TclHttpd/Html/AIDA/DISC.
> 
> Unzip the files from DISC_25May16.zip and copy them into /MIDAS/TclHttpd/Html/AIDA/DISC to replace the existing
> ones.
> 
> Make a backup copy of the file /MIDAS/TclHttpd/Html/AIDA/LOCAL.tml
> 
> Unzip the file from LOCAL_tml_25May16.zip and copy into /MIDAS/TclHttpd/Html/AIDA/LOCAL to replace the existing
> file.
> 
> 'RESET' and browser refresh the two updated control windows to load the new software.
> -----------------------------------------------------
> 
> To use the Multiplicity Trigger:-
> In the Local Controls select option 5 for the Trigger.
> 
> In the Discriminator window.
> 
> Set the 'Upper Limit'
> Set the 'Lower Limit' ( always greater than 0 ..... or the Trigger will always be present )
> Set the 'Time window'. The number of 10nS clocks. So 4 => 40nS. Register is 0 to 255.
> 
> -----------------------------------------------------
> 
> These registers are not in the Save/Restore set yet.
> 
> Tested on the T9 system using the pulser and setting/clearing Mask bits.
> 
> Please let me know how this works for you.

Attached is the specification for the MACB with the settings currently available.

The latest version of the FEE64 software Interface document ( May 2016 ) is available as a .pdf from
http://npg.dl.ac.uk/documents/edoc000/#AIDA. EDOC955

The version available via the pink "documentation" button in the Control browser window is now out of date. This
will be updated at some future time.

Other documents that are of use are there also.
MACB specification.
GREAT Data Format.

If there are more than one failures then check how they map to the MACB HDMI connections.
 Are they all connected to the just one MACB ? => Check the HDMI cable at the MACB Layer Port.
 Are they all connected to several MACBs which have a common "Parent" ? => Check the HDMI cable at the "Parent"
MACB Layer Port.

If there is just one failure then open the Local Controls browser window for the failing FEE64.
If the "Status Register" @1 is not 0x7 then the Waveform ADCs will not operate correctly and ASIC readout will
be compromised.
To remedy try 
A)    Enter 0x2B to "LMK03200 control register" @5 
          then Reload the Page 
          then Enter 0xB
          then Reload the Page and check the status, if 0x7 all ok. 

B)     Enter 0xA to "LMK03200 control register" @5 
          then Reload the Page
          then Enter 0xB
          then Reload the Page
          then enter 0x2B to "LMK03200 control register" @5 
          then Reload the Page 
          then Enter 0xB
          then Reload the Page and check the status, if 0x7 all ok.  

Connected to aidas from daresbury.
Started Firefox with windows for Rpi, Merge, Runcontrol.
Power on FEEs from Pi.
All power-up/RESET/SETUP ok.

Change Firmware from 0x17500C15 to 0x16600C1B which adds SYNC counters to the Master and Slave Timestamp logic.
Power cycle using Pi.
nnaida20 failed to start properly. Logged in as Root ok. TclHttpd Server not in place. dmesg showed no start of
XAIDA or XAIDAMEM. Instigated "reboot" by command line not power cycle. All fine after this completed.

Copied /MIDAS/TclHttpd/Html/AIDA to a backup copy.
Changed LOCAL.*, Aida.*, Check.*

System wide checks now has check SYNC counters.

Checked temperatures. Max is nnaida12 with 68.62

SYNC check :- All read 693248.
The Master counts how many SYNC pulses are issued. All Slaves count SYNC pulses received.
The Check uses a Re-Sync pulse generated without telling the Slaves a Re-Sync is due. The Master and Slaves copy
their SYNC counter to a register when the Re-SYNC occurs. These are read back and displayed.

Started the DAQ as setup with no transfer.
Good event rates range from 502Kitems/sec to 594K.

SYNC counter check : All 839680

Stop DAQ/ Enable Transfer/Merge in Pause/Go DAQ.
All the following times are DAQ local.

19:15 Good Events 414k to 484k across all 23 FEEs.

Stop DAQ/change all thresholds to 200/Go DAQ

Good Events 4.2K to 348K

19:38 Sync counter check :- 1653760 all
Temperatures Max nnaida12 69.12

20:01 Temperatures Max nnaida12 69.19
Sync counter check :- 2181120 all

20:19 Temperatures Max nnaida12 69.25
Sync counter check :- 2590720 all

21:09 Temperatures Max nnaida12 69.19
Sync counter check :- 3729408 all

21:23 Toggled Merge Pause. Doesn't SYNC.
Check statistics. SYNC rate ok, SYNC counts uneven. PAUSE range from 0 to 1158.
DAQ stop/zero statistics/restart Merge/setup and configure/Go Merge/Pause Merge/Go DAQ/Toggle Pause
21:43 Merging now at 4943691 items/sec.

Temperatures Max nnaida12 69.25

DL to RIKEN communication link failed.

23:02 Reconnect by firefox in ribfdaq.
Temperatures Max nnaida12 69.31
Merging still active.

23:20 Sync counter check :- 6722560 all
no sysc errors in system check.

23:45 Merger running 5.4M items/sec
Temperatures max nnaida12 69.44

Stop DAQ/Toggle Pause on Merger All ok.
23:51 Sync counter check 7439360 all except nnaida22 which returns -1
Retry Sync counter check 7478272 all except nnaida22 which returns -1

Log in to nnaida22, Servers look ok. AIDAExec running ok. dmesg has nothing extra.
Runcontrol shows nnaida22 as stopped.
Access nnaida22 alone and shows undefined.

Using diagnostics checked able to read registers correctly and read the sync counter register as 7478272.

Retry Sync counter check all 8006655 except nnaida22 which shows 8006388.

Check SYNC error counters and all ok except nnaida22 which has 6363 errors.

Finishing work ..... power off FEE64s using Pi, check a couple with ping from aidas1 .... fine.

Write up ELOG.

Repeat tests on Monday.

An updated copy of the merger has been downloaded ( merger64.AD ) with a copy of /MIDAS/Merger/AIDA/Startup.

These two enable the output of all of the SYNC data items to the TapeServer.

The System Wide Checks browser window has been updated and includes a report of the number SYNC pulses sent by the Master timestamp module and the number of SYNC pulses received by the Slave modules. These should all be the same value.

The ASIC4 browser window has been updated to add two actions to the Experts only menu. "Disable all ASICs For Readout"  and "Enable all ASICs for Readout". These two are useful when starting a noisy system. The sequence would be to start the Merger, toggle Pause. Then start the DAQ ( FEE64s ). "Disable all ASICs For Readout" then in the Merger toggle Pause. I found this made starting the merging much more successful. Then "Enable all ASICs for Readout" to run. I shall add similar actions to the Discriminators but will ensure the Mask Patterns set by the Restore are maintained and restored by the "Enable" action.

Before adding this capability it was quite hard to tune the system to guarantee the Merger a full set of SYNC data items at any one time. I have been running the FEE64s with low thresholds, detectors and no HV.

11^th July ---------------------------------------------------------------------------------------------------------

09:32 UK time, logged in and power-up all FEEs OK. Setup fine, Temperature Max 12@64

System Checks all fine SYNC count 2243584.

Start Merger and Toggle Pause for no merging.

DAQ start with output enabled.

Toggle Pause in Merger -> “Want First SYNC” with no change.

Tried this several times with the same result.

Checked DAQ statistics => Buffers/sec = 50 to 65, SYNC/sec = 40 to 80  ( should be 384 )

Disabled all ASIC readout and disable all discriminators. DAQ still in GO state.

Fetched new version of ASIC4 with Enable/Disable ASIC readout commands in the “Expert” menu.

SYNC statics now all FEEs are 384/sec.

Merger … Toggle Pause and its fine. 334 items/sec.

DAQ : ASIC4 : Set all slow comparator Thresholds to 250 : Enable ASICs for readout.

Merger now 532811 items/sec.

DAQ SYNC statistics range 307 to 390/sec

LT 22:36 System Checks => Sync counts 6518784 all ok, Sync errors all passed.

LT 23:30 Merger 539826 items/sec. Temperatures => 12@68.12 ; SYNC statistics 361 to 385 ; SYNC counts 8099840 all ok ; Sync errors all ok ; ASIC clock timestamp all ok.

STOP then Power-down all fine.

13^th July ----------------------------------------------------------------------------------------------------------------------

Log in and power-up fine.

Backup Merger file ( merge.AD )  and copy latest version from DL. Won’t copy as file already open.

So instead …. Disable ASIC readout, disable Discriminators and run the system.

When Merger is operating , Enable ASIC readout => Merger rate is 8,058,412 items/sec.

LT 19:44 => Temperatures max 12@68.81 , SYNC received 845824 all ok; Sync errors all ok.

LT 20:11 => Merge rate 7,991,198 items/sec, SYNC received 1551359 all ok; Sync errors all ok.

LT 21:07 => Temperatures Max 12@69.00 ; SYNC received 2742272 all.

LT 23:20 => Stopped the whole system.

Kill the Merge process and the Startup process. Copy new Merge.AD successful. Edit Merger startup script to include netint OutputAll 1.

Start up Merger using /MIDAS/Merger/AIDA/Startup in terminal connected to aidas1.

Merger : Setup and configure ; Go ; Toggle Pause All Ok.

Go DAQ   - disable all ASIC readout.

Merger : Toggle Pause ; Merging 8832 items/sec ( = 384 x 23 …. OK )

Connect to TapeServer setup for R3 into /TapeData/July2016 , Go Tape ok.

Merger : Toggle Transfer.

LT 00:08 => enable ASICS for readout. Run for a short while. Merge spinner hesitates for 2 seconds at a time. Merge rate 5,951,471 items/sec. Tape server rate 3,5148 Kbytes/sec.

Stop system, SYNC received 8085504 all ok ; sync error all ok. Power down. Sftp tape data files to nndhcp052 in Daresbury.

14^th July -------------------------------------------------------------------------------------------------------------------------

Startup and run the system to tape again R4, ASICs with no discriminators. All fine.

LT 20:19 Enable Discriminators. Merger rate 3.9Mi/sec ( Mega items/second)

LT 20:57 Temperature Max 12@69 , Merge rate 4.2Mi/sec

UK 14:49 Network failure between Daresbury and Edinburgh. Lost contact repeatedly 4 times in a row. Contacted DL network support. No local networking faults.

LT 23:05 Re-connected to Aida in RIKEN. Merger 3.3Mi/sec.

Decided to stop due to concerns about flaky network and leaving the system powered up over the weekend.

SYNC received 7222272 all ok; sync error ok.

Power down. R4_244 is the last file. Written.

Logged into aidas1 and opened the nnrpi1 relay control window ... powered up the FEE64s.

seperatly connected to ribfdaq and opened a firefox to aidas1:8015.

By the time the firefox had appeared on my desktop the FEE64s had started up.

Ran through the standard startup. Set thresholds to 255 for all 3 sets of comparators. Disabled all discriminators.

Enable histograms.

Started DAQ.

Using ELOG 74 as a guide logged into nnrpi1 and ran two putty sessions ( USB2 and USB3 ).

Based on the instructions in the HV unit manual attached to ELOG75 powered ion all 6 detectors ( current when settled at 100v = 3 to 5uA )

Using spectrum broswer at nnaida6 and ASIC4 to control the slow comparator threshold ..... needed to reach 32 before the pulser became visible.

Using nnaida6:1.3.L found the pulser peak at channel 32136 with a peak width of 67.73 channels.

Then poweroff the HV on all 6 detectors.

Power off the FEE64s.

Logout of all RIKEN machines and consider the best way to take advantage of the situation which allows the system to be operated completely remotely.

The problem :-

The DAQ stops working and can only be fixed by a reboot of the FEE64s.

The Candidate:-

The DAQ program running in the FEE64s which handles the transfer of the data items from the FPGA to the Merger, AidaExecV8, uses a device driver, aidamem, to copy the data from DMA memory into Linux memory.

Sometimes AidaExecV8 can be killed by the Kernal due to a "page allocation failure" which occurs when the Kernal memory space has become fragmented. So when a block of contiguous memory is requested to receive the copy of the data from the FPGA DMA memory the Kernal can't allocate memory and kills the process.

The effect of this would be seen at the Merger where it would be waiting for data from the FEE.

Since the AidaExecV8 has been killed then there will be no response to status requests.

A recent change to the operation of the Aida system , flushing buffers regularly, is the most likely cuplrit. The flush of the buffers on a slow FEE will use small memory blocks while faster FEE data will always require full size buffers and mean that the Kernal memory will have large contiguous areas of memory in constant use. This will explain why this failure type is relatively recent .

A solution :- Change the request for memory copy to always use the maximum size. This way there shouldn't be the fragmentation of the memory. Also the fragmentation of memory will be investigated to see if there is a away to monitor it.

Some corroboration:-

The error messages from the Kernal that indicate this has fault has occured can be read from the FEE64 root file system. At the text file /var/log/messages. A grep of these in RIKEN using the phrase "page allocation failure" shows they have occured. It remains to see if the date and times of the failures align with the date and times of the system failure.

All comments and ideas gratefuly received.

###### # ##  # # # # #   Tested this solution on T9 system and it doesn't work ! The failure still occurs when the DAQ is set to write all input to disc.

The problem indicated by "page allocation failure" and the kernal subsequently "killing" the AidaExecV8 process is due ( it appears ) to memory availability.

Solution 1:- Reduce the amount of memory used for histogramming. That would be by disabling the .V histograms or changing the size of the .H and/or .L histograms from 65536 to 32768 and using the shift attribute in the Options. This will reduce the memory required.

Solution 2:- Change a kernal setting in the /etc/sysctl.conf to request there is always a minimum amount of free memory. Insert the line "vm.min_free_kbytes=4096" in the file.

Solution 2 has been tried on the system in T9 while the flag enabling raw data to be written to disc is set. Previous to the solution, with the same operating conditions,  the fault would occur after about 15 minutes. Subsequent to the solution the system has run for 46 hours.

I propose that the .V histograms are disabled and the kernal setting is used. ( both solutions :-) The .V histograms are a late addition and not required at present.

Due to sometimes operating with very noisy systems, both at Daresbury and RIKEN, I have installed a change to the startup of the system.

The Runcontrol GO button now only starts the output of SYNC pulses. A further button labelled "Start ASIC Readout" ( see attached photo ) needs to be pressed to start the ASIC ADC, Discriminator and Correlation readout.

After carrying out the normal operations to start the system and pressing the GO button the Merger can be checked to be certain it is merging the events. Since there are only SYNC pulses it should soon be apparent if this is not happening and appropriate actions can be taken. ( Kill/Reload the Link and Merger processes using the "Merger for AIDA" icon at the top of the screen is one way. )

Once it is certain the merging is in progress then the new button labelled "Start ASIC Readout" ( see attached photo ) needs to be pressed to start the ASIC ADC, Discriminator and Correlation readout.

This must be done after every GO.

Logged into aidas1, installed AidaExecV8 with a test of the time difference between SYNC data items. Failure to be 0x40000 clocks will cause a statistic to be incremented. "SYNC Time Warp"

Installed updated versions of /MIDAS/TclHttpd/Html/AIDA/RunControl/sys.tcl and /MIDAS/TclHttpd/Html/AIDA/RunControl/implementation.tcl to allow latest System Wide Check to operate.

Installed updated versions of Check.tcl/.js/tml into /MIDAS/TclHttpd/Html/AIDA/Check/

Checked all the /MIDAS/linux-ppc_4xx/startup files including aidacommon for entries relating to the flush and push rates. Only aidacommon has an entry as required. Currently both are set to 30.

Changed the /etc/sysctl.conf file in each of the 32 root file systems (rfs) in /MIDAS/XilinxLinux/ppc_4xx/rfs/nnaida##/ to add the line "vm.min_free_kbytes=4096".

The changes will take effect from the next power-cycle of the FEEs and "reset" of the browser windows on the server.

Used the layout reported in ELOG357 and updated the layout.txt file used in the "Layout and Status" browser window.

The file is in /MIDAS/config/TclHttpd/aidas1

The browser window "Layout and Status" is accessed from the Control browser window.

It is intended to be used to determine if the Embedded Linux is operating in the FEE64. Each FEE64 in the layout.txt file is pinged with a timeout of 1 second. The layout then shows red for no response and green for a response.