Loaded new frimware into all 12 FEE64s in the system. IWR_Dec19_4.bin 

Booted ok after power-cycle.  

RESET/SETUP/GO all fine.

system wide checks ( SWC ) clocks, white rabbit ( WR ) OK, ADC calibrated : aida10 failed. Couldn't get them to calibrate.

Checked aida01 waveforms for ASIC1. set threshold for LED to 5000 to stop triggering in the noise.

Set White Rabbit register SYNC Rollover Target ( 14 ) to 0x4000 instead of 0xe to reduce the data rate to better understand the waveform activity.

All FEE64s are set to enable all channels of ASIC1 with low thresholds so a lot of data moving. Checked activity using statistics window.

Checked waveforms working for ASIC1 of aida01. Noisy but capturing the pulser pulse.

 Enabled all the waveforms on aida01 to see if they all function ok. After a few seconds waveform readout stopped. Not sure of the reason.

STOP, change all four thresholds to 5000, Check ASIC controls : all ok, GO. No activity.

RESET/SETUP ok.

SWC : clock failed, ADC failed , WR ok ????

Power-cycle...... concludes ok, RESET/SETUP ok, SWC : clocks all failed, ADC all failed, WR ok.

Checked WR status browser page for aida01. WR timestamp 0 ! status : 0xA should be 0 .

Checked tcl for SWC WR check and it doesn't check the status. Changed at DL and at GSI and now it does.

so SWC WR check no all fails.

Contacted NH and he found Fatima VME crate with WR source in had been switched off. It was powered on again.

RESET/SETUP , SWC: clocks all ok, ADC all ok, WR all ok.

set aida01 and all odd number  LED thresholds to 5000 and aida02 all even number thresholds to 10000 and activity of statistic Wave Good Events is similar across the system. 

Checked each of the Waveform capture controls pages ( only shows the status registers for ASIC1 ) and all are fine, nothing sticking.

enabled all the ASIC waveforms on aida01 again and it sticks, clear out three ASICs and operate the waveform reset in the System Functions and it restarts and runs.

So a qualified success. Don't enable too many channels at once ?

left running at 18:00 UK time and checked at 21:30 UK time , still operating ok.