AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 4 of 36  ELOG logo
Entry  Fri Mar 19 17:16:45 2021, TD, Friday 19 March 8x
Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04.
Entry  Wed Mar 24 10:31:03 2021, TD, AIDA to do (or to understand) list 
In no particular order

1) Options

variables unexpectedly changing or becoming unset/undefined - see https://elog.ph.ed.ac.uk/DESPEC/205

Is this why *.L, *.H and Rate spectra are undefined in some FEE64s?


2) Merger 'Bad Timestamp' error reports - e.g. see https://elog.ph.ed.ac.uk/DESPEC/203

Meaning of report? Significance/impact?


3) Large numbers of PAUSE/RESUME for some (but not all) FEE64s - see e.g. https://elog.ph.ed.ac.uk/DESPEC/203

Significance/impact?

-- NH to add

4) Slow but non-zero increase of White Rabbit errors (mostly timestamp but much rarer ASIC errors)
We know the White Rabbit network in general is reliable but could we see interruption to the 200 MHz clock or 10 kHz timestamp when FATIMA DAQ is busy/restarting
Wasn't uniform over all FEEs implying a more local source though?

-- OH to add
5) The dropping of links between FEEs and the MERGER. Quite often these were correlated with the 'Bad Timestamp' error messages and also the WR errors

---- PJCS to add
6) The addition of the NIM clock-divider to allow better ASIC clock synchronisation.
Entry  Thu Mar 25 18:04:51 2021, TD, MSL type BB18(DS) 24cm x 8cm cabling & bias configuration MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf
 
Entry  Mon Apr 12 16:49:40 2021, TD, Monday 22 April - pulser walkthrough & alpha background 6x
16.54 Pulser walkthrough

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      file NULL/R4

      BNC PB-5 settings changed by terminal command line using the commands

      set rep rate 25
      set attenuation 5
      set amplitude 10
      set amplitude 9
      :
      :
      set amplitude 1
      set attenuation 1
      set rep rate 2

      aida07 & aida08 1.8.L spectra - see attachment 1

17.26 run ends OK file NULL/R4_27

17.26 file NULL/R5

      amplitudes 0.6, 1, 1.4, 1.8, 2.2, 2.6, 3

      aida07 & aida08 1.8.L spectra - see attachment 2

17.47 run ends OK NULL/R5_23

17.53 system wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     11      3      2      0      2      4      2      3      4     11   : 58888
aida02 :      7     11      4      2      0      4      3      2      3      3      7   : 40436
aida03 :     38      7      7      2      6      4      1      3      3      3      6   : 36864
aida04 :     19     10      5      2      1      3      1      4      2      4     15   : 74732
aida05 :     30      5      4      3      2      3      4      3      3      4     15   : 76096
aida06 :     24      4      3      3      1      4      2      3      2      4      7   : 41808
aida07 :     22      7     11      7      1      6      3      4      3      3      6   : 37984
aida08 :     13      7      5      2      1      4      3      3      1      4      6   : 36924
aida09 :     33      9     10      6      4      4      2      2      2      4      6   : 37676
aida10 :     15      7      7      1      1      3      1      3      2      3      7   : 39364
aida11 :     30     13      8      3      4      4      3      3      3      3      6   : 37312
aida12 :     17      9      4      3      2      3      2      2      3      3      7   : 40236

17.57 Slow comparator 0xa -> 0x64
      BNC PB-5 Pulser OFF

      file NULL/R9
      alpha background

      FEE64 temperatures OK - see attachment 3
      statistics OK - see attachment 4
      DSSSD bias & leakage currents OK - see attachment 5
      tape server & merger - attachment 6

11:31 13.04.21 
      file NULL/R9 stopped for work in day
      will resume alpha over night again

 
Entry  Tue Apr 13 14:24:48 2021, TD, S460 information 
aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S460 shift plan - https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1097596683

Mattermost - https://mattermost.gsi.de/despec/channels/aida

DESPEC Zoom meeting

Zoom Meeting https://gsi-fair.zoom.us/j/94404952361
Passcode: s460
Entry  Tue Apr 13 16:49:41 2021, TD, Tuesday 13 April 8x
Rename Options directory

mv Options Options.BAK-130421

Copy Options.PRISTINE to Options

cp -r Options.PRISTINE Options
chmod -R 777 Options

Start Options directory monitor (will check directory every 5 minutes)

Python ~/oh/OptionsCheck


17.32 Power cycle FEE64s


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      5      2      2      1      3      3      2      2      3      8   : 43348
aida02 :      2      3      0      2      1      2      2      3      2      3      8   : 43424
aida03 :      3      4      0      1      1      2      2      3      2      3      8   : 43404
aida04 :      1      4      2      0      3      3      3      2      2      3      8   : 43396
aida05 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
aida06 :      2      3      2      1      1      2      2      3      2      3      8   : 43424
aida07 :      4      1      3      3      1      3      1      3      2      3      8   : 43368
aida08 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
aida09 :      2      3      3      3      2      4      2      2      2      3      8   : 43312
aida10 :      1      3      1      2      2      3      2      2      2      3      8   : 43116
aida11 :      1      3      3      1      2      3      2      2      2      3      8   : 43116
aida12 :      3      2      1      0      1      2      1      3      2      3      8   : 43116


Note that system wide checks identify 1x file of different size immediately after boot/reset/setup

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:05 CEST 2021

06:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

06:30:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable





FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0


 If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status




 aida01 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida02 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida03 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida04 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida05 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida06 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida07 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida08 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida09 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida10 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida11 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida12 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

17.14 rate spectra - attachment 1

      1.8.L spectra - attachments 2 & 3
      pulser peak width aida06 c.75 ch FWHM, aida07  c.85 ch FWHM

      1.8.W spectra - attachments 4 & 5

      good events statistics (slow comparator 0xa) - attachment 6

      FEE64 temperatures OK - attachment 7
    
      DSSSD bias & leakage currents OK - attachment 8




17.38 DAQ starts file NULL/R14
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      alpha background

10:40 CET 14.04.2021 - NH
      Close file NULL/R14
    Reply  Wed Apr 14 09:55:12 2021, TD, Tuesday 13 April 
Two further comments

1) The 'pristine' Options files produced by OH fixed *all* of the undefined spectra issues,
   i.e. we now have *L, *H, Rate, Stat, Hit, HitRate spectra for all FEE64s

2) The system has undegone 1x power cycle/reset/setup and 1x reset/setup cycle since the
   new Options files were installed and we currently have 2x errors reported in aida02.
   There is an open Options tab and we have viewed the Options settings for aida01 and aida02
   (currently viewing aida02) - *no* changes/saves/restores requested.

> 
> Rename Options directory
> 
> mv Options Options.BAK-130421
> 
> Copy Options.PRISTINE to Options
> 
> cp -r Options.PRISTINE Options
> chmod -R 777 Options
> 
> Start Options directory monitor (will check directory every 5 minutes)
> 
> Python ~/oh/OptionsCheck
> 
> 
> 17.32 Power cycle FEE64s
> 
> 
> FEE64 module aida09 global clocks failed, 6
>  Clock status test result: Passed 11, Failed 1
> 
> Understand status as follows
> Status bit 3 : firmware PLL that creates clocks from external clock not locked
> Status bit 2 : always logic '1' 
> Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
> Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
> If all these bits are not set then the operation of the firmware is unreliable
> 
> Calibration test result: Passed 12, Failed 0
> 
> If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
> 
> White Rabbit error counter test result: Passed 12, Failed 0
> 
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
> 
> 	
> FPGA Timestamp error counter test result: Passed 12, Failed 0
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
> 
> 
> 
> Returned 0 0 0 0 0 0 0 0 0 0 0 0  
> Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
> aida01 :      3      5      2      2      1      3      3      2      2      3      8   : 43348
> aida02 :      2      3      0      2      1      2      2      3      2      3      8   : 43424
> aida03 :      3      4      0      1      1      2      2      3      2      3      8   : 43404
> aida04 :      1      4      2      0      3      3      3      2      2      3      8   : 43396
> aida05 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
> aida06 :      2      3      2      1      1      2      2      3      2      3      8   : 43424
> aida07 :      4      1      3      3      1      3      1      3      2      3      8   : 43368
> aida08 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
> aida09 :      2      3      3      3      2      4      2      2      2      3      8   : 43312
> aida10 :      1      3      1      2      2      3      2      2      2      3      8   : 43116
> aida11 :      1      3      3      1      2      3      2      2      2      3      8   : 43116
> aida12 :      3      2      1      0      1      2      1      3      2      3      8   : 43116
> 
> 
> Note that system wide checks identify 1x file of different size immediately after boot/reset/setup
> 
>  Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
> 
>  FEE : aida01 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida02 =>   Options file size is 1025 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida03 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida04 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida05 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida06 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida07 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida08 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida09 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida10 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida11 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida12 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:05 CEST 2021
> 
> 06:25:00 PM
> FAULTS!
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV
E_DMA_HWM&&Ai
> da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
> 
> 06:30:00 PM
> FAULTS!
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV
E_DMA_HWM&&Ai
> da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.setting
s&&WAVE_DMA_H
> 
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_En
able
> 
> 
> 
> 
> 
> FEE64 module aida01  => 	7
> FEE64 module aida02  => 	7
> FEE64 module aida03  => 	7
> FEE64 module aida04  => 	7
> FEE64 module aida05  => 	7
> FEE64 module aida06  => 	7
> FEE64 module aida07  => 	7
> FEE64 module aida08  => 	7
> FEE64 module aida09  => 	7
> FEE64 module aida10  => 	7
> FEE64 module aida11  => 	7
> FEE64 module aida12  => 	7
> 
> 
> ASIC Clock Timestamp check test result: Passed 12, Failed 0
> 
> 
>  If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status
> 
> 
> 
> 
>  aida01 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida02 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida03 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida04 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida05 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida06 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida07 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida08 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida09 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida10 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida11 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida12 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
> 17.14 rate spectra - attachment 1
> 
>       1.8.L spectra - attachments 2 & 3
>       pulser peak width aida06 c.75 ch FWHM, aida07  c.85 ch FWHM
> 
>       1.8.W spectra - attachments 4 & 5
> 
>       good events statistics (slow comparator 0xa) - attachment 6
> 
>       FEE64 temperatures OK - attachment 7
>     
>       DSSSD bias & leakage currents OK - attachment 8
> 
> 
> 
> 
> 17.38 DAQ starts file NULL/R14
>       slow comparator 0xa -> 0x64
>       BNC PB-5 pulser OFF
>       alpha background
> 
> 10:40 CET 14.04.2021 - NH
>       Close file NULL/R14
Entry  Thu Apr 15 13:28:07 2021, TD, Thursday 15 April 19x
13.28 System wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida01 fault 	 0x44c2 : 	 0x44c5 : 	 3  
aida02 fault 	 0xc81f : 	 0xc822 : 	 3  
aida03 fault 	 0x9342 : 	 0x9344 : 	 2  
aida04 fault 	 0xdb3f : 	 0xdb41 : 	 2  
aida05 fault 	 0x4baf : 	 0x4bb3 : 	 4  
aida06 fault 	 0xd2a4 : 	 0xd2a8 : 	 4  
aida07 fault 	 0xc51c : 	 0xc51f : 	 3  
aida08 fault 	 0xe853 : 	 0xe856 : 	 3  
aida09 fault 	 0x5212 : 	 0x5216 : 	 4  
aida10 fault 	 0xf81d : 	 0xf81f : 	 2  
aida11 fault 	 0x810 : 	 0x812 : 	 2  
aida12 fault 	 0xa279 : 	 0xa27a : 	 1  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x5 : 	 5  
aida12 fault 	 0x0 : 	 0xc : 	 12  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     57     33     12     15      3      3      3      2      3      2      6   : 35276
aida02 :     15      3      5      2      1      4      1      3      3      3      6   : 36388
aida03 :     13      4      2      1      4      3      1      3      3      3      6   : 36372
aida04 :     21      5      2      1      2      4      4      2      3      3      6   : 36668
aida05 :     10      5      6      3      2      3      3      3      1      4      6   : 36880
aida06 :     22     12      1      0      3      3      1      3      3      3      6   : 36360
aida07 :     19      3      3      0      3      3      2      4      1      4      6   : 37076
aida08 :     21      6      2      4      2      3      1      3      3      3      6   : 36388
aida09 :      2      7      5      3      2      4      2      3      2      3      6   : 35696
aida10 :      7      3      1      3      1      3      3      3      2      3      6   : 35684
aida11 :      1      2      1      3      2      5      2      2      3      3      6   : 36228
aida12 :     17     10      4      2      3      2      2      4      2      3      6   : 36052



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida02 =>   Options file size is 2114 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

13.30 Output of ~/oh/OptionsCheck.py

02:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&ASIC.se
ttings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

13.37 Detector bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachments 2
      Statistics - good events, ADC data, disc data, info code 4 & 5, pause & resume, correlation - attachments 3-10
      per FEE64 rate spectra - attachments 11 & 12
      per FEE64 1.8.L spectra - attachments 13 & 19
       aida5 peak width 70 ch FWHM, aida06 peak width 89ch FWHM
      per FEE64 1.8.W spectra 20us FSR - attachments 14 & 15
      meger & tape server - attachments 16-18
 
Entry  Sun Apr 18 11:18:21 2021, TD, Sunday 18 April 12.00-16.00 25x
12.18 DAQ continues file NULL/R30_490

      ASIC settings 2019Dec19-16.19.51
      slow comparator 0x64 -> 0xa

      BNC PB-5
      amplitude 2V
      attenuation 1x
      decay time 1ms
      frequency 2Hz

12.22 all histograms zero'd

c. 12.30 219Rn setting

12.35 analysis of R30_490 - see attachment 1

13.15 system wide checks

Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida01 fault 	 0x7686 : 	 0x768a : 	 4  
aida02 fault 	 0x941d : 	 0x9421 : 	 4  
aida03 fault 	 0x7cd7 : 	 0x7cdb : 	 4  
aida04 fault 	 0xb86d : 	 0xb871 : 	 4  
aida05 fault 	 0x1a59 : 	 0x1a5f : 	 6  
aida06 fault 	 0x4f45 : 	 0x4f4b : 	 6  
aida07 fault 	 0x3bfc : 	 0x3c04 : 	 8  
aida08 fault 	 0xc7ce : 	 0xc7d4 : 	 6  
aida09 fault 	 0xb33b : 	 0xb33c : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0xa : 	 0xb : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :      2     11      7      4      2      2      2      4      2      3      6   : 36048
aida03 :     21      8      6      4      5      4      3      3      2      3      6   : 36276
aida04 :     19     11     14      3      2      3      2      4      2      3      6   : 36324
aida05 :     25      4      5      4      4      2      1      3      2      4      6   : 37460
aida06 :     13      9     15      1      3      5      1      3      3      3      6   : 36812
aida07 :     19     10      6      1      3      3      2      2      4      3      6   : 37212
aida08 :     23      6      4      2      1      1      3      4      2      3      6   : 36044
aida09 :     11      5      5      1      2      4      3      3      2      3      6   : 35908
aida10 :     14      5      6      5      3      1      3      3      2      3      6   : 35744
aida11 :     16      5      3      0      3      3      1      4      2      3      6   : 35800
aida12 :     12     10      0      2      4      4      1      3      3      3      6   : 36544

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

*no faults detected by ~/oh/OptionsCheck.py

13.10 Detector biases & leakage currents OK - see attachment 2
      Statistics - good events, adc data, disc data, info code 4 & 5, pause, resume, correlation info - attachments 3-12
      per FEE64 rate spectra - attachments 13-14
      1.8.L spectra - attachments 15-18

       pulser peak widths
       1 80
       2 113
       3 308
       4 105
       5 50
       6 91
       7 61
       8 90
       9 200
       10 133
       11 187
       12 107
  
      1.8.H spectra - attachments 19-20
      1.8.W spectra - attachments 21-22
      ucesb - attachment 23
       implant rates DSSSD#1-3 to c. 1kHz
      grafana - AIDA DSSSD leakage currents for previous 2 days - attachment 24
      merger & merger/tape server/mbs data tranfser - attachment 25
       no recent merger warning/error messages

13.30 225At setting
Entry  Thu Apr 22 15:18:20 2021, TD, Thursday 22 April 16.00-00.00 36x
16.20 zero all histograms, statistics & merger statistics

      most recent merger error messages

MERGE Data Link (29984): bad timestamp  6 3 0xc1ac7f1c 0x0dce7672 0x00002cda5dce7672 0x16782cda5dce7672 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc18b7e62 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ebc 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1b67ecb 0x0dcf0ae2 0x00002cda5dcf0ae2 0x16782cda5dcf0ae2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ea3 0x0dcf12b2 0x00002cda5dcf12b2 0x16782cda5dcf12b2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1867fba 0x0b2ee142 0x00002dc9eb2ee142 0x16782dc9eb2ee142 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0x86800889 0x0ad28158 0x00002dc9eb2ee142 0x16782dc9ead28158 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0xc1be7e6e 0x0b2ef0e2 0x00002dc9eb2ef0e2 0x16782dc9eb2ef0e2 0x16782dc9fad2c172
MERGE Data Link (29984): bad timestamp  6 3 0xc1b08006 0x0b2f5e42 0x00002dc9eb2f5e42 0x16782dc9eb2f5e42 0x16782dc9fad34502
MERGE Data Link (29984): bad timestamp  6 3 0xc1807de7 0x0ad3c342 0x00002dc9eb2f6de2 0x16782dc9ead3c342 0x16782dc9fad34502
MERGE Statistics counters now cleared

16.21 merger server terminal reset & clear

      TapeData -> /media/ThirdDRive/TapeData

[npg@aidas-gsi bin64]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   13G   34G  28% /
tmpfs                 7.9G  559M  7.3G   7% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   79G  302G  21% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  6.3T  605G  92% /media/SecondDrive
/dev/sdc1             7.2T  3.6T  3.3T  52% /media/ThirdDrive

      at current rate (10.5Mb/s) we have 3.6d remaining 
      
17.40 *all* system wide checks OK *except* FPGA timestamp errors

		 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x51 : 	 81  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

17.43 FEE64 temps OK - attachment 1
      statistics - attachments 2-8
      per FEE64 rate spectra - attachments 9 & 10
      1.8.L spectra - attachments 11-14
      1.8.H spectra - attachments 15-16
      1.8.W spectra 20us FSR - attachments 17-18
      1.8.W spectra 200us, 2ms & 20ms FSR - attachments 19-21, 30-32
      merger, tape server & merger statistics 19-25
      ucesb - attachment 26
      Grafana DSSD biases & leakage currents - attachment 27
      taperserver & tapeserver/merger server terminals - no timewarp errors since c. 15.25 - attachment 28
      DSSSD biases & leakage currents - attachment 29


22.17 *all* system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x3612 : 	 0x3613 : 	 1  
aida02 fault 	 0x523c : 	 0x523d : 	 1  
aida03 fault 	 0xc5cd : 	 0xc5ce : 	 1  
aida04 fault 	 0x14f1 : 	 0x14f2 : 	 1  
aida05 fault 	 0x9a4e : 	 0x9a4f : 	 1  
aida06 fault 	 0xfdd2 : 	 0xfdd3 : 	 1  
aida07 fault 	 0x4a0 : 	 0x4a1 : 	 1  
aida08 fault 	 0xc796 : 	 0xc797 : 	 1  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x57 : 	 87  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors

MERGE Data Link (29979): bad timestamp  1 3 0xc05080ef 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc06f80d7 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0778222 0x0c88b272 0x000047aa2c88b272 0x167847aa2c88b272 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0598098 0x0c88e152 0x000047aa2c88e152 0x167847aa2c88e152 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc042815b 0x0c894eb2 0x000047aa2c894eb2 0x167847aa2c894eb2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc05980eb 0x0c899502 0x000047aa2c899502 0x167847aa2c899502 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc075813c 0x0c89e322 0x000047aa2c89e322 0x167847aa2c89e322 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc04280d5 0x0c89f2c2 0x000047aa2c89f2c2 0x167847aa2c89f2c2 0x167847aa2c8a34d2
MERGE Actor (29995): Working with 0 from 12 data sources
MERGE Actor (29995): Working with 0 from 12 data sources

22.20 DSSSD biases & leakage currents - attachment 33
      FEE64 temps OK - attachment 34
      statistics - attachments 35
      merger statistics 36

      
Entry  Fri Apr 23 12:12:08 2021, TD, aida02 restart - 03.13UTC+2 Friday 23 April 
Overnight aida02 crashed and restarted itself. A power cycle of all FEE64s was required to recover.

The system log appears to show the aida02 embedded processor crashed (note log times are UTC+1
for some reason).

Apr 21 00:41:21 aida02 kernel: xaida: open:
Apr 23 02:09:41 aida02 kernel: Unable to handle kernel paging request for data at address 0x00200200
Apr 23 02:09:41 aida02 kernel: Faulting instruction address: 0xc0091dd8
Apr 23 02:09:41 aida02 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 23 02:09:41 aida02 kernel: PREEMPT Xilinx Virtex440
Apr 23 02:09:41 aida02 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 23 02:09:42 aida02 kernel: NIP: c0091dd8 LR: c0091e40 CTR: 00000000
Apr 23 02:09:42 aida02 kernel: REGS: c6835e60 TRAP: 0300   Not tainted  (2.6.31)
Apr 23 02:09:42 aida02 kernel: MSR: 00021000 <ME,CE>  CR: 24000088  XER: 00000000
Apr 23 02:09:42 aida02 kernel: DEAR: 00200200, ESR: 00800000
Apr 23 02:09:42 aida02 kernel: TASK = c6824880[5] 'events/0' THREAD: c6834000
Apr 23 02:09:42 aida02 kernel: GPR00: c0091e40 c6835f10 c6824880 00000001 00000003 c038c3a8 00000000 c0498000
Apr 23 02:09:42 aida02 kernel: GPR08: 00000008 00100100 00000051 00200200 00485e6c 00005aa8 4805de90 00000001
Apr 23 02:09:42 aida02 kernel: GPR16: 00000000 00000000 4801d000 00000003 c031aef4 c02fb040 c02fb008 00000002
Apr 23 02:09:42 aida02 kernel: GPR24: c6834000 00100100 00200200 c680ae00 00000001 c680daf0 c680dae0 c6940020
Apr 23 02:09:42 aida02 kernel: NIP [c0091dd8] drain_freelist+0x54/0x110
Apr 23 02:09:42 aida02 kernel: LR [c0091e40] drain_freelist+0xbc/0x110
Apr 23 02:09:42 aida02 kernel: Call Trace:
Apr 23 02:09:42 aida02 kernel: [c6835f10] [c0091e40] drain_freelist+0xbc/0x110 (unreliable)
Apr 23 02:09:42 aida02 kernel: [c6835f40] [c0093158] cache_reap+0x104/0x138
Apr 23 02:09:42 aida02 kernel: [c6835f60] [c0048908] worker_thread+0x140/0x204
Apr 23 02:09:42 aida02 kernel: [c6835fc0] [c004cfc8] kthread+0x78/0x7c
Apr 23 02:09:42 aida02 kernel: [c6835ff0] [c000e140] kernel_thread+0x4c/0x68
Apr 23 02:09:42 aida02 kernel: Instruction dump:
Apr 23 02:09:43 aida02 kernel: 3ba40010 7f80e800 419e00d4 3f200010 3f400020 54380024 63390100 635a0200
Apr 23 02:09:43 aida02 kernel: 3b800000 48000068 813f0000 817f0004 <912b0000> 91690004 933f0000 935f0004
Apr 23 02:09:43 aida02 kernel: ---[ end trace 6baa5b82e4ef4e92 ]---
Apr 23 02:09:43 aida02 kernel: note: events/0[5] exited with preempt_count 1
Apr 23 02:13:39 aida02 syslogd 1.4.2: restart.
Apr 23 02:13:39 aida02 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Apr 23 02:13:39 aida02 kernel: Using Xilinx Virtex440 machine description
Apr 23 02:13:39 aida02 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Entry  Fri Apr 23 20:19:52 2021, TD, Friday 23 April 20.00-00.00 7x
21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378
Entry  Sat Apr 24 03:16:19 2021, TD, aida-gsi window manager/login session crashed 
Looks like the aida-gsi window manager/login session crashed.

On site S460 shift restarted AnyDesk. Upon re-connection *all* browsers, terminals, applications were gone
I have re-started the basics but BNC PB-5 pulser, CAEN NIM bin, MBS rate monitor, system console monitor etc
still need to be re-started. 

Extract from /var/log/messages

Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal cpuset=/ mems_allowed=0
Apr 24 03:19:01 aidas-gsi kernel: Pid: 18955, comm: gnome-terminal Not tainted 2.6.32-696.el6.x86_64 #1
Apr 24 03:19:01 aidas-gsi kernel: Call Trace:
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131550>] ? dump_header+0x90/0x1b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8123c1dc>] ? security_real_capable_noaudit+0x3c/0x70
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811319d2>] ? oom_kill_process+0x82/0x2a0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131911>] ? select_bad_process+0xe1/0x120
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131e10>] ? out_of_memory+0x220/0x3c0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113e7ec>] ? __alloc_pages_nodemask+0x93c/0x950
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81015533>] ? native_sched_clock+0x13/0x80
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff810aaea1>] ? lock_hrtimer_base+0x31/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81177a3a>] ? alloc_pages_current+0xaa/0x110
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113bb4e>] ? __get_free_pages+0xe/0x50
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1944>] ? __pollwait+0xb4/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81348684>] ? n_tty_poll+0x194/0x1d0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff813440aa>] ? tty_poll+0x8a/0xa0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1db5>] ? do_sys_poll+0x2d5/0x5b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1890>] ? __pollwait+0x0/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1745>] ? poll_select_set_timeout+0x95/0xb0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b2281>] ? sys_poll+0x71/0x100
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Apr 24 03:19:01 aidas-gsi kernel: Mem-Info:
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 172
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  45
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd:  97
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 142
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  49
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd:  56
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 179
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd:  67
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 107
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 170
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  54
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 118
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 196
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  36
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  37
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd: 128
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 173
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd: 161
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 177
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 178
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  99
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 183
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 162
Apr 24 03:19:01 aidas-gsi kernel: active_anon:3408671 inactive_anon:401328 isolated_anon:128
Apr 24 03:19:01 aidas-gsi kernel: active_file:22995 inactive_file:61739 isolated_file:0
Apr 24 03:19:01 aidas-gsi kernel: unevictable:0 dirty:59856 writeback:22 unstable:0
Apr 24 03:19:01 aidas-gsi kernel: free:35843 slab_reclaimable:15134 slab_unreclaimable:65491
Apr 24 03:19:01 aidas-gsi kernel: mapped:129529 shmem:249820 pagetables:24586 bounce:0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA free:15752kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 874 16024 16024
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 free:64204kB min:3680kB low:4600kB high:5520kB active_anon:369500kB inactive_anon:377436kB active_file:10988kB 
inactive_file:26828kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:895084kB mlocked:0kB dirty:26856kB writeback:88kB mapped:35896kB shmem:39156kB 
slab_reclaimable:4224kB slab_unreclaimable:4344kB kernel_stack:3344kB pagetables:2032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:65792 all_unreclaimable? 
no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 15150 15150
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal free:63416kB min:63836kB low:79792kB high:95752kB active_anon:13265184kB inactive_anon:1227876kB active_file:80992kB 
inactive_file:220128kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15513600kB mlocked:0kB dirty:212568kB writeback:0kB mapped:482220kB 
shmem:960124kB slab_reclaimable:56312kB slab_unreclaimable:257620kB kernel_stack:17600kB pagetables:96312kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:17824 all_unreclaimable? no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 0 0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32: 373*4kB 559*8kB 428*16kB 296*32kB 189*64kB 111*128kB 43*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 64204kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal: 16100*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64408kB
Apr 24 03:19:01 aidas-gsi kernel: 357788 total pagecache pages
Apr 24 03:19:01 aidas-gsi kernel: 22930 pages in swap cache
Apr 24 03:19:01 aidas-gsi kernel: Swap cache stats: add 3720478, delete 3697548, find 5026852/5228898
Apr 24 03:19:01 aidas-gsi kernel: Free swap  = 1826100kB
Apr 24 03:19:01 aidas-gsi kernel: Total swap = 8200188kB
Apr 24 03:19:01 aidas-gsi kernel: 4194303 pages RAM
Apr 24 03:19:01 aidas-gsi kernel: 97868 pages reserved
Apr 24 03:19:01 aidas-gsi kernel: 578727 pages shared
Apr 24 03:19:01 aidas-gsi kernel: 3846484 pages non-shared
Apr 24 03:19:01 aidas-gsi kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 24 03:19:01 aidas-gsi kernel: [ 1388]     0  1388     2819      130   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 3927]     0  3927     7442      128  12     -17         -1000 auditd
Apr 24 03:19:01 aidas-gsi kernel: [ 3961]     0  3961    62914      755  15       0             0 rsyslogd
Apr 24 03:19:01 aidas-gsi kernel: [ 4029]     0  4029     4604      141   7       0             0 irqbalance
Apr 24 03:19:01 aidas-gsi kernel: [ 4054]    32  4054     4746      124  13       0             0 rpcbind
Apr 24 03:19:01 aidas-gsi kernel: [ 4074]    81  4074     8194      440   6       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 4089]     0  4089    26490     2647   4       0             0 NetworkManager
Apr 24 03:19:01 aidas-gsi kernel: [ 4096]     0  4096    14534      328  12       0             0 modem-manager
Apr 24 03:19:01 aidas-gsi kernel: [ 4112]    29  4112     5839       97   0       0             0 rpc.statd
Apr 24 03:19:01 aidas-gsi kernel: [ 4151]     0  4151    47297      117  15       0             0 cupsd
Apr 24 03:19:01 aidas-gsi kernel: [ 4171]     0  4171    11251       75   9       0             0 wpa_supplicant
Apr 24 03:19:01 aidas-gsi kernel: [ 4180]     0  4180     2280      162   9       0             0 dhclient
Apr 24 03:19:01 aidas-gsi kernel: [ 4210]     0  4210     1020       82   8       0             0 acpid
Apr 24 03:19:01 aidas-gsi kernel: [ 4222]    68  4222     9968      569  15       0             0 hald
Apr 24 03:19:01 aidas-gsi kernel: [ 4223]     0  4223     5100      184  12       0             0 hald-runner
Apr 24 03:19:01 aidas-gsi kernel: [ 4271]     0  4271     5630      157  12       0             0 hald-addon-inpu
Apr 24 03:19:01 aidas-gsi kernel: [ 4272]    68  4272     4502      120   0       0             0 hald-addon-acpi
Apr 24 03:19:01 aidas-gsi kernel: [ 4306]     0  4306    96540      161   0       0             0 automount
Apr 24 03:19:01 aidas-gsi kernel: [ 4492]     0  4492    26831       15   9       0             0 rpc.rquotad
Apr 24 03:19:01 aidas-gsi kernel: [ 4497]     0  4497     5450      213   4       0             0 rpc.mountd
Apr 24 03:19:01 aidas-gsi kernel: [ 4542]     0  4542     5776       42   4       0             0 rpc.idmapd
Apr 24 03:19:01 aidas-gsi kernel: [ 4567]     0  4567    16573      113   3     -17         -1000 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 4578]     0  4578     5429      117   7       0             0 xinetd
Apr 24 03:19:01 aidas-gsi kernel: [ 4589]    38  4589     7686      213  10       0             0 ntpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4675]     0  4675    20260      147   5       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [ 4682]    89  4682    20323      141   5       0             0 qmgr
Apr 24 03:19:01 aidas-gsi kernel: [ 4696]     0  4696    27153      253   0       0             0 ksmtuned
Apr 24 03:19:01 aidas-gsi kernel: [ 4709]     0  4709    29222      137  10       0             0 crond
Apr 24 03:19:01 aidas-gsi kernel: [ 4724]     0  4724     5277       50  12       0             0 atd
Apr 24 03:19:01 aidas-gsi kernel: [ 4741]     0  4741   172796     2328   0       0             0 virt-who
Apr 24 03:19:01 aidas-gsi kernel: [ 4762]     0  4762   256599      407  10       0             0 libvirtd
Apr 24 03:19:01 aidas-gsi kernel: [ 4858]     0  4858    16126      150  10       0             0 certmonger
Apr 24 03:19:01 aidas-gsi kernel: [ 4903]    99  4903     3223      102   2       0             0 dnsmasq
Apr 24 03:19:01 aidas-gsi kernel: [ 4918]     0  4918     4539      223   5       0             0 dhcpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4920]     0  4920    33509      312  11       0             0 gdm-binary
Apr 24 03:19:01 aidas-gsi kernel: [ 4927]     0  4927     1016       61  11       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4929]     0  4929     1016       61  12       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4931]     0  4931     1016       61   9       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4933]     0  4933     1016       61  15       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4935]     0  4935     1016       61   1       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4944]     0  4944     3148      440   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 4945]     0  4945     3148      438   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 5050]     0  5050  1029198      377   3       0             0 console-kit-dae
Apr 24 03:19:01 aidas-gsi kernel: [ 5126]     0  5126    12508      365   3       0             0 devkit-power-da
Apr 24 03:19:01 aidas-gsi kernel: [ 5164]     0  5164    45435      581   1       0             0 polkitd
Apr 24 03:19:01 aidas-gsi kernel: [ 5175]   499  5175    42114      146   0       0             0 rtkit-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5702]     0  5702    14009      514   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5725]     0  5725    11306       74   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18391]     0 18391    41240      293   8       0             0 gdm-simple-slav
Apr 24 03:19:01 aidas-gsi kernel: [18393]     0 18393  3476635  2412883  13       0             0 Xorg
Apr 24 03:19:01 aidas-gsi kernel: [18427]    42 18427     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18464]     0 18464    47926      269   0       0             0 gdm-session-wor
Apr 24 03:19:01 aidas-gsi kernel: [18510]   500 18510    57281      424   1       0             0 gnome-keyring-d
Apr 24 03:19:01 aidas-gsi kernel: [18520]   500 18520    64493      751   0       0             0 gnome-session
Apr 24 03:19:01 aidas-gsi kernel: [18528]   500 18528     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18529]   500 18529     8211      379   8       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18604]   500 18604    33379     1113  13       0             0 gconfd-2
Apr 24 03:19:01 aidas-gsi kernel: [18612]   500 18612   139831     1984   9       0             0 gnome-settings-
Apr 24 03:19:01 aidas-gsi kernel: [18614]   500 18614    71643      696  12       0             0 seahorse-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18616]   500 18616    34351      298  13       0             0 gvfsd
Apr 24 03:19:01 aidas-gsi kernel: [18636]   500 18636   112427     2321   4       0             0 metacity
Apr 24 03:19:01 aidas-gsi kernel: [18637]   500 18637   203008     6568  10       0             0 gnome-panel
Apr 24 03:19:01 aidas-gsi kernel: [18638]   500 18638   326834    16727  10       0             0 nautilus
Apr 24 03:19:01 aidas-gsi kernel: [18640]   500 18640   141364      341  14       0             0 bonobo-activati
Apr 24 03:19:01 aidas-gsi kernel: [18642]   500 18642    65558      741  10       0             0 bluetooth-apple
Apr 24 03:19:01 aidas-gsi kernel: [18645]   500 18645    68687      968   4       0             0 gnome-power-man
Apr 24 03:19:01 aidas-gsi kernel: [18646]   500 18646     7601      812  12       0             0 restorecond
Apr 24 03:19:01 aidas-gsi kernel: [18648]   500 18648    28879      401   5       0             0 im-settings-dae
Apr 24 03:19:01 aidas-gsi kernel: [18649]   500 18649   132278     1323   4       0             0 gnome-volume-co
Apr 24 03:19:01 aidas-gsi kernel: [18654]   500 18654    75318     2282  10       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18657]   500 18657   119660     1244   2       0             0 trashapplet
Apr 24 03:19:01 aidas-gsi kernel: [18659]   500 18659    88087     3906   5       0             0 wnck-applet
Apr 24 03:19:01 aidas-gsi kernel: [18660]   500 18660    62988      653   7       0             0 vino-server
Apr 24 03:19:01 aidas-gsi kernel: [18661]   500 18661    72490      958   0       0             0 polkit-gnome-au
Apr 24 03:19:01 aidas-gsi kernel: [18662]   500 18662    82733     1441   5       0             0 nm-applet
Apr 24 03:19:01 aidas-gsi kernel: [18666]   500 18666   126400      635   1       0             0 gvfs-gdu-volume
Apr 24 03:19:01 aidas-gsi kernel: [18667]   500 18667    74660     1042   5       0             0 gpk-update-icon
Apr 24 03:19:01 aidas-gsi kernel: [18669]   500 18669    64033      850   0       0             0 gdu-notificatio
Apr 24 03:19:01 aidas-gsi kernel: [18676]   500 18676    81107     2344   6       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [18693]   500 18693   131702     2471   3       0             0 pulseaudio
Apr 24 03:19:01 aidas-gsi kernel: [18823]   500 18823    58170      269   4       0             0 gvfs-afc-volume
Apr 24 03:19:01 aidas-gsi kernel: [18836]   500 18836    37789      300   8       0             0 gvfs-gphoto2-vo
Apr 24 03:19:01 aidas-gsi kernel: [18844]   500 18844    36051      437   0       0             0 gvfsd-trash
Apr 24 03:19:01 aidas-gsi kernel: [18868]   500 18868    10275      287   6       0             0 gconf-im-settin
Apr 24 03:19:01 aidas-gsi kernel: [18869]   500 18869    84055     4872   3       0             0 gnome-screensav
Apr 24 03:19:01 aidas-gsi kernel: [18870]   500 18870     2057       88   3       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18871]   500 18871    27123      198   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18894]   500 18894    98681     1288   0       0             0 gdm-user-switch
Apr 24 03:19:01 aidas-gsi kernel: [18896]   500 18896    72800      911   3       0             0 notification-ar
Apr 24 03:19:01 aidas-gsi kernel: [18897]   500 18897   143660     1846   2       0             0 clock-applet
Apr 24 03:19:01 aidas-gsi kernel: [18900]   500 18900    23718      242   3       0             0 gconf-helper
Apr 24 03:19:01 aidas-gsi kernel: [18918]   500 18918    34358      265   0       0             0 gvfsd-burn
Apr 24 03:19:01 aidas-gsi kernel: [18932]   500 18932    33897      307   4       0             0 gvfsd-metadata
Apr 24 03:19:01 aidas-gsi kernel: [18955]   500 18955    82196     7037   7       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18957]   500 18957     2057       97   7       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18977]   500 18977    27123      114   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18991]     0 18991    47361      130  10       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [18992]     0 18992    27524      290  15       0             0 iptraf
Apr 24 03:19:01 aidas-gsi kernel: [21973]   500 21973  1058234   192783   0       0             0 firefox
Apr 24 03:19:01 aidas-gsi kernel: [21987]   500 21987    68608      284   3       0             0 at-spi-bus-laun
Apr 24 03:19:01 aidas-gsi kernel: [22221]   500 22221   731047    34381   0       0             0 Privileged Cont
Apr 24 03:19:01 aidas-gsi kernel: [22248]   500 22248   705574    10970   0       0             0 WebExtensions
Apr 24 03:19:01 aidas-gsi kernel: [22335]   500 22335   960920   166074   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22407]   500 22407  1023120    77418   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22810]   500 22810    27123      114   2       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [22826]   500 22826    15487      185   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [22830]   500 22830    27123      114   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26234]   500 26234   839272    43450   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [26311]   500 26311    27123      282   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26884]   500 26884    27123      115   5       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19388]   500 19388    27123      293   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11683]   500 11683   791783    18526   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [ 4876]   500  4876    54985      304   5       0             0 gvfsd-computer
Apr 24 03:19:01 aidas-gsi kernel: [17248]   500 17248    29595      100   1       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17249]   500 17249    29631      157  10       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17250]   500 17250    27124      114   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17266]   500 17266    15489      181   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [19879]     0 19879    25531      154   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19883]   500 19883    25531      163   2       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19884]   500 19884    27638      106  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19907]   500 19907    29595       97   5       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [14850]   500 14850    56156      327   2       0             0 gvfsd-network
Apr 24 03:19:01 aidas-gsi kernel: [14855]   500 14855    36479      318   3       0             0 gvfsd-dnssd
Apr 24 03:19:01 aidas-gsi kernel: [22615]   500 22615    27123      117   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [13867]     0 13867    25531      161   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13880]   500 13880    25645      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13881]   500 13881    14400      138   8       0             0 sftp-server
Apr 24 03:19:01 aidas-gsi kernel: [20258]   500 20258   776888    93602   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [25910]   500 25910    15050      209   3       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [26010]   500 26010     2879       95  15       0             0 tinyproxy
Apr 24 03:19:01 aidas-gsi kernel: [26513]     0 26513    25530      204  12       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [26517]   500 26517    25530      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29203]     0 29203    25016      164  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29207]   500 29207    25534      158   6       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 1105]   500  1105    27123      143   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17163]     0 17163    25016      190  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [17167]   500 17167    25534      185   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18690]     0 18690    25530      183   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18695]   500 18695    25530      138   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18696]   500 18696    27123      127   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [20802]     0 20802    25530      173   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20806]   500 20806    25530      142   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20807]   500 20807    27123      172   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 9697]     0  9697    25530      187   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9701]   500  9701    25563      184  10       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9703]   500  9703    27123      132   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [21737]   500 21737    29350      360  10       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [21860]   500 21860   166837     1623   5       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21861]   500 21861   208277     3415   0       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21868]   500 21868   470786     1884   2       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [20297]   500 20297   731145    44730   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [23452]     0 23452    47361      305   5       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [23453]     0 23453    25240       86   2       0             0 tail
Apr 24 03:19:01 aidas-gsi kernel: [23841]   500 23841    27123      243   3       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [  345]   500   345    27156      311  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 4825]   500  4825   682055    25908   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [16950]   500 16950    26921      257   1       0             0 watch
Apr 24 03:19:01 aidas-gsi kernel: [10822]   500 10822  1060110   432573   3       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [29109]   500 29109    27186      482   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 3754]   500  3754    27123      375   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11083]   500 11083    28030      216   3       0             0 HTTPD@8115
Apr 24 03:19:01 aidas-gsi kernel: [11087]   500 11087    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11089]   500 11089    36731     3211  12       0             0 tclsh8.5_copy2
Apr 24 03:19:01 aidas-gsi kernel: [11097]   500 11097    28030      207   1       0             0 TapeServer
Apr 24 03:19:01 aidas-gsi kernel: [11104]   500 11104     5398     1567  10       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [11105]   500 11105    25231      134  12       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11106]   500 11106     4323      187   6       0             0 stats
Apr 24 03:19:01 aidas-gsi kernel: [11107]   500 11107     4918      219   5       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11108]   500 11108     4917      194   9       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11109]   500 11109     4917      194  12       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11110]   500 11110     4630      214   0       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11111]   500 11111     4630      182   9       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11114]   500 11114    28030      217  12       0             0 NewMerger
Apr 24 03:19:01 aidas-gsi kernel: [11124]   500 11124   404629     8857   0       0             0 master64
Apr 24 03:19:01 aidas-gsi kernel: [11125]   500 11125    25231      135   9       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11126]   500 11126    19761    17318  13       0             0 statrate64
Apr 24 03:19:01 aidas-gsi kernel: [11127]   500 11127    35968    11059   2       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11128]   500 11128    35968    11498  11       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11129]   500 11129    35968    11581  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11130]   500 11130    35968    12366   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11131]   500 11131    35968    10031   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11132]   500 11132    35968    10689   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11133]   500 11133    35968    11896   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11134]   500 11134    35968    10510  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11135]   500 11135    35968    15679  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11136]   500 11136    35968    14623  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11137]   500 11137    35968    12790  15       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11138]   500 11138    35968    11683   0       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11139]   500 11139    35968     8773   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11140]   500 11140    35968     8774  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11141]   500 11141    35968     8772  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11142]   500 11142    35968     8773  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11152]   500 11152   307392    47667   5       0             0 merge64
Apr 24 03:19:01 aidas-gsi kernel: [11200]   500 11200   723346    37902   5       0             0 file:// Content
Apr 24 03:19:01 aidas-gsi kernel: [11338]   500 11338    28030      216   0       0             0 HTTPD
Apr 24 03:19:01 aidas-gsi kernel: [11342]   500 11342    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11344]   500 11344    46965    14788   3       0             0 tclsh8.5
Apr 24 03:19:01 aidas-gsi kernel: [12835]   500 12835   766232    71066   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [24071]   500 24071    27123      409   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [24347]   500 24347    28030      231   4       0             0 datarelaymbs
Apr 24 03:19:01 aidas-gsi kernel: [24351]   500 24351     7751     1313  10       0             0 DataRelayMBS
Apr 24 03:19:01 aidas-gsi kernel: [  407]    89   407    20280      470  13       0             0 pickup
Apr 24 03:19:01 aidas-gsi kernel: [ 1586]     0  1586     4238      156  13       0             0 anacron
Apr 24 03:19:01 aidas-gsi kernel: [ 1876]     0  1876    25231      144  10       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: Out of memory: Kill process 18393 (Xorg) score 523 or sacrifice child
Apr 24 03:19:01 aidas-gsi kernel: Killed process 18393, UID 0, (Xorg) total-vm:13906540kB, anon-rss:9558800kB, file-rss:92712kB
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11471" (uid=500 pid=1884 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11473" (uid=500 pid=1887 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:45 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:19:47 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1961 of process 1961 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Apr 24 03:19:48 aidas-gsi pulseaudio[1961]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1965 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1966 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1967 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11485" (uid=500 pid=1981 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11487" (uid=500 pid=1983 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: DNS-SD initialization failed: Daemon not running
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: init gpgme version 1.1.8
Apr 24 03:30:44 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2243 of process 2243 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:45 aidas-gsi pulseaudio[2243]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2301 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2302 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2303 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2325 of process 2325 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2325]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2326 of process 2326 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2326]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11506" (uid=500 pid=2328 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2418 of process 2418 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2418]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11508" (uid=500 pid=2420 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:51 aidas-gsi pulseaudio[2243]: ratelimit.c: 25 events suppressed
Apr 24 03:30:51 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr 24 03:33:11 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11521" (uid=500 pid=3005 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11524" (uid=500 pid=3012 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11525" (uid=500 pid=3015 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11527" (uid=500 pid=3018 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11529" (uid=500 pid=3022 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11531" (uid=500 pid=3024 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:13 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11533" (uid=500 pid=3100 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:37 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:40:46 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:41:22 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:19 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:43:31 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:47:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.7 (192.168.11.99) from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.4 (192.168.11.99) from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.3 (192.168.11.99) from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.8 (192.168.11.99) from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.2 (192.168.11.99) from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.1 (192.168.11.99) from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.6 (192.168.11.99) from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.5 (192.168.11.99) from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:816 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:952 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:884 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:895 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:892 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:839 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:897 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.8
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.1
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.7
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.3
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.4
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.2
Apr 24 03:51:04 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.5
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:29 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:06 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.9 (192.168.11.99) from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.10 (192.168.11.99) from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.11 (192.168.11.99) from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.12 (192.168.11.99) from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:814 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:852 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:694 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:770 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.9
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.10
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.12
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.11
Apr 24 03:52:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Entry  Sat Apr 24 13:33:44 2021, TD, aida01 restart - 22.07UTC+2 Tuesday 20 April 
From aida01 /var/log/messages

Apr 20 21:04:06 aida01 kernel: BUG: scheduling while atomic: tclsh8.5/398/0x00000201
Apr 20 21:04:08 aida01 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 20 21:04:09 aida01 kernel: Call Trace:
Apr 20 21:04:11 aida01 kernel: [c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
Apr 20 21:04:12 aida01 kernel: [c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
Apr 20 21:04:14 aida01 kernel: [c6941aa0] [c02a8020] schedule+0x2f4/0x34c
Apr 20 21:07:53 aida01 syslogd 1.4.2: restart.

From ttyUSB0 system console log

20/21:04:06|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:06|Call Trace:
20/21:04:06|[c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:06|[c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:07|[c6941aa0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:07|[c6941ae0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:07|[c6941af0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:07|[c6941b10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:07|[c6941b50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:07|[c6941b90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:07|[c6941ba0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:07|[c6941bd0] [c013da58] nfs3_rpc_wrapper+0x48/0x84
20/21:04:07|[c6941bf0] [c013e734] nfs3_proc_access+0xcc/0x1ac
20/21:04:07|[c6941cc0] [c0129644] nfs_do_access+0xb8/0x438
20/21:04:07|[c6941d00] [c0129a6c] nfs_permission+0xa8/0x1c4
20/21:04:07|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:07|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:07|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:07|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:07|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:07|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:07|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:07|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:07|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:08|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:08|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:08|Call Trace:
20/21:04:08|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:08|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:08|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:08|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:08|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:08|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:08|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:08|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:08|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:08|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:08|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:08|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:08|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:08|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:09|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:09|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:09|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:09|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:09|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:09|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:09|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:09|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:09|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:09|Call Trace:
20/21:04:09|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:09|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:09|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:09|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:09|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:09|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:09|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:09|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:09|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:09|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:10|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:10|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:10|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:10|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:10|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:10|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:10|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:10|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:10|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:10|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:10|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:10|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:10|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:10|Call Trace:
20/21:04:10|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:10|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:10|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:10|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:10|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:10|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:11|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:11|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:11|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:11|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:11|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:11|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:11|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:11|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:11|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:11|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:11|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:11|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:11|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:11|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:11|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:11|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:11|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:11|Call Trace:
20/21:04:11|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:11|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:12|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:12|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:12|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:12|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:12|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:12|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:12|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:12|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:12|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:12|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:12|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:12|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:12|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:12|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:12|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:12|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:12|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:12|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:12|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:12|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:13|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:13|Call Trace:
20/21:04:13|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:13|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:13|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:13|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:13|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:13|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:13|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:13|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:13|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:13|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:13|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:13|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:13|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:13|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:13|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:13|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:13|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:13|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:14|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:14|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:14|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:14|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:14|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:14|Call Trace:
20/21:04:14|[c6941950] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:14|[c6941990] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:14|[c69419a0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:14|[c69419e0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:14|[c69419f0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:14|[c6941a10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:14|[c6941a50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:14|[c6941a90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:14|[c6941aa0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:14|[c6941ad0] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:14|[c6941b00] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:14|[c6941bc0] [c012bd70] nfs_lookup_revalidate+0x3c0/0x464
20/21:04:14|[c6941d00] [c00a0f7c] do_lookup+0x54/0x20c
20/21:04:14|[c6941d30] [c00a308c] __link_path_walk+0x57c/0xf7c
20/21:04:15|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:15|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:15|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:15|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:15|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:15|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:15|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:15|Page fault in user mode with in_atomic() = 1 mm = c6178c80
20/21:04:15|NIP = fb85a30  MSR = 2d000
20/21:04:15|Oops: Weird page fault, sig: 11 [#1]
20/21:04:15|PREEMPT Xilinx Virtex440
20/21:04:15|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:15|NIP: 0fb85a30 LR: 0ff85338 CTR: c01288f4
20/21:04:15|REGS: c6941f50 TRAP: 0401   Not tainted  (2.6.31)
20/21:04:15|MSR: 0002d000 <EE,PR,ME,CE>  CR: 44884044  XER: 00000000
20/21:04:15|TASK = c608e0c0[398] 'tclsh8.5' THREAD: c6940000
20/21:04:15|GPR00: 0ff85334 bf9a3870 4802b0a0 ffffffff 0fc094e0 54884044 fffffff2 0fc08d3c
20/21:04:15|GPR08: 0002d000 480240a8 00000000 bf9a3870 c6940000
20/21:04:15|NIP [0fb85a30] 0xfb85a30
20/21:04:15|LR [0ff85338] 0xff85338
20/21:04:16|Call Trace:
20/21:04:16|Kernel panic - not syncing: Fatal exception in interrupt
20/21:04:16|Call Trace:
20/21:04:16|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:16|[c6941e20] [c00345bc] panic+0x94/0x168
20/21:04:16|[c6941e70] [c000bd44] die+0x178/0x18c
20/21:04:16|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
20/21:04:16|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
20/21:04:16|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/21:07:16|
Found 0 errors checking kernel image
20/21:07:18|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/21:07:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/21:07:18|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/21:07:18|SPIconfig done now to set up the LMK3200 registers
20/21:07:18|LMK #0 : regInit[0]=0x80000000
20/21:07:18|LMK #0 : regInit[1]=0x10070600
20/21:07:18|LMK #0 : regInit[2]=0x60601
20/21:07:18|LMK #0 : regInit[3]=0x60602
20/21:07:18|LMK #0 : regInit[4]=0x60603
20/21:07:18|LMK #0 : regInit[5]=0x70624
20/21:07:18|LMK #0 : regInit[6]=0x70605
20/21:07:18|LMK #0 : regInit[7]=0x70606
20/21:07:18|LMK #0 : regInit[8]=0x70627
20/21:07:18|LMK #0 : regInit[9]=0x10000908
20/21:07:18|LMK #0 : regInit[10]=0xA0022A09
20/21:07:18|LMK #0 : regInit[11]=0x82800B
20/21:07:18|LMK #0 : regInit[12]=0x28C800D
20/21:07:18|LMK #0 : regInit[13]=0x830020E
20/21:07:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/21:07:19|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/21:07:19|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
ADCs calibrated
20/21:07:19|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
20/21:07:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/21:07:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xba 0x8a
20/21:07:29|Using Xilinx Virtex440 machine description
20/21:07:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/21:07:30|Zone PFN ranges:
20/21:07:30|  DMA      0x00000000 -> 0x00007000
20/21:07:30|  Normal   0x00007000 -> 0x00007000
20/21:07:30|Movable zone start PFN for each node
20/21:07:30|early_node_map[1] active PFN ranges
20/21:07:30|    0: 0x00000000 -> 0x00007000
20/21:07:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/21:07:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/21:07:30|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/21:07:30|PID hash table entries: 512 (order: 9, 2048 bytes)
20/21:07:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/21:07:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/21:07:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/21:07:30|Kernel virtual memory layout:
20/21:07:30|  * 0xffffe000..0xfffff000  : fixmap
20/21:07:31|  * 0xfde00000..0xfe000000  : consistent mem
20/21:07:31|  * 0xfde00000..0xfde00000  : early ioremap
20/21:07:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/21:07:31|NR_IRQS:512
20/21:07:31|clocksource: timebase mult[a00000] shift[22] registered
20/21:07:31|Console: colour dummy device 80x25
20/21:07:31|Mount-cache hash table entries: 512
20/21:07:31|NET: Registered protocol family 16
20/21:07:31|PCI: Probing PCI hardware
20/21:07:31|bio: create slab <bio-0> at 0
20/21:07:31|NET: Registered protocol family 2
20/21:07:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/21:07:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/21:07:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/21:07:31|TCP: Hash tables configured (established 4096 bind 4096)
20/21:07:31|TCP reno registered
20/21:07:31|NET: Registered protocol family 1
20/21:07:31|ROMFS MTD (C) 2007 Red Hat, Inc.
20/21:07:31|msgmni has been set to 214
20/21:07:31|io scheduler noop registered
20/21:07:31|io scheduler anticipatory registered
20/21:07:31|io scheduler deadline registered
20/21:07:31|io scheduler cfq registered (default)
20/21:07:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/21:07:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/21:07:32|console [ttyS0] enabled
20/21:07:32|brd: module loaded
20/21:07:32|loop: module loaded
20/21:07:32|Device Tree Probing 'ethernet'
20/21:07:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ba:8a
20/21:07:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/21:07:32|XLlTemac: DCR address: 0x80
20/21:07:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/21:07:32|XLlTemac: Allocating DMA descriptors with kmalloc
20/21:07:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/21:07:32|XTemac: PHY detected at address 7.
20/21:07:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/21:07:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33|Using buffer write method
20/21:07:33|cfi_cmdset_0001: Erase suspend on write enabled
20/21:07:33|cmdlinepart partition parsing not available
20/21:07:33|RedBoot partition parsing not available
20/21:07:33|Creating 5 MTD partitions on "fc000000.flash":
20/21:07:33|0x000000000000-0x000000500000 : "golden_firmware"
20/21:07:33|0x000000500000-0x000000800000 : "golden_kernel"
20/21:07:33|0x000000800000-0x000000d00000 : "user_firmware"
20/21:07:33|0x000000d00000-0x000000fe0000 : "user_kernel"
20/21:07:33|0x000000fe0000-0x000001000000 : "env_variables"
20/21:07:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/21:07:33|SPI: XIlinx spi: bus number now 32766
20/21:07:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/21:07:33|SPI: XIlinx spi: bus number now 32765
20/21:07:33|mice: PS/2 mouse device common for all mice
20/21:07:33|Device Tree Probing 'i2c'
20/21:07:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/21:07:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/21:07:33|TCP cubic registered
20/21:07:33|NET: Registered protocol family 17
20/21:07:34|RPC: Registered udp transport module.
20/21:07:34|RPC: Registered tcp transport module.
20/21:07:34|eth0: XLlTemac: Options: 0x3fa
20/21:07:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/21:07:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/21:07:34|eth0: XLlTemac: speed set to 1000Mb/s
20/21:07:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/21:07:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/21:07:36|Sending DHCP requests ., OK
20/21:07:38|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.1
20/21:07:38|IP-Config: Complete:
20/21:07:38|     device=eth0, addr=192.168.11.1, mask=255.255.255.0, gw=255.255.255.255,
20/21:07:38|     host=aida01, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/21:07:38|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01
20/21:07:38|Looking up port of RPC 100003/2 on 192.168.11.99
20/21:07:38|Looking up port of RPC 100005/1 on 192.168.11.99
20/21:07:38|VFS: Mounted root (nfs filesystem) on device 0:12.
20/21:07:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/21:07:39|Starting sysinit...
20/21:07:39|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/21:07:40|            Press 'I' to enter interactive startup.
20/21:07:40|Setting clock  (utc): Tue Apr 20 21:07:40 BST 2021 [  OK  ]
20/21:07:40|Building the cache [  OK  ]
20/21:07:40|Setting hostname aida01:  [  OK  ]
20/21:07:42|Mounting local filesystems:  [  OK  ]
20/21:07:43|Enabling /etc/fstab swaps:  [  OK  ]
20/21:07:46|Finishing sysinit...
INIT: Entering runlevel: 3
20/21:07:49|Entering non-interactive startup
20/21:07:49|FATAL: Module ipv6 not found.
20/21:07:51|Bringing up loopback interface:  [  OK  ]
20/21:07:53|FATAL: Module ipv6 not found.
20/21:07:53|Starting system logger: [  OK  ]
20/21:07:54|Starting kernel logger: [  OK  ]
20/21:07:54|Starting rpcbind: [  OK  ]
20/21:07:56|Mounting NFS filesystems:  [  OK  ]
20/21:07:57|Mounting other filesystems:  [  OK  ]
20/21:07:57|Starting xinetd: [  OK  ]
20/21:07:58|Starting midas:  Starting MIDAS Data Acquisition for aida01
20/21:07:58|xaida: device parameters: base=0x81000000 size=0x200000
20/21:08:04|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/21:08:04|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/21:08:05|xaida: driver assigned major number 254
20/21:08:05|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/21:08:10|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/21:08:10|AIDAMEM: aidamem: driver assigned major number 253
20/21:08:10|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/21:08:17|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/21:08:17|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/21:08:17|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/21:08:17|Computer Name = aida01; Temp Directory = /tmp/tcl361
20/21:08:22|package limit is not available: can't find package limit
20/21:08:25|Running with default file descriptor limit
20/21:08:25|
20/21:08:26|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
Entry  Sat Apr 24 15:13:33 2021, TD, aida09 crash - 08:10:52 Tuesday 20 April 
From ttyUSB6 system console log

20/07:07:10|kernel BUG at mm/slab.c:2974!
20/07:07:10|Oops: Exception in kernel mode, sig: 5 [#1]
20/07:07:10|PREEMPT Xilinx Virtex440
20/07:07:10|Modules linked in: aidamem xdriver xh_spidev_register
20/07:07:10|NIP: c009211c LR: c00920b0 CTR: c0038e70
20/07:07:10|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)
20/07:07:10|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000
20/07:07:10|TASK = c036e318[0] 'swapper' THREAD: c0390000
20/07:07:10|GPR00: 00000001 c0391ca0 c036e318 c680daf0 00000020 00000001 c6940000 00000000
20/07:07:11|GPR08: fffffff4 00010003 c680e400 c0390000 24008042 00005aa8 c03a0000 00000020
20/07:07:11|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200
20/07:07:11|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400
20/07:07:11|NIP [c009211c] cache_alloc_refill+0x130/0x608
20/07:07:11|LR [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:11|Call Trace:
20/07:07:11|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
20/07:07:11|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:11|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:11|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:11|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:11|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:11|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:11|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:11|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:11|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:11|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:11|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:12|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:12|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:12|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:12|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:12|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:12|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:12|Instruction dump:
20/07:07:12|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010
20/07:07:12|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840
20/07:07:12|Kernel panic - not syncing: Fatal exception in interrupt
20/07:07:12|Call Trace:
20/07:07:12|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/07:07:12|[c0391a80] [c00345bc] panic+0x94/0x168
20/07:07:12|[c0391ad0] [c000bd44] die+0x178/0x18c
20/07:07:12|[c0391af0] [c000c000] _exception+0x164/0x1b4
20/07:07:12|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c
20/07:07:12|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:12|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:12|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:12|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:13|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:13|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:13|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:13|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:13|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:13|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:13|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:13|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:13|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:13|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:13|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:13|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:13|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:13|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:13|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/07:10:14|
Found 0 errors checking kernel image
20/07:10:15|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/07:10:15|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/07:10:15|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/07:10:15|SPIconfig done now to set up the LMK3200 registers
20/07:10:15|LMK #0 : regInit[0]=0x80000000
20/07:10:15|LMK #0 : regInit[1]=0x10070600
20/07:10:15|LMK #0 : regInit[2]=0x60601
20/07:10:15|LMK #0 : regInit[3]=0x60602
20/07:10:16|LMK #0 : regInit[4]=0x60603
20/07:10:16|LMK #0 : regInit[5]=0x70624
20/07:10:16|LMK #0 : regInit[6]=0x70605
20/07:10:16|LMK #0 : regInit[7]=0x70606
20/07:10:16|LMK #0 : regInit[8]=0x70627
20/07:10:16|LMK #0 : regInit[9]=0x10000908
20/07:10:16|LMK #0 : regInit[10]=0xA0022A09
20/07:10:16|LMK #0 : regInit[11]=0x82800B
20/07:10:16|LMK #0 : regInit[12]=0x28C800D
20/07:10:16|LMK #0 : regInit[13]=0x830020E
20/07:10:16|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/07:10:16|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/07:10:16|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/07:10:16|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/07:10:17|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/07:10:20|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xee
20/07:10:26|Using Xilinx Virtex440 machine description
20/07:10:27|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/07:10:27|Zone PFN ranges:
20/07:10:27|  DMA      0x00000000 -> 0x00007000
20/07:10:27|  Normal   0x00007000 -> 0x00007000
20/07:10:27|Movable zone start PFN for each node
20/07:10:27|early_node_map[1] active PFN ranges
20/07:10:27|    0: 0x00000000 -> 0x00007000
20/07:10:27|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/07:10:27|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/07:10:27|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/07:10:27|PID hash table entries: 512 (order: 9, 2048 bytes)
20/07:10:27|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/07:10:27|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/07:10:28|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/07:10:28|Kernel virtual memory layout:
20/07:10:28|  * 0xffffe000..0xfffff000  : fixmap
20/07:10:28|  * 0xfde00000..0xfe000000  : consistent mem
20/07:10:28|  * 0xfde00000..0xfde00000  : early ioremap
20/07:10:28|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/07:10:28|NR_IRQS:512
20/07:10:28|clocksource: timebase mult[a00000] shift[22] registered
20/07:10:28|Console: colour dummy device 80x25
20/07:10:28|Mount-cache hash table entries: 512
20/07:10:28|NET: Registered protocol family 16
20/07:10:28|PCI: Probing PCI hardware
20/07:10:28|bio: create slab <bio-0> at 0
20/07:10:28|NET: Registered protocol family 2
20/07:10:28|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/07:10:28|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/07:10:28|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/07:10:28|TCP: Hash tables configured (established 4096 bind 4096)
20/07:10:28|TCP reno registered
20/07:10:28|NET: Registered protocol family 1
20/07:10:28|ROMFS MTD (C) 2007 Red Hat, Inc.
20/07:10:28|msgmni has been set to 214
20/07:10:29|io scheduler noop registered
20/07:10:29|io scheduler anticipatory registered
20/07:10:29|io scheduler deadline registered
20/07:10:29|io scheduler cfq registered (default)
20/07:10:29|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/07:10:29|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/07:10:29|console [ttyS0] enabled
20/07:10:29|brd: module loaded
20/07:10:29|loop: module loaded
20/07:10:29|Device Tree Probing 'ethernet'
20/07:10:29|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ee
20/07:10:29|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/07:10:29|XLlTemac: DCR address: 0x80
20/07:10:29|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/07:10:29|XLlTemac: Allocating DMA descriptors with kmalloc
20/07:10:29|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/07:10:29|XTemac: PHY detected at address 7.
20/07:10:29|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/07:10:29|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/07:10:29| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30|Using buffer write method
20/07:10:30|cfi_cmdset_0001: Erase suspend on write enabled
20/07:10:30|cmdlinepart partition parsing not available
20/07:10:30|RedBoot partition parsing not available
20/07:10:30|Creating 5 MTD partitions on "fc000000.flash":
20/07:10:30|0x000000000000-0x000000500000 : "golden_firmware"
20/07:10:30|0x000000500000-0x000000800000 : "golden_kernel"
20/07:10:30|0x000000800000-0x000000d00000 : "user_firmware"
20/07:10:30|0x000000d00000-0x000000fe0000 : "user_kernel"
20/07:10:30|0x000000fe0000-0x000001000000 : "env_variables"
20/07:10:30|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/07:10:30|SPI: XIlinx spi: bus number now 32766
20/07:10:30|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/07:10:30|SPI: XIlinx spi: bus number now 32765
20/07:10:30|mice: PS/2 mouse device common for all mice
20/07:10:31|Device Tree Probing 'i2c'
20/07:10:31| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/07:10:31|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/07:10:31|TCP cubic registered
20/07:10:31|NET: Registered protocol family 17
20/07:10:31|RPC: Registered udp transport module.
20/07:10:31|RPC: Registered tcp transport module.
20/07:10:31|eth0: XLlTemac: Options: 0x3fa
20/07:10:31|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/07:10:31|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/07:10:31|eth0: XLlTemac: speed set to 1000Mb/s
20/07:10:33|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/07:10:34|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/07:10:34|Sending DHCP requests ., OK
20/07:10:35|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
20/07:10:35|IP-Config: Complete:
20/07:10:35|     device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
20/07:10:35|     host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/07:10:35|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
20/07:10:35|Looking up port of RPC 100003/2 on 192.168.11.99
20/07:10:35|Looking up port of RPC 100005/1 on 192.168.11.99
20/07:10:35|VFS: Mounted root (nfs filesystem) on device 0:12.
20/07:10:35|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/07:10:36|Starting sysinit...
20/07:10:36|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/07:10:37|            Press 'I' to enter interactive startup.
20/07:10:37|Setting clock  (utc): Tue Apr 20 07:10:37 BST 2021 [  OK  ]
20/07:10:38|Building the cache [  OK  ]
20/07:10:38|Setting hostname aida09:  [  OK  ]
20/07:10:39|Mounting local filesystems:  [  OK  ]
20/07:10:40|Enabling /etc/fstab swaps:  [  OK  ]
20/07:10:43|Finishing sysinit...
INIT: Entering runlevel: 3
20/07:10:46|Entering non-interactive startup
20/07:10:46|FATAL: Module ipv6 not found.
20/07:10:47|Bringing up loopback interface:  [  OK  ]
20/07:10:49|FATAL: Module ipv6 not found.
20/07:10:50|Starting system logger: [  OK  ]
20/07:10:50|Starting kernel logger: [  OK  ]
20/07:10:50|Starting rpcbind: [  OK  ]
20/07:10:52|Mounting NFS filesystems:  [  OK  ]
20/07:10:53|Mounting other filesystems:  [  OK  ]
20/07:10:53|Starting xinetd: [  OK  ]
20/07:10:54|Starting midas:  Starting MIDAS Data Acquisition for aida09
20/07:10:54|xaida: device parameters: base=0x81000000 size=0x200000
20/07:11:00|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/07:11:00|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/07:11:00|xaida: driver assigned major number 254
20/07:11:00|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/07:11:06|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/07:11:06|AIDAMEM: aidamem: driver assigned major number 253
20/07:11:06|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/07:11:12|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/07:11:12|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/07:11:12|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/07:11:13|Computer Name = aida09; Temp Directory = /tmp/tcl361
20/07:11:17|package limit is not available: can't find package limit
20/07:11:20|Running with default file descriptor limit
20/07:11:20|package setuid is not available: can't find package setuid
20/07:11:21|Running as user 0 group 0
20/07:11:21|
20/07:11:22|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
20/07:11:22|
20/07:11:22|Built without pthreads
20/07:11:22|xaida: open:
20/07:11:22|
20/07:11:22|Creating NetAIDAMEM: aidamem_open:
20/07:11:22|Vars
20/07:11:22|Output buffer length = 65504; format option = 4; transfer option = 3
20/07:11:22|EB transfer option = 3
20/07:11:22|NetVars created and initialised
20/07:11:22|Statistics thread starting
20/07:11:22|Statistics thread created
20/07:11:22|Stat/Rate creation thread starting
20/07:11:22|Data Acquisition task has PID 375
20/07:11:22|Stat/Rate creation thread created
20/07:11:22|Hit/Rate creation thread starting
20/07:11:22|Hit/Rate creation thread created
20/07:11:22|AIDA Heartbeat thread starting
20/07:11:22|Heartbeat thread created
20/07:11:22|Installing signal handlers
20/07:11:22|Done
20/07:11:22|ModuleNum = 0
20/07:11:22|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706
20/07:11:22|Spectra table initialised
20/07:11:22|AIDA Data Acquisition now all ready to start
20/07:11:23|SIGBUS, SIGSEGV and SIGPIPE traps setup
20/07:11:23|/debug user "debug" password "5tvq-4iyxl 4"
20/07:11:30|httpd started on port 8015
20/07:11:30|
20/07:11:30|Cannot use /MIDAS/config/TclHttpd/aida09@8015/startup.tcl
20/07:11:30|Custom startup from /MIDAS/config/TclHttpd/aida09/startup.tcl
20/07:11:30|XAIDA Access package 1.0
20/07:11:32|/XAIDAAccessServer
20/07:11:32|XAD9252 Access package 1.0
20/07:11:32|/XAD9252AccessServer
20/07:11:32|/DataBaseAccessServer
20/07:11:32|[  OK  ]
20/07:11:32|/NetVarService
20/07:11:32|/SigTaskService
20/07:11:32|Loaded MemSasAccess
20/07:11:33|/SpectrumService
20/07:11:33|loading tcl/AIDARunControl.tcl for namespace ::
20/07:11:33|/DataAcquisitionControlServer
20/07:11:33|DefineMessage unknown
20/07:11:33|
20/07:11:33|Run Control Server Implementation for AIDA
20/07:11:33|DENX ELDK version 4.2 build 2008-04-01
20/07:11:33|Linux 2.6.31 on a ppc
20/07:11:33|
20/07:11:33|aida09 login: RunControlServer loaded
20/07:11:33|loading Html/RunControl/implementation.tcl
20/07:11:33|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
20/07:11:34|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
20/07:11:34|Created UI registers
20/07:11:34|RunControl loaded
20/07:11:34|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
20/07:11:34|AIDA RunControl loaded
20/07:11:35|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
20/07:11:35|do_GetState returned z=0 and 8
20/07:12:03|do_GetState returned z=0 and 8
20/07:12:09|do_GetState returned z=0 and 8
20/07:12:21|xaida: open:
20/07:12:35|do_GetState returned z=0 and 8
20/07:12:36|
Entry  Sat Apr 24 15:28:17 2021, TD, aida05 crash - 11:08:07 Tuesday 20 April 
From ttyUSB7 system console log

20/09:59:30|get_ASICBlk : Blk=1 : bytes = 793600 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0017F1D : 1 => 0FAC4BAF
20/09:59:31|Last Data : Databuffer  273884152 : 0 => 80426EF4 : 1 => 0BD70000
20/09:59:31| C03E7F21 : 03FCCB77 | C03D7EF5 : 03FCEE9F | C0047DA3 : 03FCF5A7 | C0207E21 : 03FCF5A7
20/09:59:31| C0317E77 : 03FCF5A7 | C0087E6D : 03FCF66F | C0107E98 : 03FCF66F | C0217ED8 : 03FCF66F
20/09:59:31| C0327E19 : 03FCF66F | C00B7DF0 : 03FCF737 | C0117DD5 : 03FCF737 | C0227F27 : 03FCF737
20/09:59:31| C03C7F23 : 03FCF737 | C0137EF9 : 03FCF7FF | C0297EB7 : 03FCF7FF | C03E7EC7 : 03FCF7FF
20/09:59:31| C0167DFE : 03FCF8C7 | C02A7FC3 : 03FCF8C7 | C03F7E82 : 03FCF8C7 | C01B7E91 : 03FCF98F
20/09:59:31| C02F7E8F : 03FCF98F | C01D7DC4 : 03FCFA57 | C01E7D40 : 03FCFB1F | 80426EF3 : 03FD0000
20/09:59:32| 8050023F : 03FD0000 | C03D7F1A : 03FD303F | 80426EF3 : 03FD4000 | 8050023F : 03FD4000
20/09:59:32| C0317F2F : 03FD5047 | C0047DF2 : 03FD510F | C01B7EAD : 03FD54F7 | C03D7F09 : 03FD5D8F
20/09:59:32|SIGPIPE ERROR - Error in xfer ASIC procedures: location=0
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_bxaida: release:
20/10:00:41|loAIDAMEM: aidamem_release:
20/10:00:41|ck busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|SIGSEGV ERROR - Error in xfer ASIC procedures: location=0
20/10:00:41|xfer_block busy
20/10:00:41|do_GetState returned z=0 and 1
20/10:01:27|executing generic doStop
20/10:02:41|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:02:42|StopC:
20/10:02:42|do_GetState returned z=0 and 6
20/10:02:47|do_GetState returned z=0 and 6
20/10:02:58|do_GetState returned z=0 and 6
20/10:03:10|do_GetState returned z=0 and 6
20/10:03:21|do_GetState returned z=0 and 6
20/10:03:33|do_GetState returned z=0 and 6
20/10:03:44|do_GetState returned z=0 and 6
20/10:03:56|do_GetState returned z=0 and 6
20/10:04:07|do_GetState returned z=0 and 6
20/10:04:19|do_GetState returned z=0 and 6
20/10:04:30|do_GetState returned z=0 and 6
20/10:04:42|do_GetState returned z=0 and 6
20/10:04:53|do_GetState returned z=0 and 6
20/10:05:04|do_GetState returned z=0 and 6
20/10:05:16|do_GetState returned z=0 and 6
20/10:05:27|do_GetState returned z=0 and 6
20/10:05:39|do_GetState returned z=0 and 6
20/10:05:50|do_GetState returned z=0 and 6
20/10:06:02|do_GetState returned z=0 and 6
20/10:06:12|executing generic doReset
20/10:06:21|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:06:22|ResetC:
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:34|do_GetState returned z=0 and 6
20/10:06:45|do_GetState returned z=0 and 6
20/10:06:57|do_GetState returned z=0 and 6
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
20/10:07:15|[  OK  ]
20/10:07:15|Stopping rpcbind: [  OK  ]
20/10:07:17|Shutting down kernel logger: [  OK  ]
20/10:07:19|Shutting down system logger: [  OK  ]
20/10:07:19|Stopping xinetd: [  OK  ]
20/10:07:19|[  OK  ]
20/10:07:19|Sending all processes the TERM signal...
20/10:07:20|xaida: release:
20/10:07:20|Sending all processes the KILL signal...
20/10:07:25|Saving random seed:
20/10:07:25|Syncing hardware clock to system time xaida: open:
20/10:07:25|xaida: release:
20/10:07:25|xaida: open:
20/10:07:25|xaida: ioctl: called with command 536899587
20/10:07:25|xaida: ioctl: called with command 1076129801
20/10:07:25|RTCxaida: release:
20/10:07:25|_RD_TIME: Inappropriate ioctl for device
20/10:07:25|ioctl() to /dev/rtc to read the time failed.
20/10:07:25|
20/10:07:25|Please stand by while rebooting the system...
Sj&#65533;5õÅ07:27|Restarting system.
20/10:07:29|ÉU&#65533;×®­Ë
                   R&#65533;   $T,W¤']Z&#65533;¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/10:07:29|
Found 0 errors checking kernel image
20/10:07:30|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/10:07:30|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/10:07:31|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/10:07:31|SPIconfig done now to set up the LMK3200 registers
20/10:07:31|LMK #0 : regInit[0]=0x80000000
20/10:07:31|LMK #0 : regInit[1]=0x10070600
20/10:07:31|LMK #0 : regInit[2]=0x60601
20/10:07:31|LMK #0 : regInit[3]=0x60602
20/10:07:31|LMK #0 : regInit[4]=0x60603
20/10:07:31|LMK #0 : regInit[5]=0x70624
20/10:07:31|LMK #0 : regInit[6]=0x70605
20/10:07:31|LMK #0 : regInit[7]=0x70606
20/10:07:31|LMK #0 : regInit[8]=0x70627
20/10:07:31|LMK #0 : regInit[9]=0x10000908
20/10:07:31|LMK #0 : regInit[10]=0xA0022A09
20/10:07:31|LMK #0 : regInit[11]=0x82800B
20/10:07:31|LMK #0 : regInit[12]=0x28C800D
20/10:07:31|LMK #0 : regInit[13]=0x830020E
20/10:07:31|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/10:07:32|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/10:07:32|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/10:07:32|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/10:07:32|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/10:07:35|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xd7 0xcc
20/10:07:42|Using Xilinx Virtex440 machine description
20/10:07:42|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/10:07:42|Zone PFN ranges:
20/10:07:42|  DMA      0x00000000 -> 0x00007000
20/10:07:42|  Normal   0x00007000 -> 0x00007000
20/10:07:42|Movable zone start PFN for each node
20/10:07:42|early_node_map[1] active PFN ranges
20/10:07:43|    0: 0x00000000 -> 0x00007000
20/10:07:43|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/10:07:43|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/10:07:43|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/10:07:43|PID hash table entries: 512 (order: 9, 2048 bytes)
20/10:07:43|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/10:07:43|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/10:07:43|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/10:07:43|Kernel virtual memory layout:
20/10:07:43|  * 0xffffe000..0xfffff000  : fixmap
20/10:07:43|  * 0xfde00000..0xfe000000  : consistent mem
20/10:07:43|  * 0xfde00000..0xfde00000  : early ioremap
20/10:07:43|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/10:07:43|NR_IRQS:512
20/10:07:43|clocksource: timebase mult[a00000] shift[22] registered
20/10:07:43|Console: colour dummy device 80x25
20/10:07:43|Mount-cache hash table entries: 512
20/10:07:43|NET: Registered protocol family 16
20/10:07:43|PCI: Probing PCI hardware
20/10:07:43|bio: create slab <bio-0> at 0
20/10:07:44|NET: Registered protocol family 2
20/10:07:44|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/10:07:44|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/10:07:44|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/10:07:44|TCP: Hash tables configured (established 4096 bind 4096)
20/10:07:44|TCP reno registered
20/10:07:44|NET: Registered protocol family 1
20/10:07:44|ROMFS MTD (C) 2007 Red Hat, Inc.
20/10:07:44|msgmni has been set to 214
20/10:07:44|io scheduler noop registered
20/10:07:44|io scheduler anticipatory registered
20/10:07:44|io scheduler deadline registered
20/10:07:44|io scheduler cfq registered (default)
20/10:07:44|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/10:07:44|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/10:07:44|console [ttyS0] enabled
20/10:07:44|brd: module loaded
20/10:07:44|loop: module loaded
20/10:07:44|Device Tree Probing 'ethernet'
20/10:07:44|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:d7:cc
20/10:07:44|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/10:07:44|XLlTemac: DCR address: 0x80
20/10:07:44|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/10:07:45|XLlTemac: Allocating DMA descriptors with kmalloc
20/10:07:45|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/10:07:45|XTemac: PHY detected at address 7.
20/10:07:45|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/10:07:45|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45|Using buffer write method
20/10:07:45|cfi_cmdset_0001: Erase suspend on write enabled
20/10:07:45|cmdlinepart partition parsing not available
20/10:07:45|RedBoot partition parsing not available
20/10:07:45|Creating 5 MTD partitions on "fc000000.flash":
20/10:07:45|0x000000000000-0x000000500000 : "golden_firmware"
20/10:07:45|0x000000500000-0x000000800000 : "golden_kernel"
20/10:07:45|0x000000800000-0x000000d00000 : "user_firmware"
20/10:07:46|0x000000d00000-0x000000fe0000 : "user_kernel"
20/10:07:46|0x000000fe0000-0x000001000000 : "env_variables"
20/10:07:46|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/10:07:46|SPI: XIlinx spi: bus number now 32766
20/10:07:46|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/10:07:46|SPI: XIlinx spi: bus number now 32765
20/10:07:46|mice: PS/2 mouse device common for all mice
20/10:07:46|Device Tree Probing 'i2c'
20/10:07:46| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/10:07:46|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/10:07:46|TCP cubic registered
20/10:07:46|NET: Registered protocol family 17
20/10:07:46|RPC: Registered udp transport module.
20/10:07:46|RPC: Registered tcp transport module.
20/10:07:46|eth0: XLlTemac: Options: 0x3fa
20/10:07:47|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/10:07:47|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/10:07:47|eth0: XLlTemac: speed set to 1000Mb/s
20/10:07:49|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/10:07:49|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/10:07:49|Sending DHCP requests ., OK
20/10:07:50|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.5
20/10:07:50|IP-Config: Complete:
20/10:07:50|     device=eth0, addr=192.168.11.5, mask=255.255.255.0, gw=255.255.255.255,
20/10:07:50|     host=aida05, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/10:07:50|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05
20/10:07:50|Looking up port of RPC 100003/2 on 192.168.11.99
20/10:07:51|Looking up port of RPC 100005/1 on 192.168.11.99
20/10:07:51|VFS: Mounted root (nfs filesystem) on device 0:12.
20/10:07:51|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/10:07:51|Starting sysinit...
20/10:07:51|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/10:07:52|            Press 'I' to enter interactive startup.
20/10:07:52|Setting clock  (utc): Tue Apr 20 10:07:52 BST 2021 [  OK  ]
20/10:07:53|Building the cache [  OK  ]
20/10:07:53|Setting hostname aida05:  [  OK  ]
20/10:07:54|Mounting local filesystems:  [  OK  ]
20/10:07:55|Enabling /etc/fstab swaps:  [  OK  ]
20/10:07:57|Finishing sysinit...
INIT: Entering runlevel: 3
20/10:08:00|Entering non-interactive startup
20/10:08:01|FATAL: Module ipv6 not found.
20/10:08:02|Bringing up loopback interface:  [  OK  ]
20/10:08:04|FATAL: Module ipv6 not found.
20/10:08:04|Starting system logger: [  OK  ]
20/10:08:05|Starting kernel logger: [  OK  ]
20/10:08:05|Starting rpcbind: [  OK  ]
20/10:08:06|Mounting NFS filesystems:  [  OK  ]
20/10:08:07|Mounting other filesystems:  [  OK  ]
20/10:08:07|Starting xinetd: [  OK  ]
20/10:08:08|Starting midas:  Starting MIDAS Data Acquisition for aida05
20/10:08:08|xaida: device parameters: base=0x81000000 size=0x200000
20/10:08:14|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/10:08:14|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/10:08:14|xaida: driver assigned major number 254
20/10:08:14|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/10:08:20|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/10:08:20|AIDAMEM: aidamem: driver assigned major number 253
20/10:08:20|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/10:08:26|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/10:08:27|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/10:08:27|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/10:08:27|Computer Name = aida05; Temp Directory = /tmp/tcl356
20/10:08:30|package limit is not available: can't find package limit
20/10:08:33|Running with default file descriptor limit
20/10:08:33|package setuid is not available: can't find package setuid
20/10:08:35|Running as user 0 group 0
20/10:08:35|
20/10:08:36|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
Entry  Sat Apr 24 15:48:38 2021, TD, aida12 crash - 08:33:03 Monday 19 April  
from ttyUSB5 system console log

19/07:25:59|page:c047b060 flags:(null) count:-1 mapcount:0 mapping:(null) index:0
19/07:25:59|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:00|Faulting instruction address: 0xc006e814
19/07:26:00|Oops: Kernel access of bad area, sig: 11 [#1]
19/07:26:00|PREEMPT Xilinx Virtex440
19/07:26:00|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:00|NIP: c006e814 LR: c006e78c CTR: c026327c
19/07:26:00|REGS: c60f3860 TRAP: 0300   Not tainted  (2.6.31)
19/07:26:00|MSR: 00021000 <ME,CE>  CR: 28222024  XER: 20000000
19/07:26:00|DEAR: 00100104, ESR: 00800000
19/07:26:00|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:00|GPR00: 00000000 c60f3910 c6b0a0a0 c038c3a8 00000000 000001a5 c047b078 00000041
19/07:26:00|GPR08: 00000001 00100100 00000004 c038c3a8 48282028 2015881c 00000001 00000000
19/07:26:00|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c60f2000
19/07:26:00|GPR24: 00000000 00000001 00029000 c038c384 00000001 c047b060 c038c39c c038c394
19/07:26:00|NIP [c006e814] get_page_from_freelist+0x188/0x4cc
19/07:26:00|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:00|Call Trace:
19/07:26:00|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:00|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:01|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:01|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:01|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:01|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:01|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:01|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:01|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:01|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:01|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:01|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:01|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:01|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:01|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:01|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:01|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:01|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:01|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:01|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:01|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:01|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:02|Instruction dump:
19/07:26:02|2f890000 419e0008 7c004a2c 38dd0018 7f861800 409effd8 7f861800 419e02e0
19/07:26:02|813d0018 81660004 7f3ccb78 912b0000 <91690004> 92dd0018 92a60004 813e0000
19/07:26:02|---[ end trace 87f7fdd1c04ff304 ]---
19/07:26:02|note: syslogd[224] exited with preempt_count 1
19/07:26:02|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:02|Faulting instruction address: 0xc006e388
19/07:26:02|Oops: Kernel access of bad area, sig: 11 [#2]
19/07:26:02|PREEMPT Xilinx Virtex440
19/07:26:02|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:02|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:02|REGS: c60f35d0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:02|MSR: 00021000 <ME,CE>  CR: 28222082  XER: 00000000
19/07:26:02|DEAR: 00100104, ESR: 00800000
19/07:26:02|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:02|GPR00: 00000000 c60f3680 c6b0a0a0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:02|GPR08: 00000008 c03c7738 00000006 00100100 00000000 2015881c c69e63f8 1fd84000
19/07:26:02|GPR16: c0390000 c60f2000 c69e63f8 c6ba2128 c6178740 c036f924 ffffffff 00000000
19/07:26:03|GPR24: 1fd84000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7720
19/07:26:03|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:03|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:03|Call Trace:
19/07:26:03|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:03|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:03|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:03|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:03|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:03|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:03|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:03|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:03|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:03|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:03|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:03|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:03|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:03|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:03|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:04|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:04|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:04|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:04|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:04|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:04|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:04|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:04|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:04|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:04|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:04|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:04|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:04|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:04|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:04|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:04|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:04|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:04|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:04|Instruction dump:
19/07:26:04|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:05|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:05|---[ end trace 87f7fdd1c04ff305 ]---
19/07:26:05|Fixing recursive fault but reboot is needed!
19/07:26:05|BUG: scheduling while atomic: syslogd/224/0x00000006
19/07:26:05|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:05|Call Trace:
19/07:26:05|[c60f34c0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:05|[c60f3500] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:05|[c60f3510] [c02a8020] schedule+0x2f4/0x34c
19/07:26:05|[c60f3550] [c00382a8] do_exit+0x544/0x61c
19/07:26:05|[c60f3590] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:05|[c60f35b0] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:05|[c60f35c0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:05|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:05|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:05|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:05|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:05|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:05|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:05|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:06|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:06|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:06|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:06|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:06|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:06|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:06|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:06|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:06|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:06|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:06|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:06|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:06|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:06|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:06|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:06|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:06|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:06|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:06|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:06|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:07|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:07|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:07|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:07|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:07|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:07|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:07|Unable to handle kernel paging request for data at address 0x00100100
19/07:26:07|Faulting instruction address: 0xc006e9e8
19/07:26:07|Oops: Kernel access of bad area, sig: 11 [#3]
19/07:26:07|PREEMPT Xilinx Virtex440
19/07:26:07|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:07|NIP: c006e9e8 LR: c006e78c CTR: c0285b38
19/07:26:07|REGS: c61adab0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:07|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:07|DEAR: 00100100, ESR: 00000000
19/07:26:07|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:07|GPR00: 00000010 c61adb60 c6b0a4c0 c038c3a8 00000000 000001a5 00000000 00000041
19/07:26:07|GPR08: 00000001 00100100 00000004 c61ac000 22024048 2079d364 00000001 00000000
19/07:26:08|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c61ac000
19/07:26:08|GPR24: 00000000 00000001 00029000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:26:08|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:08|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:08|Call Trace:
19/07:26:08|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:08|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:08|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:08|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:08|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:08|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:08|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:08|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:08|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:08|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:08|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:08|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:08|Instruction dump:
19/07:26:08|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:09|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:09|---[ end trace 87f7fdd1c04ff306 ]---
19/07:26:09|note: klogd[227] exited with preempt_count 1
19/07:26:09|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:09|Faulting instruction address: 0xc006e388
19/07:26:09|Oops: Kernel access of bad area, sig: 11 [#4]
19/07:26:09|PREEMPT Xilinx Virtex440
19/07:26:09|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:09|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:09|REGS: c61ad820 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:09|MSR: 00021000 <ME,CE>  CR: 22224082  XER: 00000000
19/07:26:09|DEAR: 00100104, ESR: 00800000
19/07:26:09|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:09|GPR00: 00000000 c61ad8d0 c6b0a4c0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:09|GPR08: 00000008 c03c7e98 00000006 00100100 00000000 2079d364 c6a5640c 20739000
19/07:26:09|GPR16: c0390000 c61ac000 c6a5640c c60a6320 c6178c80 c036f924 ffffffff 00000000
19/07:26:09|GPR24: 20739000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7e80
19/07:26:10|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:10|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:10|Call Trace:
19/07:26:10|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:10|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:10|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:10|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:10|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:10|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:10|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:10|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:10|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:10|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:10|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:10|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:10|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:10|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:10|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:10|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:10|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:11|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:11|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:11|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:11|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:11|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:11|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:11|Instruction dump:
19/07:26:11|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:11|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:11|---[ end trace 87f7fdd1c04ff307 ]---
19/07:26:11|Fixing recursive fault but reboot is needed!
19/07:26:11|BUG: scheduling while atomic: klogd/227/0x00000006
19/07:26:11|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:11|Call Trace:
19/07:26:11|[c61ad710] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:11|[c61ad750] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:11|[c61ad760] [c02a8020] schedule+0x2f4/0x34c
19/07:26:11|[c61ad7a0] [c00382a8] do_exit+0x544/0x61c
19/07:26:11|[c61ad7e0] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:11|[c61ad800] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61ad810] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:12|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:12|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:12|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:12|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:12|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:12|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:12|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:12|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:12|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:12|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:12|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:12|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:12|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:12|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:12|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:13|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:13|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:13|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:13|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:13|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:13|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:13|Faulting instruction address: 0xc006e9e8
19/07:26:13|Oops: Kernel access of bad area, sig: 11 [#5]
19/07:26:13|PREEMPT Xilinx Virtex440
19/07:26:13|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:13|NIP: c006e9e8 LR: c006e78c CTR: c006b2c4
19/07:26:13|REGS: c6349900 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:13|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:13|DEAR: 00000000, ESR: 00000000
19/07:26:13|TASK = c62fa500[379] 'AidaExecV9' THREAD: c6348000
19/07:26:13|GPR00: 00000009 c63499b0 c62fa500 c038c3a8 00000000 000001a5 00100100 00000041
19/07:26:13|GPR08: 00000001 00000000 00000004 c6348000 22022048 1005520c 00000001 00000000
19/07:26:13|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6348000
19/07:26:14|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 ffffffe8 c038c39c c038c394
19/07:26:14|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:14|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:14|Call Trace:
19/07:26:14|[c63499b0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:14|[c6349a40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:14|[c6349ac0] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:14|[c6349b20] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:26:14|[c6349b40] [c006b2e0] mempool_alloc_slab+0x1c/0x2c
19/07:26:14|[c6349b50] [c006b5d8] mempool_alloc+0x5c/0x13c
19/07:26:14|[c6349ba0] [c0294600] rpc_malloc+0x34/0x9c
19/07:26:14|[c6349bc0] [c028da44] call_allocate+0xc4/0x194
19/07:26:14|[c6349be0] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:14|[c6349c20] [c028d120] rpc_run_task+0x44/0x90
19/07:26:14|[c6349c30] [c028d234] rpc_call_sync+0x54/0x84
19/07:26:14|[c6349c60] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
19/07:26:14|[c6349c90] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
19/07:26:14|[c6349d50] [c0129b3c] nfs_permission+0x178/0x1c4
19/07:26:14|[c6349d70] [c00a0bf4] inode_permission+0x94/0xbc
19/07:26:15|[c6349d80] [c00a2bc0] __link_path_walk+0xb0/0xf7c
19/07:26:15|[c6349df0] [c00a3ad4] path_walk+0x48/0xc8
19/07:26:15|[c6349e20] [c00a3c44] do_path_lookup+0x64/0x6c
19/07:26:15|[c6349e40] [c00a4b14] do_filp_open+0xc8/0x914
19/07:26:15|[c6349f10] [c0095314] do_sys_open+0x64/0x130
19/07:26:15|[c6349f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:15|Instruction dump:
19/07:26:15|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:15|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:15|---[ end trace 87f7fdd1c04ff308 ]---
19/07:26:15|note: AidaExecV9[379] exited with preempt_count 1
19/07:26:15|Call Trace:
19/07:26:15|[c6311b50] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:15|[c6311b90] [c006da9c] bad_page+0x94/0x12c
19/07:26:15|[c6311bb0] [c006eadc] get_page_from_freelist+0x450/0x4cc
19/07:26:15|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:15|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:15|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:15|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:15|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:16|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:16|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:16|Faulting instruction address: 0xc006e9e8
19/07:26:16|Oops: Kernel access of bad area, sig: 11 [#6]
19/07:26:16|PREEMPT Xilinx Virtex440
19/07:26:16|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:16|NIP: c006e9e8 LR: c006e78c CTR: c01c13cc
19/07:26:16|REGS: c6311b00 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:16|MSR: 00021000 <ME,CE>  CR: 24222024  XER: 00000000
19/07:26:16|DEAR: 00000000, ESR: 00000000
19/07:26:16|TASK = c62fb9a0[375] 'AidaExecV9' THREAD: c6310000
19/07:26:16|GPR00: 00000009 c6311bb0 c62fb9a0 c038c3a8 c62fb9a0 00000010 00100100 0000594b
19/07:26:16|GPR08: 000000a0 00000000 00000000 c6310000 24222042 1005520c 00000001 00000000
19/07:26:16|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6310000
19/07:26:16|GPR24: 00000000 00000001 00029000 c038c384 00000001 ffffffe8 c038c39c c038c394
19/07:26:16|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:16|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:17|Call Trace:
19/07:26:17|[c6311bb0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:17|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:17|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:17|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:17|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:17|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:17|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:17|Instruction dump:
19/07:26:17|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:17|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:17|---[ end trace 87f7fdd1c04ff309 ]---
19/07:26:17|note: AidaExecV9[375] exited with preempt_count 1
19/07:26:17|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:18|Faulting instruction address: 0xc006e388
19/07:26:18|Oops: Kernel access of bad area, sig: 11 [#7]
19/07:26:18|PREEMPT Xilinx Virtex440
19/07:26:18|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:18|NIP: c006e388 LR: c006e348 CTR: 00000000
19/07:26:18|REGS: c6835e10 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:18|MSR: 00021000 <ME,CE>  CR: 24000082  XER: 00000000
19/07:26:18|DEAR: 00100104, ESR: 00800000
19/07:26:18|TASK = c6824880[5] 'events/0' THREAD: c6834000
19/07:26:18|GPR00: 00000000 c6835ec0 c6824880 00000000 00000003 c038c3a8 ffffffff c0498000
19/07:26:18|GPR08: 00000008 c0419a78 00000027 00100100 002515ba 00005aa8 bfffffff 5fffffff
19/07:26:18|GPR16: ffff7fd7 fffeffaf 00120110 7ffefd7f c031aef4 c02fb040 c02fb008 00000002
19/07:26:18|GPR24: c6834000 00000000 c038c384 c038c384 00000000 00029000 c038c39c c0419a60
19/07:26:18|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:18|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:18|Call Trace:
19/07:26:18|[c6835ec0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:18|[c6835ef0] [c00915c4] kmem_freepages+0xd8/0x134
19/07:26:19|[c6835f00] [c0091ab8] slab_destroy+0x34/0x8c
19/07:26:19|[c6835f10] [c0091e20] drain_freelist+0x9c/0x110
19/07:26:19|[c6835f40] [c0093158] cache_reap+0x104/0x138
19/07:26:19|[c6835f60] [c0048908] worker_thread+0x140/0x204
19/07:26:19|[c6835fc0] [c004cfc8] kthread+0x78/0x7c
19/07:26:19|[c6835ff0] [c000e140] kernel_thread+0x4c/0x68
19/07:26:19|Instruction dump:
19/07:26:19|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:19|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:19|---[ end trace 87f7fdd1c04ff30a ]---
19/07:26:19|note: events/0[5] exited with preempt_count 1
19/07:26:19|Unable to handle kernel paging request for data at address 0x00100100
19/07:29:19|Faulting instruction address: 0xc006e9e8
19/07:29:19|Oops: Kernel access of bad area, sig: 11 [#8]
19/07:29:19|PREEMPT Xilinx Virtex440
19/07:29:19|Modules linked in: aidamem xdriver xh_spidev_register
19/07:29:19|NIP: c006e9e8 LR: c006e78c CTR: c025c424
19/07:29:19|REGS: c0391a60 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:29:19|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:29:19|DEAR: 00100100, ESR: 00000000
19/07:29:19|TASK = c036e318[0] 'swapper' THREAD: c0390000
19/07:29:19|GPR00: 00000010 c0391b10 c036e318 c038c3a8 00000000 000001a5 00000000 00000041
19/07:29:19|GPR08: 00000001 00100100 00000004 c0390000 22022048 00005aa8 00000001 00000000
19/07:29:19|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c0390000
19/07:29:19|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:29:19|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:29:19|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:19|Call Trace:
19/07:29:19|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:29:20|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:20|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:20|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:20|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:20|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:20|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:20|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:20|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:20|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:20|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:20|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:20|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:20|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:20|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:20|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:20|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:20|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:20|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:20|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:21|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:21|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:21|Instruction dump:
19/07:29:21|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:29:21|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:29:21|Kernel panic - not syncing: Fatal exception in interrupt
19/07:29:21|Call Trace:
19/07:29:21|[c0391990] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:29:21|[c03919d0] [c00345bc] panic+0x94/0x168
19/07:29:21|[c0391a20] [c000bd44] die+0x178/0x18c
19/07:29:21|[c0391a40] [c001191c] bad_page_fault+0x90/0xd8
19/07:29:21|[c0391a50] [c000e834] handle_page_fault+0x7c/0x80
19/07:29:21|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:21|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:21|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:21|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:21|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:21|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:21|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:22|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:22|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:22|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:22|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:22|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:22|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:22|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:22|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:22|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:22|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:22|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:22|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:22|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:22|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:22|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:22|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
19/07:32:23|
Found 0 errors checking kernel image
19/07:32:24|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
19/07:32:24|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/07:32:24|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/07:32:24|SPIconfig done now to set up the LMK3200 registers
19/07:32:24|LMK #0 : regInit[0]=0x80000000
19/07:32:24|LMK #0 : regInit[1]=0x10070600
19/07:32:24|LMK #0 : regInit[2]=0x60601
19/07:32:24|LMK #0 : regInit[3]=0x60602
19/07:32:25|LMK #0 : regInit[4]=0x60603
19/07:32:25|LMK #0 : regInit[5]=0x70624
19/07:32:25|LMK #0 : regInit[6]=0x70605
19/07:32:25|LMK #0 : regInit[7]=0x70606
19/07:32:25|LMK #0 : regInit[8]=0x70627
19/07:32:25|LMK #0 : regInit[9]=0x10000908
19/07:32:25|LMK #0 : regInit[10]=0xA0022A09
19/07:32:25|LMK #0 : regInit[11]=0x82800B
19/07:32:25|LMK #0 : regInit[12]=0x28C800D
19/07:32:25|LMK #0 : regInit[13]=0x830020E
19/07:32:25|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/07:32:25|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
19/07:32:25|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/07:32:25|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
19/07:32:26|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/07:32:29|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xcf 0xac
19/07:32:35|Using Xilinx Virtex440 machine description
19/07:32:36|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/07:32:36|Zone PFN ranges:
19/07:32:36|  DMA      0x00000000 -> 0x00007000
19/07:32:36|  Normal   0x00007000 -> 0x00007000
19/07:32:36|Movable zone start PFN for each node
19/07:32:36|early_node_map[1] active PFN ranges
19/07:32:36|    0: 0x00000000 -> 0x00007000
19/07:32:36|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/07:32:36|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/07:32:36|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/07:32:36|PID hash table entries: 512 (order: 9, 2048 bytes)
19/07:32:36|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/07:32:37|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/07:32:37|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/07:32:37|Kernel virtual memory layout:
19/07:32:37|  * 0xffffe000..0xfffff000  : fixmap
19/07:32:37|  * 0xfde00000..0xfe000000  : consistent mem
19/07:32:37|  * 0xfde00000..0xfde00000  : early ioremap
19/07:32:37|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/07:32:37|NR_IRQS:512
19/07:32:37|clocksource: timebase mult[a00000] shift[22] registered
19/07:32:37|Console: colour dummy device 80x25
19/07:32:37|Mount-cache hash table entries: 512
19/07:32:37|NET: Registered protocol family 16
19/07:32:37|PCI: Probing PCI hardware
19/07:32:37|bio: create slab <bio-0> at 0
19/07:32:37|NET: Registered protocol family 2
19/07:32:37|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/07:32:37|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/07:32:37|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/07:32:37|TCP: Hash tables configured (established 4096 bind 4096)
19/07:32:37|TCP reno registered
19/07:32:37|NET: Registered protocol family 1
19/07:32:37|ROMFS MTD (C) 2007 Red Hat, Inc.
19/07:32:38|msgmni has been set to 214
19/07:32:38|io scheduler noop registered
19/07:32:38|io scheduler anticipatory registered
19/07:32:38|io scheduler deadline registered
19/07:32:38|io scheduler cfq registered (default)
19/07:32:38|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/07:32:38|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/07:32:38|console [ttyS0] enabled
19/07:32:38|brd: module loaded
19/07:32:38|loop: module loaded
19/07:32:38|Device Tree Probing 'ethernet'
19/07:32:38|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:cf:ac
19/07:32:38|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/07:32:38|XLlTemac: DCR address: 0x80
19/07:32:38|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/07:32:38|XLlTemac: Allocating DMA descriptors with kmalloc
19/07:32:38|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/07:32:38|XTemac: PHY detected at address 7.
19/07:32:38|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/07:32:38|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39|Using buffer write method
19/07:32:39|cfi_cmdset_0001: Erase suspend on write enabled
19/07:32:39|cmdlinepart partition parsing not available
19/07:32:39|RedBoot partition parsing not available
19/07:32:39|Creating 5 MTD partitions on "fc000000.flash":
19/07:32:39|0x000000000000-0x000000500000 : "golden_firmware"
19/07:32:39|0x000000500000-0x000000800000 : "golden_kernel"
19/07:32:39|0x000000800000-0x000000d00000 : "user_firmware"
19/07:32:39|0x000000d00000-0x000000fe0000 : "user_kernel"
19/07:32:39|0x000000fe0000-0x000001000000 : "env_variables"
19/07:32:39|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/07:32:39|SPI: XIlinx spi: bus number now 32766
19/07:32:39|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/07:32:39|SPI: XIlinx spi: bus number now 32765
19/07:32:40|mice: PS/2 mouse device common for all mice
19/07:32:40|Device Tree Probing 'i2c'
19/07:32:40| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/07:32:40|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/07:32:40|TCP cubic registered
19/07:32:40|NET: Registered protocol family 17
19/07:32:40|RPC: Registered udp transport module.
19/07:32:40|RPC: Registered tcp transport module.
19/07:32:40|eth0: XLlTemac: Options: 0x3fa
19/07:32:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/07:32:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/07:32:40|eth0: XLlTemac: speed set to 1000Mb/s
19/07:32:43|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/07:32:43|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/07:32:43|Sending DHCP requests ., OK
19/07:32:44|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.12
19/07:32:44|IP-Config: Complete:
19/07:32:44|     device=eth0, addr=192.168.11.12, mask=255.255.255.0, gw=255.255.255.255,
19/07:32:44|     host=aida12, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/07:32:44|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12
19/07:32:44|Looking up port of RPC 100003/2 on 192.168.11.99
19/07:32:44|Looking up port of RPC 100005/1 on 192.168.11.99
19/07:32:44|VFS: Mounted root (nfs filesystem) on device 0:12.
19/07:32:44|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/07:32:45|Starting sysinit...
19/07:32:45|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/07:32:46|            Press 'I' to enter interactive startup.
19/07:32:46|Setting clock  (utc): Mon Apr 19 07:32:47 BST 2021 [  OK  ]
19/07:32:47|Building the cache [  OK  ]
19/07:32:47|Setting hostname aida12:  [  OK  ]
19/07:32:48|Mounting local filesystems:  [  OK  ]
19/07:32:50|Enabling /etc/fstab swaps:  [  OK  ]
19/07:32:53|Finishing sysinit...
INIT: Entering runlevel: 3
19/07:32:56|Entering non-interactive startup
19/07:32:56|FATAL: Module ipv6 not found.
19/07:32:57|Bringing up loopback interface:  [  OK  ]
19/07:32:59|FATAL: Module ipv6 not found.
19/07:33:00|Starting system logger: [  OK  ]
19/07:33:01|Starting kernel logger: [  OK  ]
19/07:33:01|Starting rpcbind: [  OK  ]
19/07:33:03|Mounting NFS filesystems:  [  OK  ]
19/07:33:03|Mounting other filesystems:  [  OK  ]
19/07:33:04|Starting xinetd: [  OK  ]
19/07:33:05|Starting midas:  Starting MIDAS Data Acquisition for aida12
19/07:33:05|xaida: device parameters: base=0x81000000 size=0x200000
19/07:33:12|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/07:33:12|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/07:33:12|xaida: driver assigned major number 254
19/07:33:12|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/07:33:17|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/07:33:17|AIDAMEM: aidamem: driver assigned major number 253
19/07:33:17|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/07:33:24|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/07:33:24|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/07:33:24|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/07:33:24|Computer Name = aida12; Temp Directory = /tmp/tcl361
19/07:33:31|
19/07:33:34|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
Entry  Sun Apr 25 10:22:16 2021, TD, Sunday 25 April 9x
11.25 All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0xc95f : 	 0xc960 : 	 1  
aida02 fault 	 0x8bbe : 	 0x8bbf : 	 1  
aida03 fault 	 0xae9d : 	 0xae9e : 	 1  
aida04 fault 	 0x6e2 : 	 0x6e3 : 	 1  
aida05 fault 	 0x31a2 : 	 0x31a4 : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22ea : 	 3  
aida07 fault 	 0x438d : 	 0x438f : 	 2  
aida08 fault 	 0x61eb : 	 0x61ed : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     55     21     31      6      3      2      2      3      2      3      4   : 28148
aida02 :     28      3      5      4      0      3      1      3      3      3      6   : 36312
aida03 :     25     10      2      2      2      4      1      3      3      3      6   : 36500
aida04 :     14      7     10      1      2      2      2      3      3      3      6   : 36528
aida05 :     58     35     28      6      4      2      2      3      3      3      4   : 29312
aida06 :     16     11      6      1      0      4      1      3      3      3      6   : 36376
aida07 :     75     31     25      9      4      3      2      2      2      3      4   : 27988
aida08 :     28      7      1      1      1      4      1      3      3      3      6   : 36376
aida09 :     50     48     28     10      3      3      3      4      2      3      4   : 29320
aida10 :     26      5      4      3      1      3      2      4      2      3      6   : 36080
aida11 :     21      7      5      3      5      2      2      4      2      3      6   : 36220
aida12 :     64     29     26      7      3      1      2      3      2      3      4   : 28072

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors reported - server terminal now reset & clear'd

MERGE Data Link (4408): bad timestamp  10 3 0xc2b67d8f 0x0b50c3e2 0x0000f437cb50c3e2 0x1678f437cb50c3e2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2877f27 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a07fcd 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77c9c 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc28b7d9c 0x0b50d382 0x0000f437cb50d382 0x1678f437cb50d382 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2937e6c 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a17d78 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2817fff 0x0b50eaf2 0x0000f437cb50eaf2 0x1678f437cb50eaf2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77ecf 0x06fb3d52 0x0000f437cb50eaf2 0x1678f437c6fb3d52 0x1678f43a16f1e6b2

11.25 Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 1
      DSSSD bias |& leakage currents Ok - attachment 2
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      merger time seq errors - attachment 5

      All histograms, stats and merge stats zero'd
      WR decoder errors *new*  baseline

11.45 1.8.L spectra - attachments 6 & 7
      peak width FWHM aida05 65 ch, aida06 95 ch
      1.8.W spectra = attachments 8 & 9
Entry  Mon Apr 26 09:41:21 2021, TD, Monday 26 April 302.pngScreenshot_2021-04-26_Temperature_and_status_scan_aidas-gsi.pngScreenshot_2021-04-26_Statistics_aidas-gsi.pngScreenshot_2021-04-26_NewMerger_Statistics_aidas-gsi.pngScreenshot_2021-04-26_AIDA_Alerting_-_Grafana.png
10.42 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc962 : 	 2  
aida02 fault 	 0x8bbf : 	 0x8bc1 : 	 2  
aida03 fault 	 0xae9e : 	 0xaea0 : 	 2  
aida04 fault 	 0x6e3 : 	 0x6e5 : 	 2  
aida05 fault 	 0x31a4 : 	 0x31ab : 	 7  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f1 : 	 7  
aida07 fault 	 0x438f : 	 0x4396 : 	 7  
aida08 fault 	 0x61ed : 	 0x61f4 : 	 7  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25      8     28      8      3      2      2      3      2      3      4   : 27940
aida02 :     24      9      3      5      1      3      1      3      3      3      6   : 36408
aida03 :     24      6      5      4      3      3      1      3      3      3      6   : 36512
aida04 :     23      6      6      1      3      3      1      3      3      3      6   : 36428
aida05 :     15     21     27      7      4      2      2      3      3      3      4   : 29044
aida06 :     24      8      2      4      1      3      1      3      3      3      6   : 36352
aida07 :     29     24     28      9      3      3      2      2      2      3      4   : 27732
aida08 :     17     10      1      1      1      4      1      3      3      3      6   : 36356
aida09 :     41     49     18     14      3      4      3      4      2      3      4   : 29388
aida10 :      3      1      6      4      2      3      3      3      2      3      6   : 35828
aida11 :     22      7      5      4      2      1      2      4      2      3      6   : 35936
aida12 :     65     30     23      9      3      1      2      3      2      3      4   : 28100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021


10.47 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK - no data storage
Entry  Tue Apr 27 08:26:01 2021, TD, Tuesday 27 April - aida09 system console reports 'received undefined information code: 1' and 'code: 9' 6x
09.25 All system wide checks OK *except*


		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc963 : 	 3  
aida02 fault 	 0x8bbf : 	 0x8bc2 : 	 3  
aida03 fault 	 0xae9e : 	 0xaea1 : 	 3  
aida04 fault 	 0x6e3 : 	 0x6e6 : 	 3  
aida05 fault 	 0x31a4 : 	 0x31ac : 	 8  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f2 : 	 8  
aida07 fault 	 0x438f : 	 0x4397 : 	 8  
aida08 fault 	 0x61ed : 	 0x61f5 : 	 8  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     18     24     16      3      1      3      2      2      3      4   : 27792
aida02 :     25      6      3      5      1      4      2      2      3      3      6   : 36260
aida03 :     19     10      3      6      3      3      1      3      3      3      6   : 36556
aida04 :     21      7      3      4      2      4      2      2      3      3      6   : 36284
aida05 :     40     37     16     13      4      3      2      2      3      3      4   : 28904
aida06 :     32      4      4      3      0      4      1      3      3      3      6   : 36416
aida07 :     26     22     18     13      3      3      2      4      3      4      3   : 27672
aida08 :     15      9      4      1      2      3      1      3      3      3      6   : 36324
aida09 :     29     41     11     19      3      4      3      4      3      3      4   : 30348
aida10 :     17      5      8      4      3      3      3      3      2      3      6   : 36012
aida11 :     25      9      3      5      1      1      2      4      2      3      6   : 35900
aida12 :     80     43     12      7      4      2      3      2      2      3      4   : 27960



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

09.25 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
       aida09 zero rate
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK *except* aida09 - no data storage

10.00 from aida09 system console

24/03:04:39|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
24/03:04:39|Also in RDOGo_Operate: Enabled waveform readout
24/03:04:39|
24/03:04:39|get_WAVEBlk (C) : which = 0 , Status = 00001031 , Count = 0
24/03:04:39|get_WAVEBlk (A) which = 0 ---- status = 0540, State machines = 050A
24/03:04:39|WV:0:Buffer info. Length = 131072 : Offset = 0 
24/03:04:40|WV:1:Buffer info. Length = 131070 : Offset = 2 
24/03:04:40|WV:2:Buffer info. Length = 131068 : Offset = 4 
24/03:04:40|WV:3:Buffer info. Length = 131066 : Offset = 6 
24/03:04:40|WV:4:Buffer info. Length = 131064 : Offset = 8 
24/03:04:40|WV:5:Buffer info. Length = 131062 : Offset = 10 
24/03:04:40|WV:6:Buffer info. Length = 131060 : Offset = 12 
24/03:04:40|WV:7:Buffer info. Length = 131058 : Offset = 14 
24/03:04:40|WV:8:Buffer info. Length = 131056 : Offset = 16 
24/03:04:40|WV:9:Buffer info. Length = 131054 : Offset = 18 
24/03:04:40|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:40|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 804441A5 : 1 => 09CD0000 
24/03:04:40|Last Data : Databuffer  274139128 : 0 => C01B7F88 : 1 => 0C05B135 
24/03:04:40| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:40| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:40| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:41| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:41| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:41| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:41| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:41| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:41|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C03A7F13 : 1 => 0C05B455 
24/03:04:41|Last Data : Databuffer  274139128 : 0 => C0127D9D : 1 => 0DB9DEED 
24/03:04:41| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:41| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:41| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:42| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:42| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:42| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:42| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:42| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:42|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:42|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C02D7E26 : 1 => 0DB9DEED 
24/03:04:42|Last Data : Databuffer  274139128 : 0 => C0217F73 : 1 => 0F816B3D 
24/03:04:42| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:42| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:42| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:43| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:43| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:43| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:43| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:43| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:43|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C02D7EB4 : 1 => 0F816D95 
24/03:04:43|Last Data : Databuffer  274139128 : 0 => C0167F53 : 1 => 0148F2DD 
24/03:04:43| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:43| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:43| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:44| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:44| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:44| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:44| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:44|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0207E17 : 1 => 0148F2DD 
24/03:04:44|Last Data : Databuffer  274139128 : 0 => C0177F46 : 1 => 047FF0CD 
24/03:04:44| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:44| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:44| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:45| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:45| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:45| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:45| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:47|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:56|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:05|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:05:15|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:24|197 spectra zeroed
25/10:21:10|do_GetState returned z=0 and 1
25/10:21:11|Data Acquisition Statistics counters now cleared
25/10:24:01|Clear Statistics (1)
25/10:24:01|bad event data (0x45000034 0x84e64000)
26/16:52:50|bad event data (0x40061e21 0xc0a80b63)
26/16:52:50|received undefined information code: 1
26/16:52:50|bad event data (0x0101080a 0xe9b68def)
26/16:52:50|bad event data (0x03527827 0x0101050a)
26/16:52:50|received undefined information code: 9

From system console

Apr 24 02:52:42 aida09 kernel: IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
Apr 24 02:52:42 aida09 kernel: IP-Config: Complete:
Apr 24 02:52:42 aida09 kernel:      device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
Apr 24 02:52:42 aida09 kernel:      host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
Apr 24 02:52:42 aida09 kernel:      bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100003/2 on 192.168.11.99
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100005/1 on 192.168.11.99
Apr 24 02:52:43 aida09 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Apr 24 02:52:43 aida09 kernel: Freeing unused kernel memory: 168k init
Apr 24 02:52:43 aida09 kernel: xaida: device parameters: base=0x81000000 size=0x200000
Apr 24 02:52:44 aida09 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Apr 24 02:52:44 aida09 kernel: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Apr 24 02:52:44 aida09 kernel: xaida: driver assigned major number 254
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service telnet, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service ftp, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Apr 24 02:52:44 aida09 xinetd[273]: Started working: 2 available services
Apr 24 02:52:48 aida09 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: driver assigned major number 253
Apr 24 02:53:04 aida09 kernel: xaida: open:
Apr 24 02:53:04 aida09 kernel: AIDAMEM: aidamem_open:
Apr 24 02:57:33 aida09 kernel: xaida: open:
Apr 24 13:39:36 aida09 xinetd[273]: START: telnet pid=396 from=192.168.11.99
Apr 24 13:42:05 aida09 xinetd[273]: EXIT: telnet status=0 pid=396 duration=149(sec)
Apr 27 09:01:50 aida09 xinetd[273]: START: telnet pid=422 from=192.168.11.99
ELOG V3.1.4-unknown