Some of the FEE64s aida01 .. aida12 panic during boot
Frequencies of panics for each FEE64 can be seen in attachment 1
Below is an example of an aida04 panic following a power cycle and an automatic reboot
pi@raspberrypi:~/logs $ ./tail_aida aida04
aida04
ttyUSB7
19/11:07:04|LR [203f5680] 0x203f5680
19/11:07:04|Call Trace:
19/11:07:04|Kernel panic - not syncing: Fatal exception in interrupt
19/11:07:04|Call Trace:
19/11:07:04|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/11:07:04|[c6941e20] [c00345bc] panic+0x94/0x168
19/11:07:04|[c6941e70] [c000bd44] die+0x178/0x18c
19/11:07:04|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
19/11:07:04|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
19/11:07:04|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
19/11:10:05|
Found 0 errors checking kernel image
19/11:10:06|VHDL version number 0X18430701
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
19/11:10:06|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/11:10:06|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/11:10:06|SPIconfig done now to set up the LMK3200 registers
19/11:10:06|LMK #0 : regInit[0]=0x80000000
19/11:10:07|LMK #0 : regInit[1]=0x10070600
19/11:10:07|LMK #0 : regInit[2]=0x60601
19/11:10:07|LMK #0 : regInit[3]=0x60602
19/11:10:07|LMK #0 : regInit[4]=0x60603
19/11:10:07|LMK #0 : regInit[5]=0x70624
19/11:10:07|LMK #0 : regInit[6]=0x70605
19/11:10:07|LMK #0 : regInit[7]=0x70606
19/11:10:07|LMK #0 : regInit[8]=0x70627
19/11:10:07|LMK #0 : regInit[9]=0x10000908
19/11:10:07|LMK #0 : regInit[10]=0xA0022A09
19/11:10:07|LMK #0 : regInit[11]=0x82800B
19/11:10:07|LMK #0 : regInit[12]=0x28C800D
19/11:10:07|LMK #0 : regInit[13]=0x830020E
19/11:10:07|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/11:10:07|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
19/11:10:07|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/11:10:08|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
19/11:10:08|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/11:10:11|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xb7
19/11:10:17|Using Xilinx Virtex440 machine description
19/11:10:18|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/11:10:18|Zone PFN ranges:
19/11:10:18| DMA 0x00000000 -> 0x00007000
19/11:10:18| Normal 0x00007000 -> 0x00007000
19/11:10:18|Movable zone start PFN for each node
19/11:10:18|early_node_map[1] active PFN ranges
19/11:10:18| 0: 0x00000000 -> 0x00007000
19/11:10:18|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/11:10:18|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448
19/11:10:18|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/11:10:18|PID hash table entries: 512 (order: 9, 2048 bytes)
19/11:10:19|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/11:10:19|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/11:10:19|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/11:10:19|Kernel virtual memory layout:
19/11:10:19| * 0xffffe000..0xfffff000 : fixmap
19/11:10:19| * 0xfde00000..0xfe000000 : consistent mem
19/11:10:19| * 0xfde00000..0xfde00000 : early ioremap
19/11:10:19| * 0xd1000000..0xfde00000 : vmalloc & ioremap
19/11:10:19|NR_IRQS:512
19/11:10:19|clocksource: timebase mult[a00000] shift[22] registered
19/11:10:19|Console: colour dummy device 80x25
19/11:10:19|Mount-cache hash table entries: 512
19/11:10:19|NET: Registered protocol family 16
19/11:10:19|PCI: Probing PCI hardware
19/11:10:19|bio: create slab <bio-0> at 0
19/11:10:19|NET: Registered protocol family 2
19/11:10:19|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/11:10:19|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/11:10:19|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/11:10:19|TCP: Hash tables configured (established 4096 bind 4096)
19/11:10:20|TCP reno registered
19/11:10:20|NET: Registered protocol family 1
19/11:10:20|ROMFS MTD (C) 2007 Red Hat, Inc.
19/11:10:20|msgmni has been set to 214
19/11:10:20|io scheduler noop registered
19/11:10:20|io scheduler anticipatory registered
19/11:10:20|io scheduler deadline registered
19/11:10:20|io scheduler cfq registered (default)
19/11:10:20|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/11:10:20|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/11:10:20|console [ttyS0] enabled
19/11:10:20|brd: module loaded
19/11:10:20|loop: module loaded
19/11:10:20|Device Tree Probing 'ethernet'
19/11:10:20|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:b7
19/11:10:20|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/11:10:20|XLlTemac: DCR address: 0x80
19/11:10:20|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/11:10:20|XLlTemac: Allocating DMA descriptors with kmalloc
19/11:10:20|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/11:10:20|XTemac: PHY detected at address 7.
19/11:10:20|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/11:10:21|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21|Using buffer write method
19/11:10:21|cfi_cmdset_0001: Erase suspend on write enabled
19/11:10:21|cmdlinepart partition parsing not available
19/11:10:21|RedBoot partition parsing not available
19/11:10:21|Creating 5 MTD partitions on "fc000000.flash":
19/11:10:21|0x000000000000-0x000000500000 : "golden_firmware"
19/11:10:21|0x000000500000-0x000000800000 : "golden_kernel"
19/11:10:21|0x000000800000-0x000000d00000 : "user_firmware"
19/11:10:21|0x000000d00000-0x000000fe0000 : "user_kernel"
19/11:10:21|0x000000fe0000-0x000001000000 : "env_variables"
19/11:10:21|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/11:10:21|SPI: XIlinx spi: bus number now 32766
19/11:10:21|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/11:10:22|SPI: XIlinx spi: bus number now 32765
19/11:10:22|mice: PS/2 mouse device common for all mice
19/11:10:22|Device Tree Probing 'i2c'
19/11:10:22| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/11:10:22|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/11:10:22|TCP cubic registered
19/11:10:22|NET: Registered protocol family 17
19/11:10:22|RPC: Registered udp transport module.
19/11:10:22|RPC: Registered tcp transport module.
19/11:10:22|eth0: XLlTemac: Options: 0x3fa
19/11:10:22|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/11:10:23|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/11:10:23|eth0: XLlTemac: speed set to 1000Mb/s
19/11:10:25|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/11:10:25|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/11:10:25|Sending DHCP requests ., OK
19/11:10:26|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
19/11:10:26|IP-Config: Complete:
19/11:10:26| device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
19/11:10:26| host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/11:10:26| bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
19/11:10:26|Looking up port of RPC 100003/2 on 192.168.11.99
19/11:10:26|Looking up port of RPC 100005/1 on 192.168.11.99
19/11:10:26|VFS: Mounted root (nfs filesystem) on device 0:12.
19/11:10:26|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/11:10:27|Starting sysinit...
19/11:10:27| Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/11:10:27| Press 'I' to enter interactive startup.
19/11:10:27|Setting clock (utc): Thu Dec 19 10:10:28 GMT 2019 [ OK ]
19/11:10:28|Building the cache [ OK ]
19/11:10:28|Setting hostname aida04: [ OK ]
19/11:10:29|Mounting local filesystems: [ OK ]
19/11:10:30|Enabling /etc/fstab swaps: [ OK ]
19/11:10:32|Finishing sysinit...
INIT: Entering runlevel: 3
19/11:10:35|Entering non-interactive startup
19/11:10:36|FATAL: Module ipv6 not found.
19/11:10:37|Bringing up loopback interface: [ OK ]
19/11:10:39|FATAL: Module ipv6 not found.
19/11:10:39|Starting system logger: [ OK ]
19/11:10:40|Starting kernel logger: [ OK ]
19/11:10:40|Starting rpcbind: [ OK ]
19/11:10:41|Mounting NFS filesystems: [ OK ]
19/11:10:42|Mounting other filesystems: [ OK ]
19/11:10:42|Starting xinetd: [ OK ]
19/11:10:43|Starting midas: Starting MIDAS Data Acquisition for aida04
19/11:10:43|xaida: device parameters: base=0x81000000 size=0x200000
19/11:10:48|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/11:10:49|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/11:10:49|xaida: driver assigned major number 254
19/11:10:49|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/11:10:54|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/11:10:54|AIDAMEM: aidamem: driver assigned major number 253
19/11:10:54|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/11:11:01|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/11:11:01|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/11:11:01|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/11:11:01|Computer Name = aida04; Temp Directory = /tmp/tcl361
19/11:11:05|package limit is not available: can't find package limit
19/11:11:07|Running with default file descriptor limit
19/11:11:07|package setuid is not available: can't find package setuid
19/11:11:09|Running as user 0 group 0
19/11:11:09|
19/11:11:10|AIDA Data Acquisition Program Release 9_10.Apr 3 2019_11:34:31 starting
19/11:11:10|
19/11:11:10|Built without pthreaxaida: open:
19/11:11:10|ds
19/11:11:10|
19/11:11:10|Creating NetAIDAMEM: aidamem_open:
19/11:11:10|Vars
19/11:11:10|Output buffer length = 65504; format option = 4; transfer option = 3
19/11:11:11|EB transfer option = 3
19/11:11:11|NetVars created and initialised
19/11:11:11|Statistics thread starting
19/11:11:11|Statistics thread created
19/11:11:11|Stat/Rate creation thread starting
19/11:11:11|Data Acquisition task has PID 375
19/11:11:11|Stat/Rate creation thread created
19/11:11:11|Hit/Rate creation thread starting
19/11:11:11|Hit/Rate creation thread created
19/11:11:11|AIDA Heartbeat thread starting
19/11:11:11|Heartbeat thread created
19/11:11:11|Installing signal handlers
19/11:11:11|Done
19/11:11:11|ModuleNum = 0
19/11:11:11|Aida Initialise complete. AidaExecV9_10: Build Apr 3 2019_11:34:31. HDL version : 18430701
19/11:11:11|Spectra table initialised
19/11:11:11|AIDA Data Acquisition now all ready to start
19/11:11:11|SIGBUS, SIGSEGV and SIGPIPE traps setup
19/11:11:11|/debug user "debug" password "-f9x7ruru8cg"
19/11:11:17|httpd started on port 8015
19/11:11:18|
19/11:11:18|Cannot use /MIDAS/config/TclHttpd/aida04@8015/startup.tcl
19/11:11:18|Custom startup from /MIDAS/config/TclHttpd/aida04/startup.tcl
19/11:11:18|XAIDA Access package 1.0
19/11:11:19|/XAIDAAccessServer
19/11:11:19|XAD9252 Access package 1.0
19/11:11:20|/XAD9252AccessServer
19/11:11:20|/DataBaseAccessServer
19/11:11:20|/NetVarService
19/11:11:20|/SigTaskService
19/11:11:20|Loaded MemSasAccess
19/11:11:20|/SpectrumService
19/11:11:20|loading tcl/AIDARunControl.tcl for namespace ::
19/11:11:20|/DataAcquisitionControlServer
19/11:11:20|DefineMessage unknown
19/11:11:20|Run Control Server Implementation for AIDA
19/11:11:21|RunControlServer loaded
19/11:11:21|loading Html/RunControl/implementation.tcl
19/11:11:21|[ OK ]
19/11:11:21|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
19/11:11:21|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
19/11:11:21|
19/11:11:21|DENX ELDK version 4.2 build 2008-04-01
19/11:11:21|Linux 2.6.31 on a ppc
19/11:11:21|
19/11:11:21|aida04 login: Created UI registers
19/11:11:22|RunControl loaded
19/11:11:22|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
19/11:11:22|AIDA RunControl loaded
19/11:11:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl |