Overnight aida02 crashed and restarted itself. A power cycle of all FEE64s was required to recover.

The system log appears to show the aida02 embedded processor crashed (note log times are UTC+1
for some reason).

Apr 21 00:41:21 aida02 kernel: xaida: open:
Apr 23 02:09:41 aida02 kernel: Unable to handle kernel paging request for data at address 0x00200200
Apr 23 02:09:41 aida02 kernel: Faulting instruction address: 0xc0091dd8
Apr 23 02:09:41 aida02 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 23 02:09:41 aida02 kernel: PREEMPT Xilinx Virtex440
Apr 23 02:09:41 aida02 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 23 02:09:42 aida02 kernel: NIP: c0091dd8 LR: c0091e40 CTR: 00000000
Apr 23 02:09:42 aida02 kernel: REGS: c6835e60 TRAP: 0300   Not tainted  (2.6.31)
Apr 23 02:09:42 aida02 kernel: MSR: 00021000 <ME,CE>  CR: 24000088  XER: 00000000
Apr 23 02:09:42 aida02 kernel: DEAR: 00200200, ESR: 00800000
Apr 23 02:09:42 aida02 kernel: TASK = c6824880[5] 'events/0' THREAD: c6834000
Apr 23 02:09:42 aida02 kernel: GPR00: c0091e40 c6835f10 c6824880 00000001 00000003 c038c3a8 00000000 c0498000
Apr 23 02:09:42 aida02 kernel: GPR08: 00000008 00100100 00000051 00200200 00485e6c 00005aa8 4805de90 00000001
Apr 23 02:09:42 aida02 kernel: GPR16: 00000000 00000000 4801d000 00000003 c031aef4 c02fb040 c02fb008 00000002
Apr 23 02:09:42 aida02 kernel: GPR24: c6834000 00100100 00200200 c680ae00 00000001 c680daf0 c680dae0 c6940020
Apr 23 02:09:42 aida02 kernel: NIP [c0091dd8] drain_freelist+0x54/0x110
Apr 23 02:09:42 aida02 kernel: LR [c0091e40] drain_freelist+0xbc/0x110
Apr 23 02:09:42 aida02 kernel: Call Trace:
Apr 23 02:09:42 aida02 kernel: [c6835f10] [c0091e40] drain_freelist+0xbc/0x110 (unreliable)
Apr 23 02:09:42 aida02 kernel: [c6835f40] [c0093158] cache_reap+0x104/0x138
Apr 23 02:09:42 aida02 kernel: [c6835f60] [c0048908] worker_thread+0x140/0x204
Apr 23 02:09:42 aida02 kernel: [c6835fc0] [c004cfc8] kthread+0x78/0x7c
Apr 23 02:09:42 aida02 kernel: [c6835ff0] [c000e140] kernel_thread+0x4c/0x68
Apr 23 02:09:42 aida02 kernel: Instruction dump:
Apr 23 02:09:43 aida02 kernel: 3ba40010 7f80e800 419e00d4 3f200010 3f400020 54380024 63390100 635a0200
Apr 23 02:09:43 aida02 kernel: 3b800000 48000068 813f0000 817f0004 <912b0000> 91690004 933f0000 935f0004
Apr 23 02:09:43 aida02 kernel: ---[ end trace 6baa5b82e4ef4e92 ]---
Apr 23 02:09:43 aida02 kernel: note: events/0[5] exited with preempt_count 1
Apr 23 02:13:39 aida02 syslogd 1.4.2: restart.
Apr 23 02:13:39 aida02 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Apr 23 02:13:39 aida02 kernel: Using Xilinx Virtex440 machine description
Apr 23 02:13:39 aida02 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011