From ttyUSB6 system console log

20/07:07:10|kernel BUG at mm/slab.c:2974!
20/07:07:10|Oops: Exception in kernel mode, sig: 5 [#1]
20/07:07:10|PREEMPT Xilinx Virtex440
20/07:07:10|Modules linked in: aidamem xdriver xh_spidev_register
20/07:07:10|NIP: c009211c LR: c00920b0 CTR: c0038e70
20/07:07:10|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)
20/07:07:10|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000
20/07:07:10|TASK = c036e318[0] 'swapper' THREAD: c0390000
20/07:07:10|GPR00: 00000001 c0391ca0 c036e318 c680daf0 00000020 00000001 c6940000 00000000
20/07:07:11|GPR08: fffffff4 00010003 c680e400 c0390000 24008042 00005aa8 c03a0000 00000020
20/07:07:11|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200
20/07:07:11|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400
20/07:07:11|NIP [c009211c] cache_alloc_refill+0x130/0x608
20/07:07:11|LR [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:11|Call Trace:
20/07:07:11|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
20/07:07:11|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:11|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:11|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:11|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:11|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:11|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:11|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:11|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:11|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:11|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:11|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:12|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:12|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:12|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:12|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:12|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:12|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:12|Instruction dump:
20/07:07:12|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010
20/07:07:12|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840
20/07:07:12|Kernel panic - not syncing: Fatal exception in interrupt
20/07:07:12|Call Trace:
20/07:07:12|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/07:07:12|[c0391a80] [c00345bc] panic+0x94/0x168
20/07:07:12|[c0391ad0] [c000bd44] die+0x178/0x18c
20/07:07:12|[c0391af0] [c000c000] _exception+0x164/0x1b4
20/07:07:12|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c
20/07:07:12|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:12|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:12|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:12|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:13|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:13|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:13|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:13|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:13|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:13|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:13|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:13|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:13|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:13|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:13|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:13|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:13|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:13|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:13|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/07:10:14|
Found 0 errors checking kernel image
20/07:10:15|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/07:10:15|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/07:10:15|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/07:10:15|SPIconfig done now to set up the LMK3200 registers
20/07:10:15|LMK #0 : regInit[0]=0x80000000
20/07:10:15|LMK #0 : regInit[1]=0x10070600
20/07:10:15|LMK #0 : regInit[2]=0x60601
20/07:10:15|LMK #0 : regInit[3]=0x60602
20/07:10:16|LMK #0 : regInit[4]=0x60603
20/07:10:16|LMK #0 : regInit[5]=0x70624
20/07:10:16|LMK #0 : regInit[6]=0x70605
20/07:10:16|LMK #0 : regInit[7]=0x70606
20/07:10:16|LMK #0 : regInit[8]=0x70627
20/07:10:16|LMK #0 : regInit[9]=0x10000908
20/07:10:16|LMK #0 : regInit[10]=0xA0022A09
20/07:10:16|LMK #0 : regInit[11]=0x82800B
20/07:10:16|LMK #0 : regInit[12]=0x28C800D
20/07:10:16|LMK #0 : regInit[13]=0x830020E
20/07:10:16|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/07:10:16|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/07:10:16|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/07:10:16|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/07:10:17|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/07:10:20|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xee
20/07:10:26|Using Xilinx Virtex440 machine description
20/07:10:27|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/07:10:27|Zone PFN ranges:
20/07:10:27|  DMA      0x00000000 -> 0x00007000
20/07:10:27|  Normal   0x00007000 -> 0x00007000
20/07:10:27|Movable zone start PFN for each node
20/07:10:27|early_node_map[1] active PFN ranges
20/07:10:27|    0: 0x00000000 -> 0x00007000
20/07:10:27|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/07:10:27|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/07:10:27|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/07:10:27|PID hash table entries: 512 (order: 9, 2048 bytes)
20/07:10:27|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/07:10:27|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/07:10:28|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/07:10:28|Kernel virtual memory layout:
20/07:10:28|  * 0xffffe000..0xfffff000  : fixmap
20/07:10:28|  * 0xfde00000..0xfe000000  : consistent mem
20/07:10:28|  * 0xfde00000..0xfde00000  : early ioremap
20/07:10:28|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/07:10:28|NR_IRQS:512
20/07:10:28|clocksource: timebase mult[a00000] shift[22] registered
20/07:10:28|Console: colour dummy device 80x25
20/07:10:28|Mount-cache hash table entries: 512
20/07:10:28|NET: Registered protocol family 16
20/07:10:28|PCI: Probing PCI hardware
20/07:10:28|bio: create slab <bio-0> at 0
20/07:10:28|NET: Registered protocol family 2
20/07:10:28|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/07:10:28|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/07:10:28|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/07:10:28|TCP: Hash tables configured (established 4096 bind 4096)
20/07:10:28|TCP reno registered
20/07:10:28|NET: Registered protocol family 1
20/07:10:28|ROMFS MTD (C) 2007 Red Hat, Inc.
20/07:10:28|msgmni has been set to 214
20/07:10:29|io scheduler noop registered
20/07:10:29|io scheduler anticipatory registered
20/07:10:29|io scheduler deadline registered
20/07:10:29|io scheduler cfq registered (default)
20/07:10:29|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/07:10:29|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/07:10:29|console [ttyS0] enabled
20/07:10:29|brd: module loaded
20/07:10:29|loop: module loaded
20/07:10:29|Device Tree Probing 'ethernet'
20/07:10:29|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ee
20/07:10:29|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/07:10:29|XLlTemac: DCR address: 0x80
20/07:10:29|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/07:10:29|XLlTemac: Allocating DMA descriptors with kmalloc
20/07:10:29|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/07:10:29|XTemac: PHY detected at address 7.
20/07:10:29|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/07:10:29|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/07:10:29| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30|Using buffer write method
20/07:10:30|cfi_cmdset_0001: Erase suspend on write enabled
20/07:10:30|cmdlinepart partition parsing not available
20/07:10:30|RedBoot partition parsing not available
20/07:10:30|Creating 5 MTD partitions on "fc000000.flash":
20/07:10:30|0x000000000000-0x000000500000 : "golden_firmware"
20/07:10:30|0x000000500000-0x000000800000 : "golden_kernel"
20/07:10:30|0x000000800000-0x000000d00000 : "user_firmware"
20/07:10:30|0x000000d00000-0x000000fe0000 : "user_kernel"
20/07:10:30|0x000000fe0000-0x000001000000 : "env_variables"
20/07:10:30|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/07:10:30|SPI: XIlinx spi: bus number now 32766
20/07:10:30|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/07:10:30|SPI: XIlinx spi: bus number now 32765
20/07:10:30|mice: PS/2 mouse device common for all mice
20/07:10:31|Device Tree Probing 'i2c'
20/07:10:31| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/07:10:31|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/07:10:31|TCP cubic registered
20/07:10:31|NET: Registered protocol family 17
20/07:10:31|RPC: Registered udp transport module.
20/07:10:31|RPC: Registered tcp transport module.
20/07:10:31|eth0: XLlTemac: Options: 0x3fa
20/07:10:31|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/07:10:31|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/07:10:31|eth0: XLlTemac: speed set to 1000Mb/s
20/07:10:33|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/07:10:34|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/07:10:34|Sending DHCP requests ., OK
20/07:10:35|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
20/07:10:35|IP-Config: Complete:
20/07:10:35|     device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
20/07:10:35|     host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/07:10:35|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
20/07:10:35|Looking up port of RPC 100003/2 on 192.168.11.99
20/07:10:35|Looking up port of RPC 100005/1 on 192.168.11.99
20/07:10:35|VFS: Mounted root (nfs filesystem) on device 0:12.
20/07:10:35|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/07:10:36|Starting sysinit...
20/07:10:36|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/07:10:37|            Press 'I' to enter interactive startup.
20/07:10:37|Setting clock  (utc): Tue Apr 20 07:10:37 BST 2021 [  OK  ]
20/07:10:38|Building the cache [  OK  ]
20/07:10:38|Setting hostname aida09:  [  OK  ]
20/07:10:39|Mounting local filesystems:  [  OK  ]
20/07:10:40|Enabling /etc/fstab swaps:  [  OK  ]
20/07:10:43|Finishing sysinit...
INIT: Entering runlevel: 3
20/07:10:46|Entering non-interactive startup
20/07:10:46|FATAL: Module ipv6 not found.
20/07:10:47|Bringing up loopback interface:  [  OK  ]
20/07:10:49|FATAL: Module ipv6 not found.
20/07:10:50|Starting system logger: [  OK  ]
20/07:10:50|Starting kernel logger: [  OK  ]
20/07:10:50|Starting rpcbind: [  OK  ]
20/07:10:52|Mounting NFS filesystems:  [  OK  ]
20/07:10:53|Mounting other filesystems:  [  OK  ]
20/07:10:53|Starting xinetd: [  OK  ]
20/07:10:54|Starting midas:  Starting MIDAS Data Acquisition for aida09
20/07:10:54|xaida: device parameters: base=0x81000000 size=0x200000
20/07:11:00|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/07:11:00|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/07:11:00|xaida: driver assigned major number 254
20/07:11:00|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/07:11:06|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/07:11:06|AIDAMEM: aidamem: driver assigned major number 253
20/07:11:06|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/07:11:12|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/07:11:12|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/07:11:12|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/07:11:13|Computer Name = aida09; Temp Directory = /tmp/tcl361
20/07:11:17|package limit is not available: can't find package limit
20/07:11:20|Running with default file descriptor limit
20/07:11:20|package setuid is not available: can't find package setuid
20/07:11:21|Running as user 0 group 0
20/07:11:21|
20/07:11:22|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
20/07:11:22|
20/07:11:22|Built without pthreads
20/07:11:22|xaida: open:
20/07:11:22|
20/07:11:22|Creating NetAIDAMEM: aidamem_open:
20/07:11:22|Vars
20/07:11:22|Output buffer length = 65504; format option = 4; transfer option = 3
20/07:11:22|EB transfer option = 3
20/07:11:22|NetVars created and initialised
20/07:11:22|Statistics thread starting
20/07:11:22|Statistics thread created
20/07:11:22|Stat/Rate creation thread starting
20/07:11:22|Data Acquisition task has PID 375
20/07:11:22|Stat/Rate creation thread created
20/07:11:22|Hit/Rate creation thread starting
20/07:11:22|Hit/Rate creation thread created
20/07:11:22|AIDA Heartbeat thread starting
20/07:11:22|Heartbeat thread created
20/07:11:22|Installing signal handlers
20/07:11:22|Done
20/07:11:22|ModuleNum = 0
20/07:11:22|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706
20/07:11:22|Spectra table initialised
20/07:11:22|AIDA Data Acquisition now all ready to start
20/07:11:23|SIGBUS, SIGSEGV and SIGPIPE traps setup
20/07:11:23|/debug user "debug" password "5tvq-4iyxl 4"
20/07:11:30|httpd started on port 8015
20/07:11:30|
20/07:11:30|Cannot use /MIDAS/config/TclHttpd/aida09@8015/startup.tcl
20/07:11:30|Custom startup from /MIDAS/config/TclHttpd/aida09/startup.tcl
20/07:11:30|XAIDA Access package 1.0
20/07:11:32|/XAIDAAccessServer
20/07:11:32|XAD9252 Access package 1.0
20/07:11:32|/XAD9252AccessServer
20/07:11:32|/DataBaseAccessServer
20/07:11:32|[  OK  ]
20/07:11:32|/NetVarService
20/07:11:32|/SigTaskService
20/07:11:32|Loaded MemSasAccess
20/07:11:33|/SpectrumService
20/07:11:33|loading tcl/AIDARunControl.tcl for namespace ::
20/07:11:33|/DataAcquisitionControlServer
20/07:11:33|DefineMessage unknown
20/07:11:33|
20/07:11:33|Run Control Server Implementation for AIDA
20/07:11:33|DENX ELDK version 4.2 build 2008-04-01
20/07:11:33|Linux 2.6.31 on a ppc
20/07:11:33|
20/07:11:33|aida09 login: RunControlServer loaded
20/07:11:33|loading Html/RunControl/implementation.tcl
20/07:11:33|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
20/07:11:34|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
20/07:11:34|Created UI registers
20/07:11:34|RunControl loaded
20/07:11:34|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
20/07:11:34|AIDA RunControl loaded
20/07:11:35|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
20/07:11:35|do_GetState returned z=0 and 8
20/07:12:03|do_GetState returned z=0 and 8
20/07:12:09|do_GetState returned z=0 and 8
20/07:12:21|xaida: open:
20/07:12:35|do_GetState returned z=0 and 8
20/07:12:36|

From aida01 /var/log/messages

Apr 20 21:04:06 aida01 kernel: BUG: scheduling while atomic: tclsh8.5/398/0x00000201
Apr 20 21:04:08 aida01 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 20 21:04:09 aida01 kernel: Call Trace:
Apr 20 21:04:11 aida01 kernel: [c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
Apr 20 21:04:12 aida01 kernel: [c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
Apr 20 21:04:14 aida01 kernel: [c6941aa0] [c02a8020] schedule+0x2f4/0x34c
Apr 20 21:07:53 aida01 syslogd 1.4.2: restart.

From ttyUSB0 system console log

20/21:04:06|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:06|Call Trace:
20/21:04:06|[c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:06|[c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:07|[c6941aa0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:07|[c6941ae0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:07|[c6941af0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:07|[c6941b10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:07|[c6941b50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:07|[c6941b90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:07|[c6941ba0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:07|[c6941bd0] [c013da58] nfs3_rpc_wrapper+0x48/0x84
20/21:04:07|[c6941bf0] [c013e734] nfs3_proc_access+0xcc/0x1ac
20/21:04:07|[c6941cc0] [c0129644] nfs_do_access+0xb8/0x438
20/21:04:07|[c6941d00] [c0129a6c] nfs_permission+0xa8/0x1c4
20/21:04:07|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:07|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:07|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:07|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:07|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:07|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:07|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:07|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:07|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:08|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:08|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:08|Call Trace:
20/21:04:08|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:08|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:08|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:08|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:08|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:08|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:08|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:08|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:08|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:08|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:08|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:08|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:08|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:08|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:09|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:09|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:09|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:09|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:09|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:09|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:09|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:09|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:09|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:09|Call Trace:
20/21:04:09|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:09|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:09|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:09|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:09|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:09|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:09|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:09|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:09|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:09|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:10|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:10|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:10|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:10|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:10|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:10|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:10|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:10|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:10|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:10|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:10|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:10|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:10|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:10|Call Trace:
20/21:04:10|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:10|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:10|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:10|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:10|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:10|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:11|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:11|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:11|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:11|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:11|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:11|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:11|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:11|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:11|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:11|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:11|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:11|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:11|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:11|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:11|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:11|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:11|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:11|Call Trace:
20/21:04:11|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:11|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:12|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:12|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:12|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:12|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:12|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:12|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:12|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:12|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:12|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:12|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:12|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:12|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:12|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:12|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:12|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:12|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:12|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:12|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:12|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:12|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:13|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:13|Call Trace:
20/21:04:13|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:13|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:13|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:13|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:13|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:13|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:13|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:13|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:13|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:13|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:13|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:13|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:13|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:13|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:13|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:13|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:13|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:13|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:14|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:14|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:14|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:14|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:14|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:14|Call Trace:
20/21:04:14|[c6941950] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:14|[c6941990] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:14|[c69419a0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:14|[c69419e0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:14|[c69419f0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:14|[c6941a10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:14|[c6941a50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:14|[c6941a90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:14|[c6941aa0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:14|[c6941ad0] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:14|[c6941b00] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:14|[c6941bc0] [c012bd70] nfs_lookup_revalidate+0x3c0/0x464
20/21:04:14|[c6941d00] [c00a0f7c] do_lookup+0x54/0x20c
20/21:04:14|[c6941d30] [c00a308c] __link_path_walk+0x57c/0xf7c
20/21:04:15|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:15|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:15|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:15|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:15|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:15|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:15|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:15|Page fault in user mode with in_atomic() = 1 mm = c6178c80
20/21:04:15|NIP = fb85a30  MSR = 2d000
20/21:04:15|Oops: Weird page fault, sig: 11 [#1]
20/21:04:15|PREEMPT Xilinx Virtex440
20/21:04:15|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:15|NIP: 0fb85a30 LR: 0ff85338 CTR: c01288f4
20/21:04:15|REGS: c6941f50 TRAP: 0401   Not tainted  (2.6.31)
20/21:04:15|MSR: 0002d000 <EE,PR,ME,CE>  CR: 44884044  XER: 00000000
20/21:04:15|TASK = c608e0c0[398] 'tclsh8.5' THREAD: c6940000
20/21:04:15|GPR00: 0ff85334 bf9a3870 4802b0a0 ffffffff 0fc094e0 54884044 fffffff2 0fc08d3c
20/21:04:15|GPR08: 0002d000 480240a8 00000000 bf9a3870 c6940000
20/21:04:15|NIP [0fb85a30] 0xfb85a30
20/21:04:15|LR [0ff85338] 0xff85338
20/21:04:16|Call Trace:
20/21:04:16|Kernel panic - not syncing: Fatal exception in interrupt
20/21:04:16|Call Trace:
20/21:04:16|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:16|[c6941e20] [c00345bc] panic+0x94/0x168
20/21:04:16|[c6941e70] [c000bd44] die+0x178/0x18c
20/21:04:16|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
20/21:04:16|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
20/21:04:16|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/21:07:16|
Found 0 errors checking kernel image
20/21:07:18|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/21:07:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/21:07:18|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/21:07:18|SPIconfig done now to set up the LMK3200 registers
20/21:07:18|LMK #0 : regInit[0]=0x80000000
20/21:07:18|LMK #0 : regInit[1]=0x10070600
20/21:07:18|LMK #0 : regInit[2]=0x60601
20/21:07:18|LMK #0 : regInit[3]=0x60602
20/21:07:18|LMK #0 : regInit[4]=0x60603
20/21:07:18|LMK #0 : regInit[5]=0x70624
20/21:07:18|LMK #0 : regInit[6]=0x70605
20/21:07:18|LMK #0 : regInit[7]=0x70606
20/21:07:18|LMK #0 : regInit[8]=0x70627
20/21:07:18|LMK #0 : regInit[9]=0x10000908
20/21:07:18|LMK #0 : regInit[10]=0xA0022A09
20/21:07:18|LMK #0 : regInit[11]=0x82800B
20/21:07:18|LMK #0 : regInit[12]=0x28C800D
20/21:07:18|LMK #0 : regInit[13]=0x830020E
20/21:07:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/21:07:19|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/21:07:19|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
ADCs calibrated
20/21:07:19|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
20/21:07:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/21:07:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xba 0x8a
20/21:07:29|Using Xilinx Virtex440 machine description
20/21:07:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/21:07:30|Zone PFN ranges:
20/21:07:30|  DMA      0x00000000 -> 0x00007000
20/21:07:30|  Normal   0x00007000 -> 0x00007000
20/21:07:30|Movable zone start PFN for each node
20/21:07:30|early_node_map[1] active PFN ranges
20/21:07:30|    0: 0x00000000 -> 0x00007000
20/21:07:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/21:07:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/21:07:30|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/21:07:30|PID hash table entries: 512 (order: 9, 2048 bytes)
20/21:07:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/21:07:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/21:07:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/21:07:30|Kernel virtual memory layout:
20/21:07:30|  * 0xffffe000..0xfffff000  : fixmap
20/21:07:31|  * 0xfde00000..0xfe000000  : consistent mem
20/21:07:31|  * 0xfde00000..0xfde00000  : early ioremap
20/21:07:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/21:07:31|NR_IRQS:512
20/21:07:31|clocksource: timebase mult[a00000] shift[22] registered
20/21:07:31|Console: colour dummy device 80x25
20/21:07:31|Mount-cache hash table entries: 512
20/21:07:31|NET: Registered protocol family 16
20/21:07:31|PCI: Probing PCI hardware
20/21:07:31|bio: create slab <bio-0> at 0
20/21:07:31|NET: Registered protocol family 2
20/21:07:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/21:07:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/21:07:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/21:07:31|TCP: Hash tables configured (established 4096 bind 4096)
20/21:07:31|TCP reno registered
20/21:07:31|NET: Registered protocol family 1
20/21:07:31|ROMFS MTD (C) 2007 Red Hat, Inc.
20/21:07:31|msgmni has been set to 214
20/21:07:31|io scheduler noop registered
20/21:07:31|io scheduler anticipatory registered
20/21:07:31|io scheduler deadline registered
20/21:07:31|io scheduler cfq registered (default)
20/21:07:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/21:07:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/21:07:32|console [ttyS0] enabled
20/21:07:32|brd: module loaded
20/21:07:32|loop: module loaded
20/21:07:32|Device Tree Probing 'ethernet'
20/21:07:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ba:8a
20/21:07:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/21:07:32|XLlTemac: DCR address: 0x80
20/21:07:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/21:07:32|XLlTemac: Allocating DMA descriptors with kmalloc
20/21:07:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/21:07:32|XTemac: PHY detected at address 7.
20/21:07:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/21:07:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33|Using buffer write method
20/21:07:33|cfi_cmdset_0001: Erase suspend on write enabled
20/21:07:33|cmdlinepart partition parsing not available
20/21:07:33|RedBoot partition parsing not available
20/21:07:33|Creating 5 MTD partitions on "fc000000.flash":
20/21:07:33|0x000000000000-0x000000500000 : "golden_firmware"
20/21:07:33|0x000000500000-0x000000800000 : "golden_kernel"
20/21:07:33|0x000000800000-0x000000d00000 : "user_firmware"
20/21:07:33|0x000000d00000-0x000000fe0000 : "user_kernel"
20/21:07:33|0x000000fe0000-0x000001000000 : "env_variables"
20/21:07:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/21:07:33|SPI: XIlinx spi: bus number now 32766
20/21:07:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/21:07:33|SPI: XIlinx spi: bus number now 32765
20/21:07:33|mice: PS/2 mouse device common for all mice
20/21:07:33|Device Tree Probing 'i2c'
20/21:07:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/21:07:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/21:07:33|TCP cubic registered
20/21:07:33|NET: Registered protocol family 17
20/21:07:34|RPC: Registered udp transport module.
20/21:07:34|RPC: Registered tcp transport module.
20/21:07:34|eth0: XLlTemac: Options: 0x3fa
20/21:07:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/21:07:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/21:07:34|eth0: XLlTemac: speed set to 1000Mb/s
20/21:07:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/21:07:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/21:07:36|Sending DHCP requests ., OK
20/21:07:38|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.1
20/21:07:38|IP-Config: Complete:
20/21:07:38|     device=eth0, addr=192.168.11.1, mask=255.255.255.0, gw=255.255.255.255,
20/21:07:38|     host=aida01, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/21:07:38|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01
20/21:07:38|Looking up port of RPC 100003/2 on 192.168.11.99
20/21:07:38|Looking up port of RPC 100005/1 on 192.168.11.99
20/21:07:38|VFS: Mounted root (nfs filesystem) on device 0:12.
20/21:07:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/21:07:39|Starting sysinit...
20/21:07:39|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/21:07:40|            Press 'I' to enter interactive startup.
20/21:07:40|Setting clock  (utc): Tue Apr 20 21:07:40 BST 2021 [  OK  ]
20/21:07:40|Building the cache [  OK  ]
20/21:07:40|Setting hostname aida01:  [  OK  ]
20/21:07:42|Mounting local filesystems:  [  OK  ]
20/21:07:43|Enabling /etc/fstab swaps:  [  OK  ]
20/21:07:46|Finishing sysinit...
INIT: Entering runlevel: 3
20/21:07:49|Entering non-interactive startup
20/21:07:49|FATAL: Module ipv6 not found.
20/21:07:51|Bringing up loopback interface:  [  OK  ]
20/21:07:53|FATAL: Module ipv6 not found.
20/21:07:53|Starting system logger: [  OK  ]
20/21:07:54|Starting kernel logger: [  OK  ]
20/21:07:54|Starting rpcbind: [  OK  ]
20/21:07:56|Mounting NFS filesystems:  [  OK  ]
20/21:07:57|Mounting other filesystems:  [  OK  ]
20/21:07:57|Starting xinetd: [  OK  ]
20/21:07:58|Starting midas:  Starting MIDAS Data Acquisition for aida01
20/21:07:58|xaida: device parameters: base=0x81000000 size=0x200000
20/21:08:04|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/21:08:04|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/21:08:05|xaida: driver assigned major number 254
20/21:08:05|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/21:08:10|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/21:08:10|AIDAMEM: aidamem: driver assigned major number 253
20/21:08:10|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/21:08:17|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/21:08:17|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/21:08:17|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/21:08:17|Computer Name = aida01; Temp Directory = /tmp/tcl361
20/21:08:22|package limit is not available: can't find package limit
20/21:08:25|Running with default file descriptor limit
20/21:08:25|
20/21:08:26|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting

https://docs.google.com/spreadsheets/d/1CyL5L-WLsh9fsS7CN989GC3ktqiPQOkG8YIDJrrGxYA/edit#gid=0

I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again

Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)

System Wide Checks
Clocks: OK
FADC: OK
WR: 
		 Base 		Current 	Difference
aida05 fault 	 0x31a2 : 	 0x31a2 : 	 0  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22e8 : 	 1  
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK

aida05 ASIC maybe didn't complete setup properly (WR 0x10?)

Timestamps: OK (1)
Temps: OK

Will start gzipping S460 runs for migration to lustre & tape

Beam has now stopped (end of expriment) and background measurements starts.

Some screenshot of spectra attached

ucesb is back - see attachment

Quick system checks after crash

Temperature ok.

Still cannot connect remotely to /ucesb with my gsi web login and password (?).

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida05 fault      0x31a2 :      0x31a2 :      0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      9      3      4      2      3      2      1      4      2      3      7   : 39804
aida02 :     29      4      2      3      0      3      1      4      2      3      7   : 39828
aida03 :     20      6      1      2      2      3      1      4      2      3      7   : 39888
aida04 :     19      5      4      2      1      3      1      4      2      3      7   : 39860
aida05 :     27      5      4      1      2      3      2      3      1      4      7   : 40692
aida06 :     17     10      3      2      1      3      1      4      2      3      7   : 39876
aida07 :     22      7      2      2      2      3      1      4      2      3      7   : 39920
aida08 :     12      6      5      1      3      2      1      4      2      3      7   : 39824
aida09 :      0      9      1      3      1      3      1      4      2      3      7   : 39800
aida10 :      2      2      2      3      3      2      3      2      2      3      7   : 39256
aida11 :     21      7      4      3      0      3      2      3      2      3      7   : 39596
aida12 :     18      8      3      0      2      3      2      3      2      3      7   : 39608

Looks like the aida-gsi window manager/login session crashed.

On site S460 shift restarted AnyDesk. Upon re-connection *all* browsers, terminals, applications were gone
I have re-started the basics but BNC PB-5 pulser, CAEN NIM bin, MBS rate monitor, system console monitor etc
still need to be re-started. 

Extract from /var/log/messages

Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal cpuset=/ mems_allowed=0
Apr 24 03:19:01 aidas-gsi kernel: Pid: 18955, comm: gnome-terminal Not tainted 2.6.32-696.el6.x86_64 #1
Apr 24 03:19:01 aidas-gsi kernel: Call Trace:
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131550>] ? dump_header+0x90/0x1b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8123c1dc>] ? security_real_capable_noaudit+0x3c/0x70
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811319d2>] ? oom_kill_process+0x82/0x2a0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131911>] ? select_bad_process+0xe1/0x120
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131e10>] ? out_of_memory+0x220/0x3c0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113e7ec>] ? __alloc_pages_nodemask+0x93c/0x950
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81015533>] ? native_sched_clock+0x13/0x80
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff810aaea1>] ? lock_hrtimer_base+0x31/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81177a3a>] ? alloc_pages_current+0xaa/0x110
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113bb4e>] ? __get_free_pages+0xe/0x50
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1944>] ? __pollwait+0xb4/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81348684>] ? n_tty_poll+0x194/0x1d0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff813440aa>] ? tty_poll+0x8a/0xa0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1db5>] ? do_sys_poll+0x2d5/0x5b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1890>] ? __pollwait+0x0/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1745>] ? poll_select_set_timeout+0x95/0xb0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b2281>] ? sys_poll+0x71/0x100
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Apr 24 03:19:01 aidas-gsi kernel: Mem-Info:
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 172
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  45
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd:  97
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 142
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  49
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd:  56
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 179
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd:  67
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 107
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 170
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  54
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 118
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 196
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  36
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  37
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd: 128
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 173
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd: 161
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 177
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 178
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  99
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 183
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 162
Apr 24 03:19:01 aidas-gsi kernel: active_anon:3408671 inactive_anon:401328 isolated_anon:128
Apr 24 03:19:01 aidas-gsi kernel: active_file:22995 inactive_file:61739 isolated_file:0
Apr 24 03:19:01 aidas-gsi kernel: unevictable:0 dirty:59856 writeback:22 unstable:0
Apr 24 03:19:01 aidas-gsi kernel: free:35843 slab_reclaimable:15134 slab_unreclaimable:65491
Apr 24 03:19:01 aidas-gsi kernel: mapped:129529 shmem:249820 pagetables:24586 bounce:0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA free:15752kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 874 16024 16024
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 free:64204kB min:3680kB low:4600kB high:5520kB active_anon:369500kB inactive_anon:377436kB active_file:10988kB 
inactive_file:26828kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:895084kB mlocked:0kB dirty:26856kB writeback:88kB mapped:35896kB shmem:39156kB 
slab_reclaimable:4224kB slab_unreclaimable:4344kB kernel_stack:3344kB pagetables:2032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:65792 all_unreclaimable? 
no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 15150 15150
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal free:63416kB min:63836kB low:79792kB high:95752kB active_anon:13265184kB inactive_anon:1227876kB active_file:80992kB 
inactive_file:220128kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15513600kB mlocked:0kB dirty:212568kB writeback:0kB mapped:482220kB 
shmem:960124kB slab_reclaimable:56312kB slab_unreclaimable:257620kB kernel_stack:17600kB pagetables:96312kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:17824 all_unreclaimable? no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 0 0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32: 373*4kB 559*8kB 428*16kB 296*32kB 189*64kB 111*128kB 43*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 64204kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal: 16100*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64408kB
Apr 24 03:19:01 aidas-gsi kernel: 357788 total pagecache pages
Apr 24 03:19:01 aidas-gsi kernel: 22930 pages in swap cache
Apr 24 03:19:01 aidas-gsi kernel: Swap cache stats: add 3720478, delete 3697548, find 5026852/5228898
Apr 24 03:19:01 aidas-gsi kernel: Free swap  = 1826100kB
Apr 24 03:19:01 aidas-gsi kernel: Total swap = 8200188kB
Apr 24 03:19:01 aidas-gsi kernel: 4194303 pages RAM
Apr 24 03:19:01 aidas-gsi kernel: 97868 pages reserved
Apr 24 03:19:01 aidas-gsi kernel: 578727 pages shared
Apr 24 03:19:01 aidas-gsi kernel: 3846484 pages non-shared
Apr 24 03:19:01 aidas-gsi kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 24 03:19:01 aidas-gsi kernel: [ 1388]     0  1388     2819      130   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 3927]     0  3927     7442      128  12     -17         -1000 auditd
Apr 24 03:19:01 aidas-gsi kernel: [ 3961]     0  3961    62914      755  15       0             0 rsyslogd
Apr 24 03:19:01 aidas-gsi kernel: [ 4029]     0  4029     4604      141   7       0             0 irqbalance
Apr 24 03:19:01 aidas-gsi kernel: [ 4054]    32  4054     4746      124  13       0             0 rpcbind
Apr 24 03:19:01 aidas-gsi kernel: [ 4074]    81  4074     8194      440   6       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 4089]     0  4089    26490     2647   4       0             0 NetworkManager
Apr 24 03:19:01 aidas-gsi kernel: [ 4096]     0  4096    14534      328  12       0             0 modem-manager
Apr 24 03:19:01 aidas-gsi kernel: [ 4112]    29  4112     5839       97   0       0             0 rpc.statd
Apr 24 03:19:01 aidas-gsi kernel: [ 4151]     0  4151    47297      117  15       0             0 cupsd
Apr 24 03:19:01 aidas-gsi kernel: [ 4171]     0  4171    11251       75   9       0             0 wpa_supplicant
Apr 24 03:19:01 aidas-gsi kernel: [ 4180]     0  4180     2280      162   9       0             0 dhclient
Apr 24 03:19:01 aidas-gsi kernel: [ 4210]     0  4210     1020       82   8       0             0 acpid
Apr 24 03:19:01 aidas-gsi kernel: [ 4222]    68  4222     9968      569  15       0             0 hald
Apr 24 03:19:01 aidas-gsi kernel: [ 4223]     0  4223     5100      184  12       0             0 hald-runner
Apr 24 03:19:01 aidas-gsi kernel: [ 4271]     0  4271     5630      157  12       0             0 hald-addon-inpu
Apr 24 03:19:01 aidas-gsi kernel: [ 4272]    68  4272     4502      120   0       0             0 hald-addon-acpi
Apr 24 03:19:01 aidas-gsi kernel: [ 4306]     0  4306    96540      161   0       0             0 automount
Apr 24 03:19:01 aidas-gsi kernel: [ 4492]     0  4492    26831       15   9       0             0 rpc.rquotad
Apr 24 03:19:01 aidas-gsi kernel: [ 4497]     0  4497     5450      213   4       0             0 rpc.mountd
Apr 24 03:19:01 aidas-gsi kernel: [ 4542]     0  4542     5776       42   4       0             0 rpc.idmapd
Apr 24 03:19:01 aidas-gsi kernel: [ 4567]     0  4567    16573      113   3     -17         -1000 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 4578]     0  4578     5429      117   7       0             0 xinetd
Apr 24 03:19:01 aidas-gsi kernel: [ 4589]    38  4589     7686      213  10       0             0 ntpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4675]     0  4675    20260      147   5       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [ 4682]    89  4682    20323      141   5       0             0 qmgr
Apr 24 03:19:01 aidas-gsi kernel: [ 4696]     0  4696    27153      253   0       0             0 ksmtuned
Apr 24 03:19:01 aidas-gsi kernel: [ 4709]     0  4709    29222      137  10       0             0 crond
Apr 24 03:19:01 aidas-gsi kernel: [ 4724]     0  4724     5277       50  12       0             0 atd
Apr 24 03:19:01 aidas-gsi kernel: [ 4741]     0  4741   172796     2328   0       0             0 virt-who
Apr 24 03:19:01 aidas-gsi kernel: [ 4762]     0  4762   256599      407  10       0             0 libvirtd
Apr 24 03:19:01 aidas-gsi kernel: [ 4858]     0  4858    16126      150  10       0             0 certmonger
Apr 24 03:19:01 aidas-gsi kernel: [ 4903]    99  4903     3223      102   2       0             0 dnsmasq
Apr 24 03:19:01 aidas-gsi kernel: [ 4918]     0  4918     4539      223   5       0             0 dhcpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4920]     0  4920    33509      312  11       0             0 gdm-binary
Apr 24 03:19:01 aidas-gsi kernel: [ 4927]     0  4927     1016       61  11       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4929]     0  4929     1016       61  12       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4931]     0  4931     1016       61   9       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4933]     0  4933     1016       61  15       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4935]     0  4935     1016       61   1       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4944]     0  4944     3148      440   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 4945]     0  4945     3148      438   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 5050]     0  5050  1029198      377   3       0             0 console-kit-dae
Apr 24 03:19:01 aidas-gsi kernel: [ 5126]     0  5126    12508      365   3       0             0 devkit-power-da
Apr 24 03:19:01 aidas-gsi kernel: [ 5164]     0  5164    45435      581   1       0             0 polkitd
Apr 24 03:19:01 aidas-gsi kernel: [ 5175]   499  5175    42114      146   0       0             0 rtkit-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5702]     0  5702    14009      514   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5725]     0  5725    11306       74   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18391]     0 18391    41240      293   8       0             0 gdm-simple-slav
Apr 24 03:19:01 aidas-gsi kernel: [18393]     0 18393  3476635  2412883  13       0             0 Xorg
Apr 24 03:19:01 aidas-gsi kernel: [18427]    42 18427     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18464]     0 18464    47926      269   0       0             0 gdm-session-wor
Apr 24 03:19:01 aidas-gsi kernel: [18510]   500 18510    57281      424   1       0             0 gnome-keyring-d
Apr 24 03:19:01 aidas-gsi kernel: [18520]   500 18520    64493      751   0       0             0 gnome-session
Apr 24 03:19:01 aidas-gsi kernel: [18528]   500 18528     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18529]   500 18529     8211      379   8       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18604]   500 18604    33379     1113  13       0             0 gconfd-2
Apr 24 03:19:01 aidas-gsi kernel: [18612]   500 18612   139831     1984   9       0             0 gnome-settings-
Apr 24 03:19:01 aidas-gsi kernel: [18614]   500 18614    71643      696  12       0             0 seahorse-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18616]   500 18616    34351      298  13       0             0 gvfsd
Apr 24 03:19:01 aidas-gsi kernel: [18636]   500 18636   112427     2321   4       0             0 metacity
Apr 24 03:19:01 aidas-gsi kernel: [18637]   500 18637   203008     6568  10       0             0 gnome-panel
Apr 24 03:19:01 aidas-gsi kernel: [18638]   500 18638   326834    16727  10       0             0 nautilus
Apr 24 03:19:01 aidas-gsi kernel: [18640]   500 18640   141364      341  14       0             0 bonobo-activati
Apr 24 03:19:01 aidas-gsi kernel: [18642]   500 18642    65558      741  10       0             0 bluetooth-apple
Apr 24 03:19:01 aidas-gsi kernel: [18645]   500 18645    68687      968   4       0             0 gnome-power-man
Apr 24 03:19:01 aidas-gsi kernel: [18646]   500 18646     7601      812  12       0             0 restorecond
Apr 24 03:19:01 aidas-gsi kernel: [18648]   500 18648    28879      401   5       0             0 im-settings-dae
Apr 24 03:19:01 aidas-gsi kernel: [18649]   500 18649   132278     1323   4       0             0 gnome-volume-co
Apr 24 03:19:01 aidas-gsi kernel: [18654]   500 18654    75318     2282  10       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18657]   500 18657   119660     1244   2       0             0 trashapplet
Apr 24 03:19:01 aidas-gsi kernel: [18659]   500 18659    88087     3906   5       0             0 wnck-applet
Apr 24 03:19:01 aidas-gsi kernel: [18660]   500 18660    62988      653   7       0             0 vino-server
Apr 24 03:19:01 aidas-gsi kernel: [18661]   500 18661    72490      958   0       0             0 polkit-gnome-au
Apr 24 03:19:01 aidas-gsi kernel: [18662]   500 18662    82733     1441   5       0             0 nm-applet
Apr 24 03:19:01 aidas-gsi kernel: [18666]   500 18666   126400      635   1       0             0 gvfs-gdu-volume
Apr 24 03:19:01 aidas-gsi kernel: [18667]   500 18667    74660     1042   5       0             0 gpk-update-icon
Apr 24 03:19:01 aidas-gsi kernel: [18669]   500 18669    64033      850   0       0             0 gdu-notificatio
Apr 24 03:19:01 aidas-gsi kernel: [18676]   500 18676    81107     2344   6       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [18693]   500 18693   131702     2471   3       0             0 pulseaudio
Apr 24 03:19:01 aidas-gsi kernel: [18823]   500 18823    58170      269   4       0             0 gvfs-afc-volume
Apr 24 03:19:01 aidas-gsi kernel: [18836]   500 18836    37789      300   8       0             0 gvfs-gphoto2-vo
Apr 24 03:19:01 aidas-gsi kernel: [18844]   500 18844    36051      437   0       0             0 gvfsd-trash
Apr 24 03:19:01 aidas-gsi kernel: [18868]   500 18868    10275      287   6       0             0 gconf-im-settin
Apr 24 03:19:01 aidas-gsi kernel: [18869]   500 18869    84055     4872   3       0             0 gnome-screensav
Apr 24 03:19:01 aidas-gsi kernel: [18870]   500 18870     2057       88   3       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18871]   500 18871    27123      198   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18894]   500 18894    98681     1288   0       0             0 gdm-user-switch
Apr 24 03:19:01 aidas-gsi kernel: [18896]   500 18896    72800      911   3       0             0 notification-ar
Apr 24 03:19:01 aidas-gsi kernel: [18897]   500 18897   143660     1846   2       0             0 clock-applet
Apr 24 03:19:01 aidas-gsi kernel: [18900]   500 18900    23718      242   3       0             0 gconf-helper
Apr 24 03:19:01 aidas-gsi kernel: [18918]   500 18918    34358      265   0       0             0 gvfsd-burn
Apr 24 03:19:01 aidas-gsi kernel: [18932]   500 18932    33897      307   4       0             0 gvfsd-metadata
Apr 24 03:19:01 aidas-gsi kernel: [18955]   500 18955    82196     7037   7       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18957]   500 18957     2057       97   7       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18977]   500 18977    27123      114   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18991]     0 18991    47361      130  10       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [18992]     0 18992    27524      290  15       0             0 iptraf
Apr 24 03:19:01 aidas-gsi kernel: [21973]   500 21973  1058234   192783   0       0             0 firefox
Apr 24 03:19:01 aidas-gsi kernel: [21987]   500 21987    68608      284   3       0             0 at-spi-bus-laun
Apr 24 03:19:01 aidas-gsi kernel: [22221]   500 22221   731047    34381   0       0             0 Privileged Cont
Apr 24 03:19:01 aidas-gsi kernel: [22248]   500 22248   705574    10970   0       0             0 WebExtensions
Apr 24 03:19:01 aidas-gsi kernel: [22335]   500 22335   960920   166074   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22407]   500 22407  1023120    77418   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22810]   500 22810    27123      114   2       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [22826]   500 22826    15487      185   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [22830]   500 22830    27123      114   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26234]   500 26234   839272    43450   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [26311]   500 26311    27123      282   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26884]   500 26884    27123      115   5       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19388]   500 19388    27123      293   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11683]   500 11683   791783    18526   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [ 4876]   500  4876    54985      304   5       0             0 gvfsd-computer
Apr 24 03:19:01 aidas-gsi kernel: [17248]   500 17248    29595      100   1       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17249]   500 17249    29631      157  10       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17250]   500 17250    27124      114   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17266]   500 17266    15489      181   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [19879]     0 19879    25531      154   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19883]   500 19883    25531      163   2       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19884]   500 19884    27638      106  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19907]   500 19907    29595       97   5       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [14850]   500 14850    56156      327   2       0             0 gvfsd-network
Apr 24 03:19:01 aidas-gsi kernel: [14855]   500 14855    36479      318   3       0             0 gvfsd-dnssd
Apr 24 03:19:01 aidas-gsi kernel: [22615]   500 22615    27123      117   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [13867]     0 13867    25531      161   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13880]   500 13880    25645      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13881]   500 13881    14400      138   8       0             0 sftp-server
Apr 24 03:19:01 aidas-gsi kernel: [20258]   500 20258   776888    93602   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [25910]   500 25910    15050      209   3       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [26010]   500 26010     2879       95  15       0             0 tinyproxy
Apr 24 03:19:01 aidas-gsi kernel: [26513]     0 26513    25530      204  12       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [26517]   500 26517    25530      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29203]     0 29203    25016      164  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29207]   500 29207    25534      158   6       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 1105]   500  1105    27123      143   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17163]     0 17163    25016      190  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [17167]   500 17167    25534      185   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18690]     0 18690    25530      183   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18695]   500 18695    25530      138   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18696]   500 18696    27123      127   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [20802]     0 20802    25530      173   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20806]   500 20806    25530      142   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20807]   500 20807    27123      172   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 9697]     0  9697    25530      187   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9701]   500  9701    25563      184  10       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9703]   500  9703    27123      132   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [21737]   500 21737    29350      360  10       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [21860]   500 21860   166837     1623   5       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21861]   500 21861   208277     3415   0       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21868]   500 21868   470786     1884   2       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [20297]   500 20297   731145    44730   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [23452]     0 23452    47361      305   5       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [23453]     0 23453    25240       86   2       0             0 tail
Apr 24 03:19:01 aidas-gsi kernel: [23841]   500 23841    27123      243   3       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [  345]   500   345    27156      311  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 4825]   500  4825   682055    25908   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [16950]   500 16950    26921      257   1       0             0 watch
Apr 24 03:19:01 aidas-gsi kernel: [10822]   500 10822  1060110   432573   3       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [29109]   500 29109    27186      482   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 3754]   500  3754    27123      375   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11083]   500 11083    28030      216   3       0             0 HTTPD@8115
Apr 24 03:19:01 aidas-gsi kernel: [11087]   500 11087    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11089]   500 11089    36731     3211  12       0             0 tclsh8.5_copy2
Apr 24 03:19:01 aidas-gsi kernel: [11097]   500 11097    28030      207   1       0             0 TapeServer
Apr 24 03:19:01 aidas-gsi kernel: [11104]   500 11104     5398     1567  10       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [11105]   500 11105    25231      134  12       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11106]   500 11106     4323      187   6       0             0 stats
Apr 24 03:19:01 aidas-gsi kernel: [11107]   500 11107     4918      219   5       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11108]   500 11108     4917      194   9       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11109]   500 11109     4917      194  12       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11110]   500 11110     4630      214   0       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11111]   500 11111     4630      182   9       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11114]   500 11114    28030      217  12       0             0 NewMerger
Apr 24 03:19:01 aidas-gsi kernel: [11124]   500 11124   404629     8857   0       0             0 master64
Apr 24 03:19:01 aidas-gsi kernel: [11125]   500 11125    25231      135   9       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11126]   500 11126    19761    17318  13       0             0 statrate64
Apr 24 03:19:01 aidas-gsi kernel: [11127]   500 11127    35968    11059   2       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11128]   500 11128    35968    11498  11       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11129]   500 11129    35968    11581  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11130]   500 11130    35968    12366   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11131]   500 11131    35968    10031   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11132]   500 11132    35968    10689   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11133]   500 11133    35968    11896   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11134]   500 11134    35968    10510  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11135]   500 11135    35968    15679  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11136]   500 11136    35968    14623  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11137]   500 11137    35968    12790  15       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11138]   500 11138    35968    11683   0       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11139]   500 11139    35968     8773   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11140]   500 11140    35968     8774  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11141]   500 11141    35968     8772  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11142]   500 11142    35968     8773  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11152]   500 11152   307392    47667   5       0             0 merge64
Apr 24 03:19:01 aidas-gsi kernel: [11200]   500 11200   723346    37902   5       0             0 file:// Content
Apr 24 03:19:01 aidas-gsi kernel: [11338]   500 11338    28030      216   0       0             0 HTTPD
Apr 24 03:19:01 aidas-gsi kernel: [11342]   500 11342    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11344]   500 11344    46965    14788   3       0             0 tclsh8.5
Apr 24 03:19:01 aidas-gsi kernel: [12835]   500 12835   766232    71066   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [24071]   500 24071    27123      409   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [24347]   500 24347    28030      231   4       0             0 datarelaymbs
Apr 24 03:19:01 aidas-gsi kernel: [24351]   500 24351     7751     1313  10       0             0 DataRelayMBS
Apr 24 03:19:01 aidas-gsi kernel: [  407]    89   407    20280      470  13       0             0 pickup
Apr 24 03:19:01 aidas-gsi kernel: [ 1586]     0  1586     4238      156  13       0             0 anacron
Apr 24 03:19:01 aidas-gsi kernel: [ 1876]     0  1876    25231      144  10       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: Out of memory: Kill process 18393 (Xorg) score 523 or sacrifice child
Apr 24 03:19:01 aidas-gsi kernel: Killed process 18393, UID 0, (Xorg) total-vm:13906540kB, anon-rss:9558800kB, file-rss:92712kB
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11471" (uid=500 pid=1884 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11473" (uid=500 pid=1887 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:45 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:19:47 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1961 of process 1961 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Apr 24 03:19:48 aidas-gsi pulseaudio[1961]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1965 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1966 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1967 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11485" (uid=500 pid=1981 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11487" (uid=500 pid=1983 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: DNS-SD initialization failed: Daemon not running
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: init gpgme version 1.1.8
Apr 24 03:30:44 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2243 of process 2243 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:45 aidas-gsi pulseaudio[2243]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2301 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2302 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2303 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2325 of process 2325 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2325]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2326 of process 2326 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2326]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11506" (uid=500 pid=2328 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2418 of process 2418 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2418]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11508" (uid=500 pid=2420 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:51 aidas-gsi pulseaudio[2243]: ratelimit.c: 25 events suppressed
Apr 24 03:30:51 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr 24 03:33:11 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11521" (uid=500 pid=3005 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11524" (uid=500 pid=3012 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11525" (uid=500 pid=3015 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11527" (uid=500 pid=3018 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11529" (uid=500 pid=3022 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11531" (uid=500 pid=3024 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:13 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11533" (uid=500 pid=3100 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:37 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:40:46 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:41:22 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:19 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:43:31 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:47:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.7 (192.168.11.99) from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.4 (192.168.11.99) from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.3 (192.168.11.99) from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.8 (192.168.11.99) from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.2 (192.168.11.99) from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.1 (192.168.11.99) from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.6 (192.168.11.99) from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.5 (192.168.11.99) from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:816 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:952 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:884 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:895 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:892 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:839 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:897 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.8
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.1
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.7
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.3
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.4
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.2
Apr 24 03:51:04 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.5
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:29 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:06 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.9 (192.168.11.99) from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.10 (192.168.11.99) from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.11 (192.168.11.99) from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.12 (192.168.11.99) from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:814 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:852 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:694 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:770 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.9
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.10
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.12
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.11
Apr 24 03:52:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)

AIDA PC crashe completely at ~3:30am CET.

Tom was called and he Restarted the AIDA system at ~4am with Run 52.

Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.

system wide checks:

Clock status:
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit

         Base         Current     Difference
aida06 fault      0x679f :      0x67a4 :      5  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error page not working !!

Memory information:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     15      4      6      4      3      2      4      2      3      6   : 36420
aida02 :     36     25     24      6      0      4      2      3      2      3      4   : 28056
aida03 :     17      7      4      3      3      5      2      2      3      3      6   : 36444
aida04 :      4      5      4      5      3      4      2      3      3      3      6   : 36824
aida05 :     23      9      4      2      3      3      1      3      3      3      6   : 36452
aida06 :     26      8      5      1      3      4      2      2      3      3      6   : 36312
aida07 :     21      9      3      3      4      5      1      2      3      3      6   : 36268
aida08 :     27      9      2      2      2      4      1      3      3      3      6   : 36500
aida09 :     13      4      3      3      1      2      4      2      3      3      6   : 36388
aida10 :      1     10      0      3      1      2      3      3      2      3      6   : 35572
aida11 :     23     13      8      3      3      4      3      3      2      3      6   : 36196
aida12 :     19      7      4      3      2      3      2      3      3      3      6   : 36644

21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378

16:00 taking over from Muneerah

16:15 - temp, ucesb, database, stats etc. look ok.

16:41 - Nic restarting MBS

17:05 - temp, ucesb, database, stats etc. look ok.

17:43 - temp, ucesb, database, stats etc. look ok.

18.10 - system wide checks.  All same as previous checks - eg elog 264 + 266

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
 

Check FPGA Timestamp Error - HTML errors

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      7      4      7      4      3      2      4      2      3      6   : 36388
aida02 :      1      2      5      2      0      4      3      4      3      4      3   : 27044
aida03 :     28      4      4      4      4      4      2      2      3      3      6   : 36432
aida04 :      5      5      2      6      2      3      3      3      3      3      6   : 36892
aida05 :     24      4      7      3      3      3      2      2      3      3      6   : 36240
aida06 :     21      6      6      3      3      4      2      2      3      3      6   : 36356
aida07 :     15      7      4      3      3      4      2      2      3      3      6   : 36308
aida08 :     23      6      7      1      3      4      1      3      3      3      6   : 36572
aida09 :      1      8      2      1      1      2      4      2      3      3      6   : 36292
aida10 :      3      4      6      2      2      2      3      3      2      3      6   : 35660
aida11 :     15      7      4      1      2      3      2      4      2      3      6   : 36052
aida12 :     25     13      5      2      2      3      2      3      3      3      6   : 36700
 

18:45 - temp, ucesb, database, stats etc. look ok.

19:03 - temp, ucesb, database, stats etc. look ok.

19:26  - temp, ucesb, database, stats etc. look ok.

TapeData (df -h) - 7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7

19:30 - beam being shared with R3B, implantation rates fluctuation by~  50% back to normal 19:40

19:50 system checks - all as previous

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

    
         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Check FPGA Timestamp Error - HTML errors

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     25      6      4      7      4      3      2      4      2      3      6   : 36404
aida02 :     20      7      5      1      0      4      2      3      2      3      4   : 27384
aida03 :     28      6      3      5      3      5      2      2      3      3      6   : 36528
aida04 :     19      8      4      6      2      4      4      2      3      3      6   : 36876
aida05 :      7      3      8      3      4      3      1      3      3      3      6   : 36500
aida06 :     25      8      4      3      3      4      2      3      3      3      6   : 36868
aida07 :     18      7      3      4      3      4      2      2      3      3      6   : 36336
aida08 :     16     10      6      2      3      3      1      3      3      3      6   : 36464
aida09 :      0      3      1      1      1      3      4      2      3      3      6   : 36360
aida10 :      8      1     10      0      2      2      2      4      2      3      6   : 35912
aida11 :      2      2      2      3      3      4      3      3      2      3      6   : 35928
aida12 :     12      7      4      1      3      3      2      3      3      3      6   : 36616
 

14.15 System wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors - reports 'server internal error'

14.16 DSSSD bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachment 2
      good events stats - attachment 3

14.22 DAQ continues file S460/R51_205
      beam intensity 2.2e+09/s


15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6


Clock check

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC check

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



White Rabbit


		 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




FPGA
Loge page error!

Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      6      2      1      2      3      2      4      2      3      6   : 36044
aida02 :     15      3      2      2      0      3      2      3      2      3      4   : 27188
aida03 :     29      4      0      3      2      4      2      2      3      3      6   : 36212
aida04 :      4      6      4      4      2      3      3      3      3      3      6   : 36864
aida05 :     19      5      4      2      3      3      1      3      3      3      6   : 36404
aida06 :     24      9      5      2      3      3      3      2      3      3      6   : 36472
aida07 :     20      4      9      0      2      3      2      2      3      3      6   : 36096
aida08 :     22      9      4      3      1      3      1      3      3      3      6   : 36352
aida09 :     20      7      3      2      2      3      3      2      3      3      6   : 36344
aida10 :     25      9      9      5      2      3      2      3      3      3      6   : 36828
aida11 :     24     11      2      0      1      3      1      3      3      3      6   : 36248
aida12 :     26      5      4      2      1      3      2      2      3      3      6   : 36048

Overnight aida02 crashed and restarted itself. A power cycle of all FEE64s was required to recover.

The system log appears to show the aida02 embedded processor crashed (note log times are UTC+1
for some reason).

Apr 21 00:41:21 aida02 kernel: xaida: open:
Apr 23 02:09:41 aida02 kernel: Unable to handle kernel paging request for data at address 0x00200200
Apr 23 02:09:41 aida02 kernel: Faulting instruction address: 0xc0091dd8
Apr 23 02:09:41 aida02 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 23 02:09:41 aida02 kernel: PREEMPT Xilinx Virtex440
Apr 23 02:09:41 aida02 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 23 02:09:42 aida02 kernel: NIP: c0091dd8 LR: c0091e40 CTR: 00000000
Apr 23 02:09:42 aida02 kernel: REGS: c6835e60 TRAP: 0300   Not tainted  (2.6.31)
Apr 23 02:09:42 aida02 kernel: MSR: 00021000 <ME,CE>  CR: 24000088  XER: 00000000
Apr 23 02:09:42 aida02 kernel: DEAR: 00200200, ESR: 00800000
Apr 23 02:09:42 aida02 kernel: TASK = c6824880[5] 'events/0' THREAD: c6834000
Apr 23 02:09:42 aida02 kernel: GPR00: c0091e40 c6835f10 c6824880 00000001 00000003 c038c3a8 00000000 c0498000
Apr 23 02:09:42 aida02 kernel: GPR08: 00000008 00100100 00000051 00200200 00485e6c 00005aa8 4805de90 00000001
Apr 23 02:09:42 aida02 kernel: GPR16: 00000000 00000000 4801d000 00000003 c031aef4 c02fb040 c02fb008 00000002
Apr 23 02:09:42 aida02 kernel: GPR24: c6834000 00100100 00200200 c680ae00 00000001 c680daf0 c680dae0 c6940020
Apr 23 02:09:42 aida02 kernel: NIP [c0091dd8] drain_freelist+0x54/0x110
Apr 23 02:09:42 aida02 kernel: LR [c0091e40] drain_freelist+0xbc/0x110
Apr 23 02:09:42 aida02 kernel: Call Trace:
Apr 23 02:09:42 aida02 kernel: [c6835f10] [c0091e40] drain_freelist+0xbc/0x110 (unreliable)
Apr 23 02:09:42 aida02 kernel: [c6835f40] [c0093158] cache_reap+0x104/0x138
Apr 23 02:09:42 aida02 kernel: [c6835f60] [c0048908] worker_thread+0x140/0x204
Apr 23 02:09:42 aida02 kernel: [c6835fc0] [c004cfc8] kthread+0x78/0x7c
Apr 23 02:09:42 aida02 kernel: [c6835ff0] [c000e140] kernel_thread+0x4c/0x68
Apr 23 02:09:42 aida02 kernel: Instruction dump:
Apr 23 02:09:43 aida02 kernel: 3ba40010 7f80e800 419e00d4 3f200010 3f400020 54380024 63390100 635a0200
Apr 23 02:09:43 aida02 kernel: 3b800000 48000068 813f0000 817f0004 <912b0000> 91690004 933f0000 935f0004
Apr 23 02:09:43 aida02 kernel: ---[ end trace 6baa5b82e4ef4e92 ]---
Apr 23 02:09:43 aida02 kernel: note: events/0[5] exited with preempt_count 1
Apr 23 02:13:39 aida02 syslogd 1.4.2: restart.
Apr 23 02:13:39 aida02 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Apr 23 02:13:39 aida02 kernel: Using Xilinx Virtex440 machine description
Apr 23 02:13:39 aida02 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011

08:30

System Checks

Clock Status error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 1

Temperatures - See attachment 2

Bias & Leakage Currents - See attachment 3

10:02

System Checks:

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration Error:

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 4

Temperatures - See attachment 5

Bias & Leakage Currents - See attachment 6

10:45

System Checks

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration check Error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 7

Temperatures - See attachment 8

Bias & Leakage Currents - See attachment 9

Took over from TD ... All is well.

00:45 System Wide Checks

Clock: 12 passed, 0 failed.

ADC Calibration: Passed 12, 0 failed.

White Rabbit Decoder: Passed 4,failed 8. - See screenshot.

FPGA Passed 11, failed 1. - See screenshot.

FEE64 Linux: See screenshot.

-----

02:49 run closed. Checking beam.

03:00 File 55 opened.

03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.

File 56 opened at 4:24.

---------

04:30 System Wide Checks (See screenshots)

White Rabbit Decoder: Passed 12, failed 0.

FPGA: page not loading properly.

----

05:37 System wide checks. See screenshots

Clock: Status: 11 passed. 1 fail.

ADC calibration: Passed 9, failed 3.

White Rabbit decoder: Passed 12, failed 0.

-----

07:32 System wide checks (See screenshots).

16.20 zero all histograms, statistics & merger statistics

      most recent merger error messages

MERGE Data Link (29984): bad timestamp  6 3 0xc1ac7f1c 0x0dce7672 0x00002cda5dce7672 0x16782cda5dce7672 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc18b7e62 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ebc 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1b67ecb 0x0dcf0ae2 0x00002cda5dcf0ae2 0x16782cda5dcf0ae2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ea3 0x0dcf12b2 0x00002cda5dcf12b2 0x16782cda5dcf12b2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1867fba 0x0b2ee142 0x00002dc9eb2ee142 0x16782dc9eb2ee142 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0x86800889 0x0ad28158 0x00002dc9eb2ee142 0x16782dc9ead28158 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0xc1be7e6e 0x0b2ef0e2 0x00002dc9eb2ef0e2 0x16782dc9eb2ef0e2 0x16782dc9fad2c172
MERGE Data Link (29984): bad timestamp  6 3 0xc1b08006 0x0b2f5e42 0x00002dc9eb2f5e42 0x16782dc9eb2f5e42 0x16782dc9fad34502
MERGE Data Link (29984): bad timestamp  6 3 0xc1807de7 0x0ad3c342 0x00002dc9eb2f6de2 0x16782dc9ead3c342 0x16782dc9fad34502
MERGE Statistics counters now cleared

16.21 merger server terminal reset & clear

      TapeData -> /media/ThirdDRive/TapeData

[npg@aidas-gsi bin64]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   13G   34G  28% /
tmpfs                 7.9G  559M  7.3G   7% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   79G  302G  21% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  6.3T  605G  92% /media/SecondDrive
/dev/sdc1             7.2T  3.6T  3.3T  52% /media/ThirdDrive

      at current rate (10.5Mb/s) we have 3.6d remaining 
      
17.40 *all* system wide checks OK *except* FPGA timestamp errors

		 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x51 : 	 81  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

17.43 FEE64 temps OK - attachment 1
      statistics - attachments 2-8
      per FEE64 rate spectra - attachments 9 & 10
      1.8.L spectra - attachments 11-14
      1.8.H spectra - attachments 15-16
      1.8.W spectra 20us FSR - attachments 17-18
      1.8.W spectra 200us, 2ms & 20ms FSR - attachments 19-21, 30-32
      merger, tape server & merger statistics 19-25
      ucesb - attachment 26
      Grafana DSSD biases & leakage currents - attachment 27
      taperserver & tapeserver/merger server terminals - no timewarp errors since c. 15.25 - attachment 28
      DSSSD biases & leakage currents - attachment 29


22.17 *all* system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x3612 : 	 0x3613 : 	 1  
aida02 fault 	 0x523c : 	 0x523d : 	 1  
aida03 fault 	 0xc5cd : 	 0xc5ce : 	 1  
aida04 fault 	 0x14f1 : 	 0x14f2 : 	 1  
aida05 fault 	 0x9a4e : 	 0x9a4f : 	 1  
aida06 fault 	 0xfdd2 : 	 0xfdd3 : 	 1  
aida07 fault 	 0x4a0 : 	 0x4a1 : 	 1  
aida08 fault 	 0xc796 : 	 0xc797 : 	 1  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x57 : 	 87  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors

MERGE Data Link (29979): bad timestamp  1 3 0xc05080ef 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc06f80d7 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0778222 0x0c88b272 0x000047aa2c88b272 0x167847aa2c88b272 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0598098 0x0c88e152 0x000047aa2c88e152 0x167847aa2c88e152 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc042815b 0x0c894eb2 0x000047aa2c894eb2 0x167847aa2c894eb2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc05980eb 0x0c899502 0x000047aa2c899502 0x167847aa2c899502 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc075813c 0x0c89e322 0x000047aa2c89e322 0x167847aa2c89e322 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc04280d5 0x0c89f2c2 0x000047aa2c89f2c2 0x167847aa2c89f2c2 0x167847aa2c8a34d2
MERGE Actor (29995): Working with 0 from 12 data sources
MERGE Actor (29995): Working with 0 from 12 data sources

22.20 DSSSD biases & leakage currents - attachment 33
      FEE64 temps OK - attachment 34
      statistics - attachments 35
      merger statistics 36

      

12:00 Taking over from Liam, all is well.

12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz

12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they 
need to check and will need apprx. 15 min.")

12:30
ucesb ok (no beam)
DB No faults
Stats ok.

12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.

13:00
We are taking intermittent beam while they work on the FRS

DB No faults
ucesb ok  elog:261/1
Current ok elog:261/2
Stats ok  elog:261/3
Temps ok elog:261/4

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x46 : 	 70  
FPGA Timestamp error counter test result: Passed 11, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      6      5      3      2      2      2      3      2      3      7   : 39580
aida02 :     21      7      4      2      2      3      2      3      2      3      7   : 39692
aida03 :     20     10      3      3      0      3      2      3      2      3      7   : 39600
aida04 :      8      7      4      5      2      4      3      2      2      3      7   : 39608
aida05 :     17     14      1      2      3      3      4      2      2      3      7   : 39748
aida06 :     17     10      2      3      2      2      4      2      2      3      7   : 39572
aida07 :     27     10      8      7      2      3      3      2      2      3      7   : 39708
aida08 :      9      6      2      3      2      1      3      3      2      3      7   : 39636
aida09 :      1      2      1      4      0      3      3      2      2      3      7   : 39204
aida10 :      5      8      4      4      2      2      1      3      2      3      7   : 39316
aida11 :      6      4      5      3      2      3      2      2      2      3      7   : 39144
aida12 :     15      5      8      1      0      4      1      3      2      3      7   : 39428


13:32
ucesb ok
DB No faults
Stats ok

14:01
ucesb ok
DB No faults
Stats ok

14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
      cat /var/log/messages | grep /MIDAS - attachments 6 & 7
      Grafana DSSSD bias & leakage currents for last 7 days - attachment 8

14:55
DB No faults 
ucesb ok
Current ok elog:261/9
Stats ok  elog:261/10
Temps ok elog:261/11

FPGA Timestamp Check Error:
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 11, Failed 1


15:30
ucesb ok
DB No faults
Stats ok

15:59
ucesb ok elog:261/12
DB No faults
Stats ok