ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 18 of 35

Full | Summary | Threaded | Hide attachments

687 Entries

Goto page Previous 1, 2, 3 ... 17, 18, 19 ... 33, 34, 35 Next

ID	Date	Author	Subject
356	Fri Jun 4 09:47:44 2021	TD	aida15 - system console output - 00.02 Friday 4 June reboot
03/23:58:55\|kernel BUG at mm/slab.c:2974! 03/23:58:55\|Oops: Exception in kernel mode, sig: 5 [#1] 03/23:58:55\|PREEMPT Xilinx Virtex440 03/23:58:55\|Modules linked in: aidamem xdriver xh_spidev_register 03/23:58:55\|NIP: c009211c LR: c00920b0 CTR: c0038e70 03/23:58:55\|REGS: c631fc60 TRAP: 0700 Not tainted (2.6.31) 03/23:58:55\|MSR: 00021000 <ME,CE> CR: 22022048 XER: 00000000 03/23:58:56\|TASK = c627a0c0[375] 'AidaExecV9' THREAD: c631e000 03/23:58:56\|GPR00: 00000001 c631fd10 c627a0c0 c680daf0 00000020 00000001 c6940020 00000000 03/23:58:56\|GPR08: fffffff6 00010002 c680e400 c631e000 22008022 1005520c c03a0000 00000020 03/23:58:56\|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 03/23:58:56\|GPR24: 00100100 c631e000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400 03/23:58:56\|NIP [c009211c] cache_alloc_refill+0x130/0x608 03/23:58:56\|LR [c00920b0] cache_alloc_refill+0xc4/0x608 03/23:58:56\|Call Trace: 03/23:58:56\|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable) 03/23:58:56\|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc 03/23:58:56\|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8 03/23:58:56\|[c631fdb0] [c0042938] __send_signal+0x78/0x260 03/23:58:56\|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c 03/23:58:56\|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c 03/23:58:56\|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30 03/23:58:56\|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240 03/23:58:56\|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c 03/23:58:56\|[c631feb0] [c0040bf4] update_process_times+0x30/0x70 03/23:58:57\|[c631fed0] [c005a000] tick_periodic+0x34/0xe8 03/23:58:57\|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120 03/23:58:57\|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c 03/23:58:57\|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18 03/23:58:57\|Instruction dump: 03/23:58:57\|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 03/23:58:57\|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 03/23:58:57\|Kernel panic - not syncing: Fatal exception in interrupt 03/23:58:57\|Call Trace: 03/23:58:57\|[c631fab0] [c0005de8] show_stack+0x44/0x16c (unreliable) 03/23:58:57\|[c631faf0] [c00345bc] panic+0x94/0x168 03/23:58:57\|[c631fb40] [c000bd44] die+0x178/0x18c 03/23:58:57\|[c631fb60] [c000c000] _exception+0x164/0x1b4 03/23:58:57\|[c631fc50] [c000e978] ret_from_except_full+0x0/0x4c 03/23:58:57\|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 03/23:58:57\|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc 03/23:58:57\|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8 03/23:58:57\|[c631fdb0] [c0042938] __send_signal+0x78/0x260 03/23:58:57\|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c 03/23:58:58\|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c 03/23:58:58\|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30 03/23:58:58\|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240 03/23:58:58\|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c 03/23:58:58\|[c631feb0] [c0040bf4] update_process_times+0x30/0x70 03/23:58:58\|[c631fed0] [c005a000] tick_periodic+0x34/0xe8 03/23:58:58\|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120 03/23:58:58\|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c 03/23:58:58\|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18 03/23:58:58\|Rebooting in 180 seconds.. ISOL Version 1.00 Date 9th January 2017 Flash base address=FC000000 Set Flash to ASync Mode XST_SUCCESS\| Finished copying zImage to RAM 04/00:01:59\| Found 0 errors checking kernel image 04/00:02:00\|VHDL version number 0X03350706 Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012 Starting LMK 3200 setup 04/00:02:00\| Setting LMK03200 to standard clock settings -- External Clock 23Nov15 .... SPI Base Address=0x81400000 clk_control_reg=0x4 04/00:02:00\|Next step is SPIconfig Control 32(0x81400000)=0x180 SlaveSel(0x81400000)=0x3 Ctrl(0x81400000)=0xE6 Ctrl(0x81400000)=0x86 04/00:02:00\|SPIconfig done now to set up the LMK3200 registers 04/00:02:00\|LMK #0 : regInit[0]=0x80000000 04/00:02:00\|LMK #0 : regInit[1]=0x10070600 04/00:02:00\|LMK #0 : regInit[2]=0x60601 04/00:02:00\|LMK #0 : regInit[3]=0x60602 04/00:02:00\|LMK #0 : regInit[4]=0x60603 04/00:02:00\|LMK #0 : regInit[5]=0x70624 04/00:02:00\|LMK #0 : regInit[6]=0x70605 04/00:02:00\|LMK #0 : regInit[7]=0x70606 04/00:02:00\|LMK #0 : regInit[8]=0x70627 04/00:02:00\|LMK #0 : regInit[9]=0x10000908 04/00:02:00\|LMK #0 : regInit[10]=0xA0022A09 04/00:02:00\|LMK #0 : regInit[11]=0x82800B 04/00:02:01\|LMK #0 : regInit[12]=0x28C800D 04/00:02:01\|LMK #0 : regInit[13]=0x830020E 04/00:02:01\|LMK #0 : regInit[14]=0xC800180F Calibrate completed at 941 counts Setting Clock Control =0x0000000B, to set GOE and sync bit Ctrl @ SPIstop (0x81400000)=0x186 Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004 04/00:02:01\| Finished Clock setup LMK03200 completed LMK 3200 setup Loaded all four ASICs with default settings Setting the ADCs into calibration mode 04/00:02:01\| Control 32(0x81400400)=0x180 SlaveSel(0x81400400)=0xFF Ctrl(0x81400400)=0xE6 Ctrl(0x81400400)=0x86 Init : Config of AD9252 SPI ok 04/00:02:01\| Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised Cal DCMs not locked ADC calibrate failed Jumping to kernel simpleboot... 04/00:02:01\| zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0) Allocating 0x3b78cc bytes for kernel ... gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes 04/00:02:05\| Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M Finalizing device tree... flat tree at 0xbd1300 Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 04/00:02:11\|Using Xilinx Virtex440 machine description 04/00:02:12\|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011 04/00:02:12\|Zone PFN ranges: 04/00:02:12\| DMA 0x00000000 -> 0x00007000 04/00:02:12\| Normal 0x00007000 -> 0x00007000 04/00:02:12\|Movable zone start PFN for each node 04/00:02:12\|early_node_map[1] active PFN ranges 04/00:02:12\| 0: 0x00000000 -> 0x00007000 04/00:02:12\|MMU: Allocated 1088 bytes of context maps for 255 contexts 04/00:02:12\|Built 1 zonelists in Zone order, mobility grouping on. Total pages: 28448 04/00:02:12\|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M 04/00:02:12\|PID hash table entries: 512 (order: 9, 2048 bytes) 04/00:02:12\|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) 04/00:02:12\|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) 04/00:02:12\|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init) 04/00:02:12\|Kernel virtual memory layout: 04/00:02:12\| * 0xffffe000..0xfffff000 : fixmap 04/00:02:12\| * 0xfde00000..0xfe000000 : consistent mem 04/00:02:12\| * 0xfde00000..0xfde00000 : early ioremap 04/00:02:13\| * 0xd1000000..0xfde00000 : vmalloc & ioremap 04/00:02:13\|NR_IRQS:512 04/00:02:13\|clocksource: timebase mult[a00000] shift[22] registered 04/00:02:13\|Console: colour dummy device 80x25 04/00:02:13\|Mount-cache hash table entries: 512 04/00:02:13\|NET: Registered protocol family 16 04/00:02:13\|PCI: Probing PCI hardware 04/00:02:13\|bio: create slab <bio-0> at 0 04/00:02:13\|NET: Registered protocol family 2 04/00:02:13\|IP route cache hash table entries: 1024 (order: 0, 4096 bytes) 04/00:02:13\|TCP established hash table entries: 4096 (order: 3, 32768 bytes) 04/00:02:13\|TCP bind hash table entries: 4096 (order: 2, 16384 bytes) 04/00:02:13\|TCP: Hash tables configured (established 4096 bind 4096) 04/00:02:13\|TCP reno registered 04/00:02:13\|NET: Registered protocol family 1 04/00:02:13\|ROMFS MTD (C) 2007 Red Hat, Inc. 04/00:02:13\|msgmni has been set to 214 04/00:02:13\|io scheduler noop registered 04/00:02:13\|io scheduler anticipatory registered 04/00:02:13\|io scheduler deadline registered 04/00:02:13\|io scheduler cfq registered (default) 04/00:02:13\|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 04/00:02:13\|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550 04/00:02:14\|console [ttyS0] enabled 04/00:02:14\|brd: module loaded 04/00:02:14\|loop: module loaded 04/00:02:14\|Device Tree Probing 'ethernet' 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode. 04/00:02:14\|XLlTemac: DCR address: 0x80 04/00:02:14\|XLlTemac: buffer descriptor size: 32768 (0x8000) 04/00:02:14\|XLlTemac: Allocating DMA descriptors with kmalloc 04/00:02:14\|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000 04/00:02:14\|XTemac: PHY detected at address 7. 04/00:02:14\|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17 04/00:02:14\|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\| Intel/Sharp Extended Query Table at 0x010A 04/00:02:14\|Using buffer write method 04/00:02:15\|cfi_cmdset_0001: Erase suspend on write enabled 04/00:02:15\|cmdlinepart partition parsing not available 04/00:02:15\|RedBoot partition parsing not available 04/00:02:15\|Creating 5 MTD partitions on "fc000000.flash": 04/00:02:15\|0x000000000000-0x000000500000 : "golden_firmware" 04/00:02:15\|0x000000500000-0x000000800000 : "golden_kernel" 04/00:02:15\|0x000000800000-0x000000d00000 : "user_firmware" 04/00:02:15\|0x000000d00000-0x000000fe0000 : "user_kernel" 04/00:02:15\|0x000000fe0000-0x000001000000 : "env_variables" 04/00:02:15\|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20 04/00:02:15\|SPI: XIlinx spi: bus number now 32766 04/00:02:15\|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21 04/00:02:15\|SPI: XIlinx spi: bus number now 32765 04/00:02:15\|mice: PS/2 mouse device common for all mice 04/00:02:15\|Device Tree Probing 'i2c' 04/00:02:15\| #0 at 0x81600000 mapped to 0xD1030000, irq=22 04/00:02:15\|at24 0-0050: 1024 byte 24c08 EEPROM (writable) 04/00:02:15\|TCP cubic registered 04/00:02:15\|NET: Registered protocol family 17 04/00:02:15\|RPC: Registered udp transport module. 04/00:02:15\|RPC: Registered tcp transport module. 04/00:02:16\|eth0: XLlTemac: Options: 0x3fa 04/00:02:16\|eth0: XLlTemac: allocating interrupt 19 for dma mode tx. 04/00:02:16\|eth0: XLlTemac: allocating interrupt 18 for dma mode rx. 04/00:02:16\|eth0: XLlTemac: speed set to 1000Mb/s 04/00:02:18\|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4 04/00:02:18\|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254 04/00:02:18\|Sending DHCP requests ., OK 04/00:02:19\|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.15 04/00:02:20\|IP-Config: Complete: 04/00:02:20\| device=eth0, addr=192.168.11.15, mask=255.255.255.0, gw=255.255.255.255, 04/00:02:20\| host=aida15, domain=dl.ac.uk, nis-domain=nuclear.physics, 04/00:02:20\| bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15 04/00:02:20\|Looking up port of RPC 100003/2 on 192.168.11.99 04/00:02:20\|Looking up port of RPC 100005/1 on 192.168.11.99 04/00:02:20\|VFS: Mounted root (nfs filesystem) on device 0:12. 04/00:02:20\|Freeing unused kernel memory: 168k init INIT: version 2.86 booting 04/00:02:20\|Starting sysinit... 04/00:02:21\| Welcome to DENX & STFC Daresbury Embedded Linux Environment 04/00:02:22\| Press 'I' to enter interactive startup. 04/00:02:22\|Setting clock (utc): Thu Jun 3 23:02:22 BST 2021 [ OK ] 04/00:02:22\|Building the cache [ OK ] 04/00:02:22\|Setting hostname aida15: [ OK ] 04/00:02:23\|Mounting local filesystems: [ OK ] 04/00:02:25\|Enabling /etc/fstab swaps: [ OK ] 04/00:02:27\|Finishing sysinit... INIT: Entering runlevel: 3 04/00:02:30\|Entering non-interactive startup 04/00:02:30\|FATAL: Module ipv6 not found. 04/00:02:31\|Bringing up loopback interface: [ OK ] 04/00:02:33\|FATAL: Module ipv6 not found. 04/00:02:34\|Starting system logger: [ OK ] 04/00:02:34\|Starting kernel logger: [ OK ] 04/00:02:35\|Starting rpcbind: [ OK ] 04/00:02:36\|Mounting NFS filesystems: [ OK ] 04/00:02:37\|Mounting other filesystems: [ OK ] 04/00:02:37\|Starting xinetd: [ OK ] 04/00:02:38\|Starting midas: Starting MIDAS Data Acquisition for aida15 04/00:02:38\|xaida: device parameters: base=0x81000000 size=0x200000 04/00:02:44\|Trying to free nonexistent resource <0000000081000000-00000000811fffff> 04/00:02:44\|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000 04/00:02:44\|xaida: driver assigned major number 254 04/00:02:44\|Trying to free nonexistent resource <0000000007000000-0000000007ffffff> 04/00:02:50\|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000 04/00:02:50\|AIDAMEM: aidamem: driver assigned major number 253 04/00:02:50\|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31 04/00:02:56\|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx 04/00:02:57\|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx 04/00:02:57\|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin 04/00:02:57\|Computer Name = aida15; Temp Directory = /tmp/tcl361 04/00:03:01\|package limit is not available: can't find package limit 04/00:03:04\|Running with default file descriptor limit 04/00:03:04\|package setuid is not available: can't find package setuid 04/00:03:05\|Running as user 0 group 0 04/00:03:05\| 04/00:03:06\|AIDA Data Acquisition Program Release 9_10.Apr 3 2019_11:34:31 starting 04/00:03:06\| 04/00:03:06\|Built without pthreads 04/00:03:06\| 04/00:03:06\|Creating NetVars 04/00:03:06\|Output buffer length = 65xaida: open: 04/00:03:06\|504; format option = 4AIDAMEM: aidamem_open: 04/00:03:06\|; transfer option = 3 04/00:03:06\|EB transfer option = 3 04/00:03:06\|NetVars created and initialised 04/00:03:06\|Statistics thread starting 04/00:03:06\|Statistics thread created 04/00:03:06\|Stat/Rate creation thread starting 04/00:03:06\|Data Acquisition task has PID 375 04/00:03:06\|Stat/Rate creation thread created 04/00:03:06\|Hit/Rate creation thread starting 04/00:03:06\|Hit/Rate creation thread created 04/00:03:07\|AIDA Heartbeat thread starting 04/00:03:07\|Heartbeat thread created 04/00:03:07\|Installing signal handlers 04/00:03:07\|Done 04/00:03:07\|ModuleNum = 0 04/00:03:07\|Aida Initialise complete. AidaExecV9_10: Build Apr 3 2019_11:34:31. HDL version : 03350706 04/00:03:07\|Spectra table initialised 04/00:03:07\|AIDA Data Acquisition now all ready to start 04/00:03:07\|SIGBUS, SIGSEGV and SIGPIPE traps setup 04/00:03:07\|/debug user "debug" password " bjkhcfyuqis" 04/00:03:14\|httpd started on port 8015 04/00:03:14\| 04/00:03:14\|Cannot use /MIDAS/config/TclHttpd/aida15@8015/startup.tcl 04/00:03:14\|Custom startup from /MIDAS/config/TclHttpd/aida15/startup.tcl 04/00:03:14\|XAIDA Access package 1.0 04/00:03:16\|/XAIDAAccessServer 04/00:03:16\|XAD9252 Access package 1.0 04/00:03:16\|/XAD9252AccessServer 04/00:03:16\|/DataBaseAccessServer 04/00:03:16\|[ OK ] 04/00:03:16\|/NetVarService 04/00:03:17\|/SigTaskService 04/00:03:17\|Loaded MemSasAccess 04/00:03:17\|/SpectrumService 04/00:03:17\|loading tcl/AIDARunControl.tcl for namespace :: 04/00:03:17\| 04/00:03:17\|DENX ELDK version 4.2 build 2008-04-01 04/00:03:17\|Linux 2.6.31 on a ppc 04/00:03:17\| 04/00:03:17\|aida15 login: /DataAcquisitionControlServer 04/00:03:17\|DefineMessage unknown 04/00:03:17\|Run Control Server Implementation for AIDA 04/00:03:17\|RunControlServer loaded 04/00:03:17\|loading Html/RunControl/implementation.tcl 04/00:03:18\|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory 04/00:03:18\|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist 04/00:03:18\|Created UI registers 04/00:03:18\|RunControl loaded 04/00:03:19\|loading Html/AIDA/RunControl/implementation.tcl for namespace :: 04/00:03:19\|AIDA RunControl loaded 04/00:03:19\|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl 04/00:03:19\|do_GetState returned z=0 and 8 04/09:31:05\|do_GetState returned z=0 and 8 04/09:31:17\|do_GetState returned z=0 and 8 04/09:31:30\|do_GetState returned z=0 and 8 04/09:31:43\|do_GetState returned z=0 and 8 04/09:31:55\|do_GetState returned z=0 and 8 04/09:32:07\|do_GetState returned z=0 and 8 04/09:32:19\|do_GetState returned z=0 and 8 04/09:32:32\|do_GetState returned z=0 and 8 04/09:32:45\|do_GetState returned z=0 and 8 04/09:32:57\|do_GetState returned z=0 and 8 04/09:33:10\|do_GetState returned z=0 and 8 04/09:33:22\|do_GetState returned z=0 and 8 04/09:33:35\|do_GetState returned z=0 and 8 04/09:33:47\|do_GetState returned z=0 and 8 04/09:34:00\|do_GetState returned z=0 and 8 04/09:34:13\|do_GetState returned z=0 and 8 04/09:34:25\|do_GetState returned z=0 and 8 04/09:34:38\|do_GetState returned z=0 and 8 04/09:34:50\|do_GetState returned z=0 and 8 04/09:35:03\|do_GetState returned z=0 and 8 04/09:35:16\|do_GetState returned z=0 and 8 04/09:35:28\|do_GetState returned z=0 and 8 04/09:35:41\|do_GetState returned z=0 and 8 04/09:35:53\|do_GetState returned z=0 and 8 04/09:36:06\|do_GetState returned z=0 and 8 04/09:36:18\|do_GetState returned z=0 and 8 04/09:36:31\|do_GetState returned z=0 and 8 04/09:36:43\|do_GetState returned z=0 and 8 04/09:36:56\|do_GetState returned z=0 and 8 04/09:37:09\|do_GetState returned z=0 and 8 04/09:37:21\|do_GetState returned z=0 and 8 04/09:37:34\|do_GetState returned z=0 and 8 04/09:37:46\|do_GetState returned z=0 and 8 04/09:37:59\|do_GetState returned z=0 and 8 04/09:38:12\|do_GetState returned z=0 and 8 04/09:38:24\|do_GetState returned z=0 and 8 04/09:38:37\|do_GetState returned z=0 and 8 04/09:38:49\|do_GetState returned z=0 and 8 04/09:39:02\|do_GetState returned z=0 and 8 04/09:39:15\|do_GetState returned z=0 and 8 04/09:39:27\|do_GetState returned z=0 and 8 04/09:39:40\|do_GetState returned z=0 and 8 04/09:39:52\|do_GetState returned z=0 and 8 04/09:40:05\|do_GetState returned z=0 and 8 04/09:40:18\|do_GetState returned z=0 and 8 04/09:40:30\|do_GetState returned z=0 and 8 04/09:40:43\|do_GetState returned z=0 and 8 04/09:40:55\|do_GetState returned z=0 and 8 04/09:41:08\|do_GetState returned z=0 and 8 04/09:41:21\|do_GetState returned z=0 and 8 04/09:41:33\|do_GetState returned z=0 and 8 04/09:41:46\|do_GetState returned z=0 and 8 04/09:41:58\|do_GetState returned z=0 and 8 04/09:42:11\|do_GetState returned z=0 and 8 04/09:42:23\|do_GetState returned z=0 and 8 04/09:42:36\|do_GetState returned z=0 and 8 04/09:42:48\|do_GetState returned z=0 and 8 04/09:43:01\|do_GetState returned z=0 and 8 04/09:43:14\|do_GetState returned z=0 and 8 04/09:43:26\|do_GetState returned z=0 and 8 04/09:43:39\|do_GetState returned z=0 and 8 04/09:43:51\|do_GetState returned z=0 and 8 04/09:44:04\|do_GetState returned z=0 and 8 04/09:44:17\|do_GetState returned z=0 and 8 04/09:44:29\|do_GetState returned z=0 and 8 04/09:44:42\|do_GetState returned z=0 and 8 04/09:44:54\|do_GetState returned z=0 and 8 04/09:45:07\|do_GetState returned z=0 and 8 04/09:45:19\|do_GetState returned z=0 and 8 04/09:45:32\|do_GetState returned z=0 and 8 04/09:45:44\|do_GetState returned z=0 and 8 04/09:45:57\|do_GetState returned z=0 and 8 04/09:46:09\|do_GetState returned z=0 and 8 04/09:46:22\|do_GetState returned z=0 and 8 04/09:46:34\|do_GetState returned z=0 and 8 04/09:46:47\|do_GetState returned z=0 and 8 04/09:47:00\|do_GetState returned z=0 and 8 04/09:47:12\|do_GetState returned z=0 and 8 04/09:47:25\|do_GetState returned z=0 and 8 04/09:47:37\|do_GetState returned z=0 and 8 04/09:47:50\|do_GetState returned z=0 and 8 04/09:48:03\|do_GetState returned z=0 and 8 04/09:48:15\|do_GetState returned z=0 and 8 04/09:48:28\|do_GetState returned z=0 and 8 04/09:48:40\|do_GetState returned z=0 and 8 04/09:48:53\|do_GetState returned z=0 and 8 04/09:49:05\|do_GetState returned z=0 and 8 04/09:49:18\|do_GetState returned z=0 and 8 04/09:49:31\|do_GetState returned z=0 and 8 04/09:49:43\|do_GetState returned z=0 and 8 04/09:49:56\|do_GetState returned z=0 and 8 04/09:50:08\|do_GetState returned z=0 and 8 04/09:50:21\|do_GetState returned z=0 and 8 04/09:50:33\|do_GetState returned z=0 and 8 04/09:50:46\|do_GetState returned z=0 and 8 04/09:50:58\|do_GetState returned z=0 and 8 04/09:51:11\|do_GetState returned z=0 and 8 04/09:51:23\|do_GetState returned z=0 and 8 04/09:51:36\|do_GetState returned z=0 and 8 04/09:51:49\|do_GetState returned z=0 and 8 04/09:52:01\|do_GetState returned z=0 and 8 04/09:52:14\|do_GetState returned z=0 and 8 04/09:52:26\|do_GetState returned z=0 and 8 04/09:52:39\|do_GetState returned z=0 and 8 04/09:52:51\|do_GetState returned z=0 and 8 04/09:53:04\|do_GetState returned z=0 and 8 04/09:53:16\|do_GetState returned z=0 and 8 04/09:53:29\|do_GetState returned z=0 and 8 04/09:53:42\|do_GetState returned z=0 and 8 04/09:53:54\|do_GetState returned z=0 and 8 04/09:54:07\|do_GetState returned z=0 and 8 04/09:54:19\|do_GetState returned z=0 and 8 04/09:54:32\|do_GetState returned z=0 and 8 04/09:54:44\|do_GetState returned z=0 and 8 04/09:54:57\|do_GetState returned z=0 and 8 04/09:55:09\|do_GetState returned z=0 and 8 04/09:55:22\|do_GetState returned z=0 and 8 04/09:55:34\|do_GetState returned z=0 and 8 04/09:55:47\|do_GetState returned z=0 and 8 04/09:56:00\|do_GetState returned z=0 and 8 04/09:56:12\|do_GetState returned z=0 and 8 04/09:56:25\|do_GetState returned z=0 and 8 04/09:56:37\|do_GetState returned z=0 and 8 04/09:56:50\|do_GetState returned z=0 and 8 04/09:57:02\|do_GetState returned z=0 and 8 04/09:57:15\|do_GetState returned z=0 and 8 04/09:57:27\|do_GetState returned z=0 and 8 04/09:57:40\|do_GetState returned z=0 and 8 04/09:57:53\|do_GetState returned z=0 and 8 04/09:58:05\|do_GetState returned z=0 and 8 04/09:58:18\|do_GetState returned z=0 and 8 04/09:58:30\|do_GetState returned z=0 and 8 04/09:58:43\|do_GetState returned z=0 and 8 04/09:58:56\|do_GetState returned z=0 and 8 04/09:59:08\|do_GetState returned z=0 and 8 04/09:59:21\|do_GetState returned z=0 and 8 04/09:59:33\|do_GetState returned z=0 and 8 04/09:59:46\|do_GetState returned z=0 and 8 04/09:59:58\|do_GetState returned z=0 and 8 04/10:00:11\|do_GetState returned z=0 and 8 04/10:00:23\|do_GetState returned z=0 and 8 04/10:00:36\|do_GetState returned z=0 and 8 04/10:00:48\|do_GetState returned z=0 and 8 04/10:01:01\|do_GetState returned z=0 and 8 04/10:01:13\|do_GetState returned z=0 and 8 04/10:01:26\|do_GetState returned z=0 and 8 04/10:01:38\|do_GetState returned z=0 and 8 04/10:01:51\|do_GetState returned z=0 and 8 04/10:02:04\|do_GetState returned z=0 and 8 04/10:02:16\|do_GetState returned z=0 and 8 04/10:02:29\|do_GetState returned z=0 and 8 04/10:02:41\|do_GetState returned z=0 and 8 04/10:02:54\|do_GetState returned z=0 and 8 04/10:03:06\|do_GetState returned z=0 and 8 04/10:03:19\|do_GetState returned z=0 and 8 04/10:03:31\|do_GetState returned z=0 and 8 04/10:03:44\|do_GetState returned z=0 and 8 04/10:03:57\|do_GetState returned z=0 and 8 04/10:04:09\|do_GetState returned z=0 and 8 04/10:04:22\|do_GetState returned z=0 and 8 04/10:04:34\|do_GetState returned z=0 and 8 04/10:04:47\|do_GetState returned z=0 and 8 04/10:04:59\|do_GetState returned z=0 and 8 04/10:05:12\|do_GetState returned z=0 and 8 04/10:05:25\|do_GetState returned z=0 and 8 04/10:05:37\|do_GetState returned z=0 and 8 04/10:05:50\|do_GetState returned z=0 and 8 04/10:06:02\|do_GetState returned z=0 and 8 04/10:06:15\|do_GetState returned z=0 and 8 04/10:06:27\|do_GetState returned z=0 and 8 04/10:06:40\|do_GetState returned z=0 and 8 04/10:06:52\|do_GetState returned z=0 and 8 04/10:07:05\|do_GetState returned z=0 and 8 04/10:07:18\|do_GetState returned z=0 and 8 04/10:07:30\|do_GetState returned z=0 and 8
355	Fri Jun 4 08:40:52 2021	TD	Friday 4 June
09.20 DAQ found stopped files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight aida15 rebooted at 00.02 regained control of DAQ and issued DAQ STOP by restarting NewMerger twice 11.00 DAQ reset and setup All system wide checks OK except Base Current Difference aida05 fault 0x4d3c : 0x4d3c : 0 aida05 : WR status 0x10 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 4 2 8 5 4 1 2 2 4 7 : 41352 aida02 : 0 3 3 6 6 4 1 2 2 4 7 : 41352 aida03 : 2 4 2 12 1 3 2 2 2 4 7 : 41352 aida04 : 2 2 1 7 4 3 2 2 2 4 7 : 41352 aida05 : 2 3 1 5 3 2 3 2 2 4 7 : 41360 aida06 : 2 4 2 6 4 3 2 2 2 4 7 : 41352 aida07 : 1 5 2 9 4 2 4 3 1 4 7 : 41324 aida08 : 0 3 1 7 4 3 4 3 1 4 7 : 41352 aida09 : 3 3 0 8 2 2 3 2 2 4 7 : 41380 aida10 : 1 2 2 11 3 2 3 3 3 3 7 : 41044 aida11 : 2 3 1 6 6 4 2 3 3 3 7 : 41072 aida12 : 4 2 1 8 3 3 3 3 1 4 7 : 41072 aida13 : 2 4 1 8 4 4 2 3 3 3 7 : 41016 aida14 : 1 4 1 9 4 4 2 3 3 3 7 : 41044 aida15 : 0 5 1 2 2 1 2 4 3 4 7 : 42872 aida16 : 1 3 2 9 4 4 4 4 2 3 7 : 41052 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 11.03 DAQ resumes file S496/R32 alpha background slow comparator 0x64 BNC PB-5 pulser OFF 11.09 1.8.W spectra - 20us FSR - attachments 1 & 2 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart 13.07 analysis of file S496/R32_16 - attachment 8 max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png

Attachment 2: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png

Attachment 4: Screenshot_2021-06-04_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 980.png

Attachment 7: 981.png

Attachment 8: R32_16
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 49166189 ( 17465.0 Hz) Other data format: 212753815 ( 75575.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 137 ( 0.0 Hz) RESUME: 137 ( 0.0 Hz) SYNC100: 42486 ( 15.1 Hz) WR48-63: 42486 ( 15.1 Hz) FEE64 disc: 1037793 ( 368.6 Hz) MBS info: 211630776 ( 75176.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2815.130 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2759.088 2 0.741 0.000 3 4.946 0.000 4 0.000 0.000 5 0.000 2173.993 6 0.000 0.000 7 0.000 2104.432 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 184 0 0 0 0 0 0 0 0 0 0 0 1 115458 62 0 0 0 0 31 31 0 0 0 0 2 444 105932150 0 0 36 36 18754 18754 0 105894570 0 0 3 39683483 106817705 0 0 101 101 21752 21752 1037793 105736206 0 0 4 154 0 0 0 0 0 0 0 0 0 0 0 5 4413391 1852 0 0 0 0 926 926 0 0 0 0 6 586 0 0 0 0 0 0 0 0 0 0 0 7 4948958 2044 0 0 0 0 1022 1022 0 0 0 0 8 489 0 0 0 0 0 0 0 0 0 0 0 9 77 0 0 0 0 0 0 0 0 0 0 0 10 1172 0 0 0 0 0 0 0 0 0 0 0 11 163 2 0 0 0 0 1 1 0 0 0 0 12 754 0 0 0 0 0 0 0 0 0 0 0 13 94 0 0 0 0 0 0 0 0 0 0 0 14 605 0 0 0 0 0 0 0 0 0 0 0 15 177 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 29.027s ( 1102.409 blocks/s, 68.901 Mb/s)
354	Thu Jun 3 10:16:39 2021	TD	Thursday 3 June
11.19 Hot HEC channels multiple FEE64s ASIC check fixes System wide checks OK * except* Base Current Difference aida01 fault 0x8940 : 0x8941 : 1 aida02 fault 0x9b32 : 0x9b33 : 1 aida03 fault 0xf266 : 0xf267 : 1 aida04 fault 0x579b : 0x579c : 1 White Rabbit error counter test result: Passed 12, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 11 4 2 2 2 2 2 3 2 4 7 : 41516 aida02 : 10 7 2 0 1 2 2 3 2 4 7 : 41408 aida03 : 10 8 2 2 2 4 1 4 2 3 7 : 40008 aida04 : 5 4 4 2 2 3 1 4 2 3 7 : 39860 aida05 : 12 3 2 2 1 2 2 3 2 4 7 : 41448 aida06 : 15 9 2 2 3 2 1 3 2 4 7 : 41380 aida07 : 12 8 1 2 2 3 3 2 2 4 7 : 41408 aida08 : 9 6 3 1 2 3 3 2 2 4 7 : 41380 aida09 : 15 6 1 1 2 2 2 3 2 4 7 : 41500 aida10 : 15 6 3 2 1 3 3 2 1 4 7 : 40348 aida11 : 8 2 1 1 1 3 1 3 2 4 7 : 41248 aida12 : 18 3 2 1 3 3 2 2 2 4 7 : 41184 aida13 : 7 4 1 4 3 4 3 3 1 4 7 : 41100 aida14 : 18 6 1 4 3 4 2 2 1 4 7 : 40392 aida15 : 7 7 1 3 1 2 1 3 2 4 7 : 41220 aida16 : 4 4 2 3 3 3 2 4 1 4 7 : 41200 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Wed Jun 02 16:27:30 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 1.8.W spectra - 20us FSR - attachments 2 & 3 ADC data items - attachment 4 FEE64 temperatures OK - attachment 5 DSSSD bias & leakage currents OK - attachment 6 Merger/Tape Server/Merger statistics - attachment 7 no merger errors reported since previous restart 11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36 13.09 analysis of file S496/R31_16 - attachment 8 max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-06-03_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png

Attachment 6: 970.png

Attachment 7: 971.png

Attachment 8: R31_16
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 50135749 ( 17884.4 Hz) Other data format: 211784253 ( 75547.7 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 199 ( 0.1 Hz) RESUME: 199 ( 0.1 Hz) SYNC100: 42443 ( 15.1 Hz) WR48-63: 42443 ( 15.1 Hz) FEE64 disc: 1081490 ( 385.8 Hz) MBS info: 210617479 ( 75131.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 4 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2803.320 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2522.967 2 0.535 0.000 3 8.636 0.000 4 0.000 0.000 5 0.000 2119.634 6 0.000 0.000 7 0.000 2067.036 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 196 2 0 0 0 0 0 0 2 0 0 2 1 193203 74 0 0 0 0 37 37 0 0 0 0 2 471 105497692 0 0 26 26 18605 18605 0 105460430 0 0 3 40159698 106282506 0 0 173 173 21812 21812 1081487 105157049 0 0 4 128 0 0 0 0 0 0 0 0 0 0 0 5 4517663 1914 0 0 0 0 957 957 0 0 0 0 6 641 2 0 0 0 0 1 1 0 0 0 0 7 5260254 2060 0 0 0 0 1030 1030 0 0 0 0 8 463 0 0 0 0 0 0 0 0 0 0 0 9 104 0 0 0 0 0 0 0 0 0 0 0 10 1026 2 0 0 0 0 1 1 0 0 0 0 11 141 1 0 0 0 0 0 0 1 0 0 1 12 786 0 0 0 0 0 0 0 0 0 0 0 13 109 0 0 0 0 0 0 0 0 0 0 0 14 687 0 0 0 0 0 0 0 0 0 0 0 15 179 0 0 0 0 0 0 0 0 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 28.445s ( 1124.966 blocks/s, 70.310 Mb/s)
353	Wed Jun 2 14:53:05 2021	OH, TD	Merger/Tapeserver issues after reboot of aida-3
On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen. We were also unable to connected to MIDAS via ssh port forwarding. Since the restart we have been unable to write to file. The tapeserver will allocate/mount/open a file but nothing will be written to it. The merger will be receiving data but never shows as having any "Current links with data". /MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23 2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 The current disk structure is: [npg@aidas-gsi npg]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sdb 8:16 0 7.3T 0 disk ├─sdb1 8:17 0 200M 0 part ├─sdb2 8:18 0 500M 0 part └─sdb3 8:19 0 7.3T 0 part ├─vg_aidas2-lv_root (dm-2) 253:2 0 50G 0 lvm ├─vg_aidas2-lv_home (dm-3) 253:3 0 7.2T 0 lvm /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 └─vg_aidas2-lv_swap (dm-4) 253:4 0 7.8G 0 lvm sdc 8:32 0 7.3T 0 disk └─sdc1 8:33 0 7.3T 0 part /media/ThirdDrive sdd 8:48 0 7.3T 0 disk └─sdd1 8:49 0 7.3T 0 part /media/SecondDrive sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 200M 0 part /boot/efi ├─sda2 8:2 0 500M 0 part /boot └─sda3 8:3 0 465.1G 0 part ├─vg_aidasgsi-lv_root (dm-0) 253:0 0 50G 0 lvm / ├─vg_aidasgsi-lv_swap (dm-1) 253:1 0 7.8G 0 lvm [SWAP] └─vg_aidasgsi-lv_home (dm-5) 253:5 0 407.3G 0 lvm /home Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3 TapeServer config file - attachment 4 Merger options - attachment 5 Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png
Attachment 1: 210602_MergerTape_HTTPd_Startup
tidy up tclsh8.5_copy2: no process killed System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64 Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64 MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64 PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin Computer Name = aidas-gsi; Temp Directory = /tmp/tcl25624 package limit is not available: can't find package limit Running with default file descriptor limit package setuid is not available: can't find package setuid Could not change to user 50 group 50: not owner /debug user "debug" password "z5y30nlwy2e4" httpd started on port 8115 Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl /DataBaseAccessServer /NetVarService /SigTaskService Loaded MemSasAccess /SpectrumService /TapeServer loading tcl/NewMergerControl.tcl for namespace :: /DataAcquisitionControlServer DefineMessage unknown Run Control Server Implementation for MERGE RunControlServer loaded loading Html/RunControl/implementation.tcl /MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist Created UI registers RunControl loaded loading Html/NewMerger/RunControl/implementation.tcl for namespace :: ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist Created UI registers NewMerge Control loaded Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl Shared memory area located at 0x7f34d3028000 Tape Server comms table located at 0x7f34d3028000 MERGE command 30 setup MERGER Stop Merge Halt (1) Action has completed: MergeCommand 2 Halt Setup Merge Setup (1) Action has completed: MergeCommand 1 Setup MERGE command 32 go MERGER Go Merge Go (1) Action has completed: MergeCommand 3 Go link 0 GO (1) Action has completed: MergeCommand 22 {link 0 GO} 0 2 link 1 GO (1) Action has completed: MergeCommand 22 {link 1 GO} 1 2 link 2 GO (1) Action has completed: MergeCommand 22 {link 2 GO} 2 2 link 3 GO (1) Action has completed: MergeCommand 22 {link 3 GO} 3 2 link 4 GO (1) Action has completed: MergeCommand 22 {link 4 GO} 4 2 link 5 GO (1) Action has completed: MergeCommand 22 {link 5 GO} 5 2 link 6 GO (1) Action has completed: MergeCommand 22 {link 6 GO} 6 2 link 7 GO (1) Action has completed: MergeCommand 22 {link 7 GO} 7 2 link 8 GO (1) Action has completed: MergeCommand 22 {link 8 GO} 8 2 link 9 GO (1) Action has completed: MergeCommand 22 {link 9 GO} 9 2 link 10 GO (1) Action has completed: MergeCommand 22 {link 10 GO} 10 2 link 11 GO (1) Action has completed: MergeCommand 22 {link 11 GO} 11 2 link 12 GO (1) Action has completed: MergeCommand 22 {link 12 GO} 12 2 link 13 GO (1) Action has completed: MergeCommand 22 {link 13 GO} 13 2 link 14 GO (1) Action has completed: MergeCommand 22 {link 14 GO} 14 2 link 15 GO (1) Action has completed: MergeCommand 22 {link 15 GO} 15 2 Resume MERGER
Attachment 2: 210602_NewMerger_Startup
Tidy up MERGE: no process killed merge64.AD: no process killed Starting New Merger [1] 25659 MERGE Master (25659): Message logger not contacted. MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021 20:32:01 Creating NetVars Creating NetVars complete MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021 20:32:01 MERGE Master (25659): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0 MERGE Master (25659): File mapped object /SHM_11000 of size 536628 created MERGE Master (25659): Memory mapping 536628 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0a99b8000. MERGE Master (25659): File mapped object /SHM_11001 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0a39b6000. MERGE Master (25659): File mapped object /SHM_11002 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc09d9b4000. MERGE Master (25659): File mapped object /SHM_11003 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0979b2000. MERGE Master (25659): File mapped object /SHM_11004 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0919b0000. MERGE Master (25659): File mapped object /SHM_11005 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc08b9ae000. MERGE Master (25659): File mapped object /SHM_11006 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0859ac000. MERGE Master (25659): File mapped object /SHM_11007 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc07f9aa000. MERGE Master (25659): File mapped object /SHM_11008 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0799a8000. MERGE Master (25659): File mapped object /SHM_11009 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0739a6000. MERGE Master (25659): File mapped object /SHM_11010 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc06d9a4000. MERGE Master (25659): File mapped object /SHM_11011 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0679a2000. MERGE Master (25659): File mapped object /SHM_11012 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc0619a0000. MERGE Master (25659): File mapped object /SHM_11013 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc05b99e000. MERGE Master (25659): File mapped object /SHM_11014 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc05599c000. MERGE Master (25659): File mapped object /SHM_11015 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc04f99a000. MERGE Master (25659): File mapped object /SHM_11016 of size 100669480 created MERGE Master (25659): Memory mapping 100669480 bytes MERGE Master (25659): Shared memory segment located at address 0x7fc049998000. MERGE Master (25659): Shared memory segment located at address 0x7fc049998000. MERGE Master (25659): Starting stats/rates task MERGE Master (25659): Stats/Rates task has pid 25661 MERGE Master (25659): Starting 16 link tasks MERGE Master (25659): Link task has pid 25662 MERGE Master (25659): Link task has pid 25663 MERGE Statistics/Rates (25661): Message logger not contacted. MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021 20:32:03 Creating NetVars Creating NetVars complete MERGE Master (25659): Link task has pid 25664 MERGE Data Link (25662): Message logger not contacted. MERGE Data Link (25662): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25662): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0 MERGE Data Link (25662): File mapped object /SHM_11000 accessed MERGE Data Link (25662): Memory mapping 536628 bytes MERGE Data Link (25662): Shared memory segment located at address 0x7f198ee7a000. MERGE Data Link (25662): Allocated 71720 bytes for local input buffer MERGE Data Link (25662): File mapped object /SHM_11001 accessed MERGE Data Link (25662): Memory mapping 100669480 bytes MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000. MERGE Master (25659): Link task has pid 25665 MERGE Data Link (25663): Message logger not contacted. MERGE Data Link (25663): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25663): Configuration: index = 1, SHM key=11000, TCP port base = 11002, module = 0, hardware = 0 MERGE Data Link (25663): File mapped object /SHM_11000 accessed MERGE Data Link (25663): Memory mapping 536628 bytes MERGE Data Link (25663): Shared memory segment located at address 0x7f3bee603000. MERGE Data Link (25663): Allocated 71720 bytes for local input buffer MERGE Data Link (25663): File mapped object /SHM_11002 accessed MERGE Data Link (25663): Memory mapping 100669480 bytes MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000. MERGE Master (25659): Link task has pid 25666 MERGE Data Link (25664): Message logger not contacted. MERGE Data Link (25665): Message logger not contacted. MERGE Data Link (25664): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 MERGE Data Link (25665): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars Creating NetVars complete MERGE Data Link (25665): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0 MERGE Data Link (25665): File mapped object /SHM_11000 accessed MERGE Data Link (25665): Memory mapping 536628 bytes MERGE Data Link (25665): Shared memory segment located at address 0x7f7418ae6000. MERGE Data Link (25665): Allocated 71720 bytes for local input buffer MERGE Data Link (25665): File mapped object /SHM_11004 accessed MERGE Data Link (25665): Memory mapping 100669480 bytes MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000. Creating NetVars complete MERGE Data Link (25664): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0 MERGE Data Link (25664): File mapped object /SHM_11000 accessed MERGE Data Link (25664): Memory mapping 536628 bytes MERGE Data Link (25664): Shared memory segment located at address 0x7f0da72c8000. MERGE Data Link (25664): Allocated 71720 bytes for local input buffer MERGE Data Link (25664): File mapped object /SHM_11003 accessed MERGE Data Link (25664): Memory mapping 100669480 bytes MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000. MERGE Master (25659): Link task has pid 25667 MERGE Master (25659): Link task has pid 25668 MERGE Data Link (25666): Message logger not contacted. MERGE Data Link (25666): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25666): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0 MERGE Data Link (25666): File mapped object /SHM_11000 accessed MERGE Data Link (25666): Memory mapping 536628 bytes MERGE Data Link (25666): Shared memory segment located at address 0x7fb7f4ec2000. MERGE Data Link (25666): Allocated 71720 bytes for local input buffer MERGE Data Link (25666): File mapped object /SHM_11005 accessed MERGE Data Link (25666): Memory mapping 100669480 bytes MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000. MERGE Master (25659): Link task has pid 25669 MERGE Data Link (25667): Message logger not contacted. MERGE Data Link (25667): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25667): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0 MERGE Data Link (25667): File mapped object /SHM_11000 accessed MERGE Data Link (25667): Memory mapping 536628 bytes MERGE Data Link (25667): Shared memory segment located at address 0x7f8b878c9000. MERGE Data Link (25667): Allocated 71720 bytes for local input buffer MERGE Data Link (25667): File mapped object /SHM_11006 accessed MERGE Data Link (25667): Memory mapping 100669480 bytes MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000. MERGE Master (25659): Link task has pid 25670 MERGE Data Link (25668): Message logger not contacted. MERGE Data Link (25668): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Master (25659): Link task has pid 25671 Creating NetVars complete MERGE Data Link (25668): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0 MERGE Data Link (25668): File mapped object /SHM_11000 accessed MERGE Data Link (25668): Memory mapping 536628 bytes MERGE Data Link (25668): Shared memory segment located at address 0x7f88c7c9a000. MERGE Data Link (25668): Allocated 71720 bytes for local input buffer MERGE Data Link (25668): File mapped object /SHM_11007 accessed MERGE Data Link (25668): Memory mapping 100669480 bytes MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000. MERGE Data Link (25669): Message logger not contacted. MERGE Data Link (25669): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Master (25659): Link task has pid 25672 Creating NetVars complete MERGE Data Link (25669): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0 MERGE Data Link (25669): File mapped object /SHM_11000 accessed MERGE Data Link (25669): Memory mapping 536628 bytes MERGE Data Link (25669): Shared memory segment located at address 0x7f87e7704000. MERGE Data Link (25669): Allocated 71720 bytes for local input buffer MERGE Data Link (25669): File mapped object /SHM_11008 accessed MERGE Data Link (25669): Memory mapping 100669480 bytes MERGE Data Link (25669): Shared memory segment located at address 0x7f87e1702000. MERGE Data Link (25671): Message logger not contacted. MERGE Data Link (25671): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25671): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0 MERGE Data Link (25671): File mapped object /SHM_11000 accessed MERGE Data Link (25671): Memory mapping 536628 bytes MERGE Data Link (25671): Shared memory segment located at address 0x7f38b4f59000. MERGE Data Link (25671): Allocated 71720 bytes for local input buffer MERGE Data Link (25671): File mapped object /SHM_11010 accessed MERGE Data Link (25671): Memory mapping 100669480 bytes MERGE Data Link (25671): Shared memory segment located at address 0x7f38aef57000. MERGE Data Link (25670): Message logger not contacted. MERGE Data Link (25670): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Master (25659): Link task has pid 25673 Creating NetVars complete MERGE Data Link (25670): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0 MERGE Data Link (25670): File mapped object /SHM_11000 accessed MERGE Data Link (25670): Memory mapping 536628 bytes MERGE Data Link (25672): Message logger not contacted. MERGE Data Link (25670): Shared memory segment located at address 0x7ff7744ce000. MERGE Data Link (25672): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars MERGE Data Link (25670): Allocated 71720 bytes for local input buffer MERGE Data Link (25670): File mapped object /SHM_11009 accessed MERGE Data Link (25670): Memory mapping 100669480 bytes MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000. Creating NetVars complete MERGE Data Link (25672): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0 MERGE Data Link (25672): File mapped object /SHM_11000 accessed MERGE Data Link (25672): Memory mapping 536628 bytes MERGE Data Link (25672): Shared memory segment located at address 0x7f4bb287e000. MERGE Data Link (25672): Allocated 71720 bytes for local input buffer MERGE Data Link (25672): File mapped object /SHM_11011 accessed MERGE Data Link (25672): Memory mapping 100669480 bytes MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000. MERGE Master (25659): Link task has pid 25674 MERGE Data Link (25673): Message logger not contacted. MERGE Data Link (25673): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25673): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0 MERGE Data Link (25673): File mapped object /SHM_11000 accessed MERGE Data Link (25673): Memory mapping 536628 bytes MERGE Data Link (25673): Shared memory segment located at address 0x7fe3cd54a000. MERGE Data Link (25673): Allocated 71720 bytes for local input buffer MERGE Data Link (25673): File mapped object /SHM_11012 accessed MERGE Data Link (25673): Memory mapping 100669480 bytes MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000. MERGE Master (25659): Link task has pid 25675 MERGE Data Link (25674): Message logger not contacted. MERGE Data Link (25674): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25674): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0 MERGE Data Link (25674): File mapped object /SHM_11000 accessed MERGE Data Link (25674): Memory mapping 536628 bytes MERGE Data Link (25674): Shared memory segment located at address 0x7faec5dbd000. MERGE Data Link (25674): Allocated 71720 bytes for local input buffer MERGE Data Link (25674): File mapped object /SHM_11013 accessed MERGE Data Link (25674): Memory mapping 100669480 bytes MERGE Data Link (25674): Shared memory segment located at address 0x7faebfdbb000. MERGE Master (25659): Link task has pid 25676 MERGE Master (25659): Link task has pid 25677 MERGE Data Link (25675): Message logger not contacted. MERGE Data Link (25675): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25675): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0 MERGE Data Link (25675): File mapped object /SHM_11000 accessed MERGE Data Link (25675): Memory mapping 536628 bytes MERGE Data Link (25675): Shared memory segment located at address 0x7fef43120000. MERGE Data Link (25675): Allocated 71720 bytes for local input buffer MERGE Data Link (25675): File mapped object /SHM_11014 accessed MERGE Data Link (25675): Memory mapping 100669480 bytes MERGE Data Link (25675): Shared memory segment located at address 0x7fef3d11e000. MERGE Data Link (25676): Message logger not contacted. MERGE Data Link (25676): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25676): Configuration: index = 14, SHM key=11000, TCP port base = 11015, module = 13, hardware = 0 MERGE Data Link (25676): File mapped object /SHM_11000 accessed MERGE Data Link (25676): Memory mapping 536628 bytes MERGE Data Link (25676): Shared memory segment located at address 0x7f6aea271000. MERGE Data Link (25676): Allocated 71720 bytes for local input buffer MERGE Data Link (25676): File mapped object /SHM_11015 accessed MERGE Data Link (25676): Memory mapping 100669480 bytes MERGE Data Link (25676): Shared memory segment located at address 0x7f6ae426f000. MERGE Data Link (25677): Message logger not contacted. MERGE Data Link (25677): MIDAS MERGE Data Link 64 bit Build May 11 2021 20:32:02 Creating NetVars Creating NetVars complete MERGE Data Link (25677): Configuration: index = 15, SHM key=11000, TCP port base = 11016, module = 14, hardware = 0 MERGE Data Link (25677): File mapped object /SHM_11000 accessed MERGE Data Link (25677): Memory mapping 536628 bytes MERGE Data Link (25677): Shared memory segment located at address 0x7fc4281e9000. MERGE Data Link (25677): Allocated 71720 bytes for local input buffer MERGE Data Link (25677): File mapped object /SHM_11016 accessed MERGE Data Link (25677): Memory mapping 100669480 bytes MERGE Data Link (25677): Shared memory segment located at address 0x7fc4221e7000. MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021 20:32:03 MERGE Statistics thread starting MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000. MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000. MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000. MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000. MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000. MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000. MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000. MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000. MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000. MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000. MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000. MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000. MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000. MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000. MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000. MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000. MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000. MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000. MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000. ... 201 more lines ...
Attachment 3: 210602_TapeServerStartup
Tidy up master(4596): Operation not permitted Starting Tape Server [1] 25639 MIDAS Tape Server: Message logger not contacted. MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021 12:22:14 MIDAS Tape Server: Unable to change scheduling priority - Permission denied MIDAS Tape Server: Using default startup MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205. MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Server: Shared memory ID is 3 MIDAS Tape Server: Shared memory segment located at address 7fdf19721000. MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305 MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002 MIDAS Tape Server: Message reporting level = 0x180fff8 MIDAS Tape Server: Message logging level = 0xfff8 MIDAS Tape Server: Tape Server Options = 0x0 MIDAS Tape Server: File device path base = /TapeData MIDAS Tape Server: Data buffer size = 65536 MIDAS Tape Server: Tape block size = 65536 MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created MIDAS Tape Server: Shared memory ID is 3 MIDAS Tape Server: Shared memory segment located at address 7fdf19320000. MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created MIDAS Tape Server: Shared memory ID is 3 MIDAS Tape Server: Shared memory segment located at address 7fdf19881000. MIDAS Tape Server: Capabilities restored. MIDAS Tape Server: Master global area initialised. MIDAS Tape Server: Stats task has pid 25641 MIDAS Tape Server: Driver process for /dev/file/0 has pid 25642 MIDAS Tape Server: Driver process for /dev/file/1 has pid 25643 MIDAS Tape Server: Driver process for /dev/null/0 has pid 25644 MIDAS Tape Server: Link task 0 has pid 25645 MIDAS Tape Server: Link task 1 has pid 25646 MIDAS Tape Server: Starting the RPC interface MIDAS Tape Statistics: Message logger not contacted. MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021 12:22:20 MIDAS Tape Statistics: Started with args 10205 MIDAS Tape Statistics: Configuration: SHM key=10205 MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Statistics: Shared memory ID is 3 MIDAS Tape Statistics: Shared memory segment located at address 7f9c551f7000. MIDAS Data Link (25645): Message logger not contacted. MIDAS Data Link (25645): MIDAS Tape Data Link Build Mar 16 2021 12:22:19 MIDAS Data Link (25645): Started with args 10205 10305 MIDAS Data Link (25645): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1 MIDAS Tape Driver (25644): Message logger not contacted. MIDAS Tape Driver (25644): Message logger not contacted. MIDAS Data Link (25646): Message logger not contacted. MIDAS Tape Driver (25644): Started with args 2 10205 MIDAS Data Link (25645): File mapped object /SHM_10205 of size 1331104 created MIDAS Data Link (25645): Shared memory ID is 3 MIDAS Data Link (25645): Shared memory segment located at address 7f6ba30ef000. MIDAS Data Link (25646): MIDAS Tape Data Link Build Mar 16 2021 12:22:19 MIDAS Tape Driver (25644): Configuration: driver=2, key=10205. MIDAS Data Link (25646): Started with args 10205 21001 21002 MIDAS Data Link (25645): Starting the network interface MIDAS Tape Driver (25644): File mapped object /SHM_10205 of size 1331104 created MIDAS Data Link (25646): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2 MIDAS Tape Driver (25644): Shared memory ID is 3 MIDAS Tape Driver (25644): Shared memory segment located at address 7f09d718d000. MIDAS Tape Driver (25644): Using device /dev/null/0 of type sink. MIDAS Data Link (25646): File mapped object /SHM_10205 of size 1331104 created MIDAS Data Link (25646): Shared memory ID is 3 MIDAS Data Link (25646): Shared memory segment located at address 7fa75ae3c000. MIDAS Data Link (25646): Starting the network interface MIDAS Tape Driver (25642): Message logger not contacted. MIDAS Data Link (25645): TCP socket receive buffer was 87380 - now 249856 MIDAS Tape Driver (25642): Message logger not contacted. MIDAS Tape Driver (25642): Started with args 0 10205 MIDAS Tape Driver (25642): Configuration: driver=0, key=10205. MIDAS Tape Driver (25643): Message logger not contacted. MIDAS Tape Driver (25642): File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Driver (25643): Message logger not contacted. MIDAS Tape Driver (25642): Shared memory ID is 3 MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856 MIDAS Data Link (25645): TCP socket send buffer was 16384 - now 249856 MIDAS Tape Driver (25643): Started with args 1 10205 MIDAS Tape Driver (25642): Shared memory segment located at address 7f48b1ea4000. MIDAS Tape Driver (25642): Using device /dev/file/0 of type file. MIDAS Data Link (25645): MIDAS Data Link thread 0 using TCP port 10305. MIDAS Data Link (25645): Entering server loop MIDAS Tape Driver (25643): Configuration: driver=1, key=10205. MIDAS Data Link (25645): thread 0 listening on port 10305 MIDAS Tape Driver (25643): File mapped object /SHM_10205 of size 1331104 created MIDAS Tape Driver (25643): Shared memory ID is 3 MIDAS Tape Driver (25643): Shared memory segment located at address 7f6044052000. MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856 MIDAS Tape Driver (25643): Using device /dev/file/1 of type file. MIDAS Data Link (25646): MIDAS Data Link thread 0 using TCP port 21001. MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856 MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856 MIDAS Data Link (25646): MIDAS Data Link thread 1 using TCP port 21002. MIDAS Data Link (25646): Entering server loop MIDAS Data Link (25646): thread 0 listening on port 21001 MIDAS Data Link (25646): thread 1 listening on port 21002 MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205. MIDAS Tape Server: Entering server loop MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205. MIDAS Data Link (25645): thread 0 accepted connection from 127.0.0.1, port 37330 MIDAS Data Link (25645): buffer size changed to 65536 MIDAS Tape Server: device /dev/file/0 allocated. MIDAS Tape Server: Mounting volume S496 on device /dev/file/0. MIDAS Tape Server: Opening file R31 on device /dev/file/0. 2222 64 0 31056336 3333 31056336 2048000 1 15
Attachment 4: TS_configuration
#device configuration information # any line starting with a # is a comment and is ignored stats /MIDAS/TapeServer/Linux64/stats # devices available - list ends with a null line # format class_name device_name driver_task_path_name # class_name MUST be one of dlt, exabyte, scsitape, sink file /dev/file/0 /MIDAS/TapeServer/Linux64/driver file /dev/file/1 /MIDAS/TapeServer/Linux64/driver sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver #data link configuration information - list end with a null line # format link_task_path_name /MIDAS/TapeServer/Linux64/linkTCP 10305 /MIDAS/TapeServer/Linux64/linkTCP 21001 21002 #program options - list end with a null line #tapeserver_options 0x2 tapeserver_options 0x0 msg_reporting_level 0x0180fff8 # use 0x0080fff8 to enable msg logging msg_logging_level 0xfff8 # default for following is 16Kbytes - both MUST be the same at present data_buffer_size 64 tape_block_size 64 disc_file_size 2000 # file_path_base /TapeData #end of information
Attachment 5: CONTENTS
Index string MERGE.LinksAvailable&&RunNumber&&MERGE.LinksInUse&&MERGE.RunOptions Stat.shift string 0 0x0006dead string 0x0000 MERGE.LinksAvailable string 16 Stat.offset string 0 RunNumber string 0 Rate.channels string 1024 Stat.channels string 1024 MERGE.RunOptions string 3 MERGE.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1% Merge.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%
Attachment 6: 210602_merger.png

352	Sun May 30 08:57:53 2021	TD	Sunday 30 May
09.55 DAQ continues - file S496/R18_588 alpha background increased data rate due to hot HEC channels since c. 06.00 this am - ASIC check fixes problem All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b7 : 7 aida02 fault 0xfe72 : 0xfe79 : 7 aida03 fault 0x97f4 : 0x97fb : 7 aida04 fault 0x950 : 0x957 : 7 aida05 fault 0x8e93 : 0x8e98 : 5 aida06 fault 0x545a : 0x545f : 5 aida07 fault 0xc04d : 0xc052 : 5 aida08 fault 0xa0f7 : 0xa0fc : 5 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 6 9 5 1 3 2 2 2 2 4 6 : 37008 aida02 : 3 5 5 2 2 4 3 3 2 4 6 : 37956 aida03 : 22 12 7 5 3 3 2 3 3 3 6 : 36872 aida04 : 40 13 5 5 3 4 2 3 3 3 6 : 37048 aida05 : 15 5 5 4 2 2 2 2 2 4 6 : 37044 aida06 : 11 6 2 1 2 3 3 4 2 4 6 : 38300 aida07 : 17 5 4 3 1 3 3 2 2 4 6 : 37324 aida08 : 1 8 1 2 2 4 4 3 2 4 6 : 38164 aida09 : 10 8 6 1 2 4 3 3 1 4 6 : 36968 aida10 : 14 7 5 2 1 3 3 3 1 4 6 : 36800 aida11 : 7 9 3 1 2 3 3 3 1 4 6 : 36788 aida12 : 3 5 2 3 1 4 2 3 2 4 6 : 37620 aida13 : 12 4 4 2 1 3 2 2 2 4 6 : 37008 aida14 : 11 10 5 1 1 3 2 2 2 4 6 : 37036 aida15 : 11 5 1 2 1 3 3 4 2 4 6 : 38244 aida16 : 10 4 8 1 2 3 2 2 2 4 6 : 37096 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.24 All histograms, statistics and merger statistics zero'd 16.16 analysis of file S496/R18_541 - attachment 9 max deadtime 0.02% aida03 18.40 DAQ continues - file S496/R18_589 alpha background increased data rate due to hot HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-30_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-30_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-30_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-30_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 960.png

Attachment 8: 961.png

Attachment 9: R18_589
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 31985 ADC data format: 58630511 ( 25364.6 Hz) Other data format: 203166719 ( 87893.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 35 ( 0.0 Hz) RESUME: 35 ( 0.0 Hz) SYNC100: 40596 ( 17.6 Hz) WR48-63: 40596 ( 17.6 Hz) FEE64 disc: 1341777 ( 580.5 Hz) MBS info: 201743680 ( 87277.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2311.511 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 1879.226 2 0.365 0.000 3 0.313 0.000 4 0.000 1697.189 5 0.000 557.330 6 0.000 1686.560 7 0.000 647.171 8 0.000 0.000 9 0.000 0.000 10 0.382 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 157 0 0 0 0 0 0 0 0 0 0 0 1 3314609 1296 0 0 0 0 648 648 0 0 0 0 2 413 87031526 0 0 19 19 14856 14856 0 87001776 0 0 3 42258127 88383449 0 0 15 15 18954 18954 1341775 87003736 0 0 4 153 6936262 0 0 0 0 860 860 0 6934542 0 0 5 6312304 6939162 0 0 0 0 2310 2310 0 6934542 0 0 6 544 6936170 0 0 0 0 814 814 0 6934542 0 0 7 6741331 6938850 0 0 0 0 2154 2154 0 6934542 0 0 8 448 0 0 0 0 0 0 0 0 0 0 0 9 79 0 0 0 0 0 0 0 0 0 0 0 10 1189 2 0 0 1 1 0 0 0 0 0 0 11 111 2 0 0 0 0 0 0 2 0 0 0 12 767 0 0 0 0 0 0 0 0 0 0 0 13 99 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 180 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.504s ( 1015.271 blocks/s, 63.454 Mb/s)
351	Sat May 29 13:13:45 2021	TD	Saturday 29 May
14.10 DAQ continues - file S496/R18_539 alpha background increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b7 : 7 aida02 fault 0xfe72 : 0xfe79 : 7 aida03 fault 0x97f4 : 0x97fb : 7 aida04 fault 0x950 : 0x957 : 7 aida05 fault 0x8e93 : 0x8e97 : 4 aida06 fault 0x545a : 0x545e : 4 aida07 fault 0xc04d : 0xc051 : 4 aida08 fault 0xa0f7 : 0xa0fb : 4 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 8 7 2 2 2 2 2 2 4 6 : 37036 aida02 : 14 3 5 2 2 4 3 3 2 4 6 : 37984 aida03 : 15 11 8 6 3 3 2 3 3 3 6 : 36884 aida04 : 43 13 5 5 3 4 2 3 3 3 6 : 37060 aida05 : 19 3 5 4 2 2 2 2 2 4 6 : 37044 aida06 : 17 4 1 3 2 3 3 4 2 4 6 : 38356 aida07 : 16 6 2 1 0 2 2 3 2 4 6 : 37296 aida08 : 21 8 1 1 3 4 4 3 2 4 6 : 38276 aida09 : 13 10 2 1 2 3 2 2 2 4 6 : 37060 aida10 : 13 9 4 1 1 3 3 3 1 4 6 : 36764 aida11 : 13 9 2 1 2 3 3 3 1 4 6 : 36796 aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724 aida13 : 7 6 6 2 1 3 2 2 2 4 6 : 37036 aida14 : 9 7 7 1 1 3 2 2 2 4 6 : 37036 aida15 : 14 5 2 2 1 3 3 4 2 4 6 : 38272 aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.24 All histograms, statistics and merger statistics zero'd 16.16 analysis of file S496/R18_541 - attachment 9 max deadtime 0.12% aida04 18.40 DAQ continues - file S496/R18_551 alpha background increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-29_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-29_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-29_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-29_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 950.png

Attachment 8: 951.png

Attachment 9: R18_541
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 46792076 ( 18702.9 Hz) Other data format: 215127929 ( 85987.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 89 ( 0.0 Hz) RESUME: 89 ( 0.0 Hz) SYNC100: 41318 ( 16.5 Hz) WR48-63: 41318 ( 16.5 Hz) FEE64 disc: 1013016 ( 404.9 Hz) MBS info: 214032099 ( 85549.1 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2501.862 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 1964.282 2 0.467 0.000 3 3.050 0.000 4 0.000 1832.921 5 0.000 844.521 6 0.000 1829.325 7 0.000 869.373 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 499 0 0 0 0 0 0 0 0 0 0 0 1 74891 24 0 0 0 0 12 12 0 0 0 0 2 1032 92085039 0 0 27 27 16251 16251 0 92052483 0 0 3 37271457 93008741 0 0 62 62 19135 19135 1013015 91957332 0 0 4 552 7507421 0 0 0 0 925 925 0 7505571 0 0 5 4051453 7509613 0 0 0 0 2021 2021 0 7505571 0 0 6 1401 7507471 0 0 0 0 950 950 0 7505571 0 0 7 5387708 7509619 0 0 0 0 2024 2024 0 7505571 0 0 8 460 0 0 0 0 0 0 0 0 0 0 0 9 183 0 0 0 0 0 0 0 0 0 0 0 10 537 0 0 0 0 0 0 0 0 0 0 0 11 388 0 0 0 0 0 0 0 0 0 0 0 12 674 0 0 0 0 0 0 0 0 0 0 0 13 244 1 0 0 0 0 0 0 1 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 597 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.855s ( 1004.537 blocks/s, 62.784 Mb/s)
350	Fri May 28 10:29:07 2021	TD	Friday 28 May
11.30 DAQ continues - file S496/R18_472 alpha background All system wide checks OK except FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Base Current Difference aida01 fault 0x82b0 : 0x82b2 : 2 aida02 fault 0xfe72 : 0xfe74 : 2 aida03 fault 0x97f4 : 0x97f6 : 2 aida04 fault 0x950 : 0x952 : 2 aida05 fault 0x8e93 : 0x8e95 : 2 aida06 fault 0x545a : 0x545c : 2 aida07 fault 0xc04d : 0xc04f : 2 aida08 fault 0xa0f7 : 0xa0f9 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 4 7 1 3 2 2 2 2 4 6 : 36980 aida02 : 3 1 5 1 2 4 3 3 2 4 6 : 37892 aida03 : 8 8 6 6 3 3 2 3 3 3 6 : 36800 aida04 : 37 12 5 6 3 4 2 3 3 3 6 : 37060 aida05 : 11 5 6 4 2 2 2 2 2 4 6 : 37044 aida06 : 4 3 3 3 2 2 2 3 3 4 6 : 38456 aida07 : 1 12 3 2 0 3 3 2 2 4 6 : 37204 aida08 : 13 8 1 2 3 4 3 3 2 4 6 : 38020 aida09 : 7 7 5 1 1 3 2 2 2 4 6 : 36996 aida10 : 12 7 9 2 1 3 3 3 1 4 6 : 36856 aida11 : 2 9 3 1 2 3 3 3 1 4 6 : 36768 aida12 : 11 4 3 3 2 4 2 3 2 4 6 : 37724 aida13 : 10 8 6 2 2 3 2 2 2 4 6 : 37128 aida14 : 16 7 7 1 0 4 2 2 2 4 6 : 37128 aida15 : 12 6 2 2 1 3 3 4 2 4 6 : 38272 aida16 : 9 4 10 1 2 3 2 2 2 4 6 : 37124 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.30 analysis of file S496/R18_471 - attachment 9 max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-28_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 940.png

Attachment 8: 941.png

Attachment 9: R18_471
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 57407666 ( 20753.5 Hz) Other data format: 204512336 ( 73933.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 116 ( 0.0 Hz) RESUME: 116 ( 0.0 Hz) SYNC100: 42303 ( 15.3 Hz) WR48-63: 42303 ( 15.3 Hz) FEE64 disc: 1160567 ( 419.6 Hz) MBS info: 203266931 ( 73483.1 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 109 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2766.172 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2700.364 2 0.332 0.000 3 4.856 0.000 4 0.000 0.000 5 0.000 1685.309 6 0.000 0.000 7 0.000 1567.327 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 181 23 0 0 0 0 0 0 23 0 0 13 1 838470 386 0 0 0 0 166 166 54 0 0 0 2 162 101753181 0 0 19 19 17931 17931 15 101717266 0 5 3 42392398 102752755 0 0 97 97 21331 21331 1160234 101549665 0 12 4 211 21 0 0 0 0 0 0 21 0 0 10 5 6478076 2740 0 0 0 0 1337 1337 66 0 0 0 6 260 17 0 0 0 0 0 0 17 0 0 4 7 7695649 3163 0 0 0 0 1537 1537 89 0 0 28 8 132 14 0 0 0 0 1 1 12 0 0 10 9 368 5 0 0 0 0 0 0 5 0 0 4 10 173 18 0 0 0 0 0 0 18 0 0 5 11 303 5 0 0 0 0 0 0 5 0 0 5 12 186 1 0 0 0 0 0 0 1 0 0 7 13 585 2 0 0 0 0 0 0 2 0 0 2 14 0 0 0 0 0 0 0 0 0 0 0 0 15 512 5 0 0 0 0 0 0 5 0 0 4 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.742s ( 1008.122 blocks/s, 63.008 Mb/s)
349	Thu May 27 12:25:53 2021	TD	Thursday 27 May
13.29 DAQ continues - file S496/R18_443 alpha background All system wide checks OK except FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Base Current Difference aida01 fault 0x82b0 : 0x82b2 : 2 aida02 fault 0xfe72 : 0xfe74 : 2 aida03 fault 0x97f4 : 0x97f6 : 2 aida04 fault 0x950 : 0x952 : 2 aida05 fault 0x8e93 : 0x8e95 : 2 aida06 fault 0x545a : 0x545c : 2 aida07 fault 0xc04d : 0xc04f : 2 aida08 fault 0xa0f7 : 0xa0f9 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 7 7 1 3 2 2 2 2 4 6 : 37008 aida02 : 3 1 5 1 3 4 3 3 2 4 6 : 37956 aida03 : 2 6 5 6 2 4 2 3 3 3 6 : 36808 aida04 : 24 12 5 5 4 3 2 3 3 3 6 : 36912 aida05 : 8 1 9 3 2 2 2 2 2 4 6 : 37016 aida06 : 3 6 2 2 2 2 2 3 3 4 6 : 38428 aida07 : 6 17 1 2 0 3 3 2 2 4 6 : 37232 aida08 : 4 2 1 2 3 4 3 3 2 4 6 : 37936 aida09 : 15 7 5 2 3 4 3 3 1 4 6 : 37060 aida10 : 7 6 9 2 0 3 3 3 1 4 6 : 36764 aida11 : 19 10 2 0 2 3 3 3 1 4 6 : 36796 aida12 : 11 6 4 2 2 4 2 3 2 4 6 : 37724 aida13 : 17 9 2 0 0 4 2 2 2 4 6 : 37036 aida14 : 15 4 7 1 1 3 2 2 2 4 6 : 37036 aida15 : 9 6 1 2 1 3 3 4 2 4 6 : 38244 aida16 : 11 11 8 2 0 3 2 2 2 4 6 : 37060 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.30 analysis of file S496/R18_442 - attachment 9 max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-05-27_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-27_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-27_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-27_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 930.png

Attachment 8: 931.png

Attachment 9: R18_442
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 68693190 ( 21525.3 Hz) Other data format: 193226813 ( 60548.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 114 ( 0.0 Hz) RESUME: 114 ( 0.0 Hz) SYNC100: 43889 ( 13.8 Hz) WR48-63: 43889 ( 13.8 Hz) FEE64 disc: 1335474 ( 418.5 Hz) MBS info: 191803333 ( 60102.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 3 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 3191.275 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2989.890 2 0.511 643.077 3 8.760 0.000 4 0.000 0.000 5 0.000 1578.752 6 0.000 0.000 7 0.000 1340.150 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 155 0 0 0 0 0 0 0 0 0 0 0 1 527962 216 0 0 0 0 108 108 0 0 0 0 2 533 96091128 0 0 28 28 16832 16832 0 96057408 0 0 3 49735236 97127126 0 0 86 86 22778 22778 1335473 95745925 0 0 4 118 2 0 0 0 0 1 1 0 0 0 0 5 8268444 3874 0 0 0 0 1937 1937 0 0 0 0 6 632 0 0 0 0 0 0 0 0 0 0 0 7 10156762 4466 0 0 0 0 2233 2233 0 0 0 0 8 500 0 0 0 0 0 0 0 0 0 0 0 9 102 0 0 0 0 0 0 0 0 0 0 0 10 1498 0 0 0 0 0 0 0 0 0 0 0 11 105 0 0 0 0 0 0 0 0 0 0 0 12 886 0 0 0 0 0 0 0 0 0 0 0 13 95 1 0 0 0 0 0 0 1 0 0 2 14 0 0 0 0 0 0 0 0 0 0 0 0 15 162 0 0 0 0 0 0 0 0 0 0 1 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 32.027s ( 999.146 blocks/s, 62.447 Mb/s)
348	Wed May 26 13:21:59 2021	TD	Wednesday 26 May
14.22 DAQ continues - file S496/R18_369 alpha background All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b1 : 1 aida02 fault 0xfe72 : 0xfe73 : 1 aida03 fault 0x97f4 : 0x97f5 : 1 aida04 fault 0x950 : 0x951 : 1 aida05 fault 0x8e93 : 0x8e94 : 1 aida06 fault 0x545a : 0x545b : 1 aida07 fault 0xc04d : 0xc04e : 1 aida08 fault 0xa0f7 : 0xa0f8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 12 4 6 1 2 3 2 2 2 4 6 : 37072 aida02 : 12 2 4 1 3 4 3 3 2 4 6 : 37984 aida03 : 12 6 7 6 2 4 3 2 3 3 6 : 36624 aida04 : 36 11 4 6 4 3 2 3 3 3 6 : 36968 aida05 : 2 4 6 3 2 2 2 2 2 4 6 : 36968 aida06 : 8 5 3 2 2 2 2 3 3 4 6 : 38456 aida07 : 10 4 6 4 3 3 3 3 2 4 6 : 37992 aida08 : 21 8 3 2 3 3 3 3 2 4 6 : 37956 aida09 : 17 10 3 2 3 4 3 3 1 4 6 : 37060 aida10 : 3 6 10 2 1 3 3 3 1 4 6 : 36828 aida11 : 12 6 4 2 2 3 3 3 1 4 6 : 36832 aida12 : 2 9 5 1 3 3 2 3 2 4 6 : 37632 aida13 : 4 7 2 1 3 3 3 3 2 4 6 : 37832 aida14 : 15 6 6 1 0 4 2 2 2 4 6 : 37100 aida15 : 5 4 3 2 1 3 3 4 2 4 6 : 38244 aida16 : 17 6 6 2 1 3 2 2 2 4 6 : 37076 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - attachments 3 & 4 ADC data item stats - attachment 5 FEE64 temps OK - attachment 6 DSSSD bias & leakage current OK - attachment 7 Merger/Tape Server/Merger stats - attachment 8 no merger errors reported since previous restart 14.30 analysis of file S496/R18_340 - attachment 9 max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-05-26_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-26_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-26_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-05-26_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 920.png

Attachment 8: 921.png

Attachment 9: R18_368
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 60865731 ( 22299.7 Hz) Other data format: 201054280 ( 73661.2 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 152 ( 0.1 Hz) RESUME: 152 ( 0.1 Hz) SYNC100: 42166 ( 15.4 Hz) WR48-63: 42166 ( 15.4 Hz) FEE64 disc: 1309767 ( 479.9 Hz) MBS info: 199659877 ( 73150.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 107 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2729.447 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2401.079 2 0.453 0.000 3 8.570 0.000 4 0.000 0.000 5 0.000 1507.937 6 0.000 0.000 7 0.000 1499.170 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 1115 29 0 0 0 0 1 1 27 0 0 10 1 2766811 11645 0 0 0 0 574 574 10497 0 0 18 2 2194 100014163 0 0 27 27 17354 17354 0 99979401 0 1 3 44009553 100997602 0 0 125 125 21286 21286 1274310 99680470 0 5 4 902 6 0 0 0 0 0 0 0 6 0 0 5 6872755 14651 0 0 0 0 1451 1451 11749 0 0 10 6 1948 13 0 0 0 0 1 1 11 0 0 9 7 7202790 16106 0 0 0 0 1497 1497 13112 0 0 9 8 531 0 0 0 0 0 0 0 0 0 0 0 9 394 0 0 0 0 0 0 0 0 0 0 3 10 1118 1 0 0 0 0 0 0 1 0 0 0 11 900 26 0 0 0 0 0 0 26 0 0 12 12 1756 13 0 0 0 0 0 0 13 0 0 8 13 1346 11 0 0 0 0 1 1 9 0 0 7 14 0 0 0 0 0 0 0 0 0 0 0 0 15 1618 14 0 0 0 0 1 1 12 0 0 15 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 31.695s ( 1009.613 blocks/s, 63.101 Mb/s)
347	Tue May 25 11:16:54 2021	TD	Tuesday 25 May
12.16 DAQ continues file S496/R18_324 alpha background Tape server c. 2.5Mb/s ASIC check Tape server c. 300kb/s - presumably hot HEC channels 12.20 All system wide checks OK except Base Current Difference aida01 fault 0x82b0 : 0x82b1 : 1 aida02 fault 0xfe72 : 0xfe73 : 1 aida03 fault 0x97f4 : 0x97f5 : 1 aida04 fault 0x950 : 0x951 : 1 aida05 fault 0x8e93 : 0x8e94 : 1 aida06 fault 0x545a : 0x545b : 1 aida07 fault 0xc04d : 0xc04e : 1 aida08 fault 0xa0f7 : 0xa0f8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 6 6 2 1 3 2 2 2 4 6 : 37016 aida02 : 3 6 5 3 3 4 3 2 1 3 7 : 38572 aida03 : 21 10 8 6 3 3 2 3 2 4 6 : 37924 aida04 : 29 15 4 5 3 4 2 3 3 3 6 : 37004 aida05 : 17 5 2 2 2 2 2 2 2 4 6 : 36940 aida06 : 1 6 0 1 2 2 2 3 3 4 6 : 38356 aida07 : 11 6 7 4 3 3 3 3 2 4 6 : 38028 aida08 : 13 8 3 2 2 3 3 2 1 3 7 : 38372 aida09 : 7 9 2 1 3 3 3 3 2 4 6 : 37860 aida10 : 9 8 4 2 0 3 3 3 1 4 6 : 36708 aida11 : 16 6 3 2 2 3 3 3 1 4 6 : 36832 aida12 : 21 10 7 4 2 4 2 2 1 3 7 : 38420 aida13 : 8 3 3 1 3 3 3 3 2 4 6 : 37832 aida14 : 14 3 6 1 0 4 2 2 2 4 6 : 37072 aida15 : 2 4 2 2 1 3 3 4 2 4 6 : 38216 aida16 : 16 5 5 3 0 3 2 2 2 4 6 : 37016 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1 Rate spectra - attachment 2 ADC data item stats - attachment 3 FEE64 temps OK - attachment 4 DSSSD bias & leakage current OK - attachment 5 Merger/Tape Server/Merger stats - attachment 6 no merger errors reported since previous restart 17.30 analysis of file S496/R18_340 max deadtime 0.36% aida04
Attachment 1: Screenshot_2021-05-25_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-25_Spectrum_Browser_aidas-gsi.png

Attachment 3: Screenshot_2021-05-25_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-05-25_Temperature_and_status_scan_aidas-gsi.png

Attachment 5: 910.png

Attachment 6: 911.png

Attachment 7: R18_340
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 63855540 ( 23748.1 Hz) Other data format: 198064462 ( 73660.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 109 ( 0.0 Hz) RESUME: 109 ( 0.0 Hz) SYNC100: 42015 ( 15.6 Hz) WR48-63: 42015 ( 15.6 Hz) FEE64 disc: 1291036 ( 480.1 Hz) MBS info: 196689178 ( 73149.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 35 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2688.871 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2304.866 2 0.344 0.000 3 9.625 0.000 4 0.000 0.000 5 0.000 1450.665 6 0.000 0.000 7 0.000 1243.447 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 303 7 0 0 0 0 1 1 5 0 0 5 1 3031185 1240 0 0 0 0 620 620 0 0 0 0 2 536 98548687 0 0 20 20 16872 16872 2 98514901 0 3 3 44821969 99507955 0 0 89 89 21248 21248 1291004 98174277 0 0 4 514 3 0 0 0 0 1 1 1 0 0 1 5 7529304 3056 0 0 0 0 1528 1528 0 0 0 0 6 1314 2 0 0 0 0 0 0 2 0 0 2 7 8467764 3488 0 0 0 0 1744 1744 0 0 0 0 8 226 4 0 0 0 0 0 0 4 0 0 4 9 144 4 0 0 0 0 0 0 4 0 0 4 10 524 1 0 0 0 0 0 0 1 0 0 0 11 432 2 0 0 0 0 0 0 2 0 0 2 12 719 0 0 0 0 0 0 0 0 0 0 0 13 154 7 0 0 0 0 1 1 5 0 0 7 14 0 0 0 0 0 0 0 0 0 0 0 0 15 452 6 0 0 0 0 0 0 6 0 0 7 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 39.492s ( 810.287 blocks/s, 50.643 Mb/s)
346	Mon May 24 09:57:07 2021	TD	Monday 24 May
07.30 Hot HEC channels multiple FEE64s ASIC check fixes 10.56 DAQ continues OK file S496/R18_75 System wide checks OK * except* FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Base Current Difference aida01 fault 0x82b0 : 0x82b1 : 1 aida02 fault 0xfe72 : 0xfe73 : 1 aida03 fault 0x97f4 : 0x97f5 : 1 aida04 fault 0x950 : 0x951 : 1 aida05 fault 0x8e93 : 0x8e94 : 1 aida06 fault 0x545a : 0x545b : 1 aida07 fault 0xc04d : 0xc04e : 1 aida08 fault 0xa0f7 : 0xa0f8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 2 7 2 1 4 3 2 1 4 7 : 40452 aida02 : 13 3 3 1 3 4 2 2 2 4 7 : 41308 aida03 : 17 6 2 2 1 4 2 3 2 3 7 : 39700 aida04 : 8 6 1 0 1 3 1 4 2 3 7 : 39712 aida05 : 10 3 5 3 4 3 3 2 1 4 7 : 40560 aida06 : 4 4 2 0 2 3 2 3 3 4 7 : 42576 aida07 : 5 3 2 3 1 3 1 3 2 4 7 : 41324 aida08 : 11 5 1 1 2 3 3 2 2 4 7 : 41348 aida09 : 7 4 3 0 2 3 1 3 2 4 7 : 41324 aida10 : 11 4 2 2 2 3 2 4 1 4 7 : 41132 aida11 : 18 5 0 2 2 4 2 2 1 4 7 : 40240 aida12 : 2 3 0 1 2 3 2 2 2 4 7 : 41024 aida13 : 4 5 1 2 2 3 2 4 1 4 7 : 41096 aida14 : 9 3 1 2 1 4 2 2 2 4 7 : 41164 aida15 : 2 3 1 3 2 2 3 3 2 4 7 : 41744 aida16 : 5 5 2 0 3 3 2 4 1 4 7 : 41116 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 12:04:25 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5 1.8.H spectra - attachments 6-8 1.8.L spectra - attachments 9 & 10 Stat spectra - attachments 11 & 12 1.8.W spectra - 20us FSR - attachments 13 & 14 ADC data items - attachment 15 FEE64 temperatures OK - attachment 16 DSSSD bias & leakage currents OK - attachment 17 Merger/Tape Server/Merger statistics - attachment 18 no merger errors reported since previous restart analysis of file R18_74 max dead time 0.05% 10.17 ASIC check All histograms, statistics, merger statistics zero'd
Attachment 1: Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-24_Statistics_aidas-gsi(4).png

Attachment 3: Screenshot_2021-05-24_Statistics_aidas-gsi(3).png

Attachment 4: Screenshot_2021-05-24_Statistics_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-24_Statistics_aidas-gsi(1).png

Attachment 6: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png

Attachment 7: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png

Attachment 8: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png

Attachment 9: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png

Attachment 10: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png

Attachment 11: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png

Attachment 12: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png

Attachment 13: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png

Attachment 14: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png

Attachment 15: Screenshot_2021-05-24_Statistics_aidas-gsi.png

Attachment 16: Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png

Attachment 17: 900.png

Attachment 18: 901.png

Attachment 19: R18_74
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 42982076 ( 14466.2 Hz) Other data format: 218937925 ( 73686.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 59 ( 0.0 Hz) RESUME: 59 ( 0.0 Hz) SYNC100: 43069 ( 14.5 Hz) WR48-63: 43069 ( 14.5 Hz) FEE64 disc: 1163669 ( 391.6 Hz) MBS info: 217688000 ( 73265.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 2971.209 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 0.000 2 0.425 0.000 3 1.488 0.000 4 0.000 0.000 5 0.000 2622.333 6 0.000 0.000 7 0.000 2340.476 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 64 0 0 0 0 0 0 0 0 0 0 0 1 2227 3 0 0 0 0 1 1 1 0 0 0 2 75 108903724 0 0 25 25 19964 19964 0 108863746 0 0 3 35357926 110031040 0 0 34 34 21525 21525 1163668 108824254 0 0 4 89 0 0 0 0 0 0 0 0 0 0 0 5 3145189 1316 0 0 0 0 658 658 0 0 0 0 6 64 0 0 0 0 0 0 0 0 0 0 0 7 4475895 1842 0 0 0 0 921 921 0 0 0 0 8 56 0 0 0 0 0 0 0 0 0 0 0 9 67 0 0 0 0 0 0 0 0 0 0 0 10 115 0 0 0 0 0 0 0 0 0 0 0 11 65 0 0 0 0 0 0 0 0 0 0 0 12 65 0 0 0 0 0 0 0 0 0 0 0 13 77 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 102 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 40.766s ( 784.975 blocks/s, 49.061 Mb/s)
345	Sat May 22 23:40:32 2021	TD	Sunday 23 May
00.40 Unable to connect to FEE64s 1-7 DAQ power cycled OK ASIC settings 2021Apr29-13-16-00 slow comparator 0x64 all FEE64s BNC PB-5 OFF file S496/R16 alpha background 07.17 Some hot HEC channels ASIC check fixes issue System wide checks OK except Base Current Difference aida06 fault 0x5457 : 0x5457 : 0 aida06 : WR status 0x10 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x1cd : 461 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 16 9 4 3 1 4 3 2 1 4 7 : 40552 aida02 : 3 2 2 1 2 3 3 2 2 4 7 : 41308 aida03 : 14 5 0 1 1 3 2 4 2 3 7 : 40000 aida04 : 14 4 5 1 3 3 1 4 2 3 7 : 39944 aida05 : 12 4 1 3 2 3 3 2 2 4 7 : 41408 aida06 : 8 3 2 1 2 4 2 3 3 4 7 : 42744 aida07 : 10 4 3 2 1 2 2 3 2 4 7 : 41464 aida08 : 7 5 2 0 3 3 3 2 2 4 7 : 41380 aida09 : 10 5 1 1 1 2 2 3 2 4 7 : 41408 aida10 : 11 6 3 2 1 3 2 2 2 4 7 : 41100 aida11 : 12 8 3 2 2 2 3 2 1 4 7 : 40288 aida12 : 9 9 3 1 2 3 3 2 1 4 7 : 40380 aida13 : 10 4 0 1 1 2 1 3 2 4 7 : 41128 aida14 : 9 4 2 1 1 2 1 3 2 4 7 : 41156 aida15 : 5 4 0 3 3 3 3 3 2 4 7 : 41940 aida16 : 9 6 3 2 0 2 1 3 2 4 7 : 41156 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 23 00:29:48 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE64 temperatures OK - attachment 1 ADC data item stats - attachment 2 Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3 DSSSD bias & leakage currents OK - attachment 4 11.18 Analysis of file S496/R16_33 - attachment 5 [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 685M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.0T 867G 88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive 867/2 x 3279s = 16.5 days to filesystem full - OK 12.11 aida15 zero data/poll stats - re-loaded DAQ program OK aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK DAQ continues file S496/R18
Attachment 1: Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-05-23_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png

Attachment 4: 810.png

Attachment 5: R16_33
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 47385414 ( 14450.2 Hz) Other data format: 214534594 ( 65422.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 61 ( 0.0 Hz) RESUME: 61 ( 0.0 Hz) SYNC100: 44215 ( 13.5 Hz) WR48-63: 44215 ( 13.5 Hz) FEE64 disc: 1311957 ( 400.1 Hz) MBS info: 213134085 ( 64995.2 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 3279.230 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 0.000 2 0.386 0.000 3 1.000 0.000 4 0.000 0.000 5 0.000 2849.824 6 0.000 0.000 7 0.000 2608.689 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 71 0 0 0 0 0 0 0 0 0 0 0 1 1455 0 0 0 0 0 0 0 0 0 0 0 2 87 106617422 0 0 35 35 20186 20186 0 106576980 0 0 3 39184250 107913590 0 0 26 26 22238 22238 1311957 106557105 0 0 4 63 0 0 0 0 0 0 0 0 0 0 0 5 3345685 1512 0 0 0 0 756 756 0 0 0 0 6 60 0 0 0 0 0 0 0 0 0 0 0 7 4853274 2070 0 0 0 0 1035 1035 0 0 0 0 8 69 0 0 0 0 0 0 0 0 0 0 0 9 45 0 0 0 0 0 0 0 0 0 0 0 10 70 0 0 0 0 0 0 0 0 0 0 0 11 68 0 0 0 0 0 0 0 0 0 0 0 12 89 0 0 0 0 0 0 0 0 0 0 0 13 57 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 71 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 44.293s ( 722.462 blocks/s, 45.154 Mb/s)
344	Sat May 22 13:40:56 2021	TD	Saturday 22 May
14.37 DAQ continues file S496/R15_14 Grafana - DSSSD bias & leakage current - attachment 1 1.8.L spectra - attachments 2 & 3 Stat spectra - attachments 4 & 5 FEE64 temperatures OK - attachment 6 stats - ADC data items, pause items OK - attachments 7 & 8 DSSSD bias & leakage current OK - attachment 9 Merger/Tape Server/Merger stats - attachment 10 analysis of file S496/R14_1920 - attachment 11 all FEE64 deadtime 0% as expected 17.30 aida01, 3, 5, 9 & 11 have hot HEC channels ASIC check fixes All histogram, statistics & merger statistics zero'd DAQ continues Note gamma-ray source tests have started with FATIMA and DEGAS 20.20 Analysis file R15_19 - attachment 12 [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 665M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.3T 530G 93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 90/2 x 3168s = 40h to filesystem full - OK 20.22 All system wide checks OK except Base Current Difference aida01 fault 0xf82c : 0xf82f : 3 aida02 fault 0xdda6 : 0xdda9 : 3 aida03 fault 0xb13d : 0xb140 : 3 aida04 fault 0xf4cd : 0xf4d0 : 3 aida05 fault 0x500 : 0x575 : 117 aida06 fault 0xc6e7 : 0xc6e9 : 2 aida07 fault 0x4920 : 0x4922 : 2 aida08 fault 0xbd9a : 0xbd9c : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 8 4 6 6 1 4 3 3 4 3 6 : 38048 aida02 : 11 8 10 3 4 4 3 4 3 3 6 : 37740 aida03 : 2 8 3 3 2 2 1 4 2 4 6 : 37720 aida04 : 26 15 17 4 4 2 1 4 2 4 6 : 38256 aida05 : 15 6 4 1 1 2 1 3 3 4 6 : 38156 aida06 : 7 7 16 6 3 4 2 2 3 4 6 : 38612 aida07 : 11 5 12 3 3 3 2 3 1 3 7 : 38836 aida08 : 13 6 7 2 1 2 2 2 3 4 6 : 37972 aida09 : 23 3 8 5 0 5 3 3 4 3 6 : 38164 aida10 : 12 5 7 1 2 2 4 3 2 4 6 : 37992 aida11 : 19 2 7 1 2 2 1 2 3 4 6 : 37740 aida12 : 21 5 3 3 3 3 1 2 3 4 6 : 37964 aida13 : 18 7 7 1 2 4 2 3 2 4 6 : 37776 aida14 : 8 3 13 7 2 4 2 3 2 4 6 : 37992 aida15 : 11 5 14 1 3 4 2 3 2 4 6 : 37908 aida16 : 17 6 6 2 2 3 1 2 3 4 6 : 37908 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19.30 Lost activity monitors - attachments 13-15 20.54 1.8.W spectra 20us FSR attachments 16 & 23 1.8.W spectra 200us FSR attachments 17 & 22 1.8.W spectra 2ms FSR attachments 18 & 21 1.8.W spectra 20ms FSR attachments 19 & 20
Attachment 1: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana(1).png

Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(11).png

Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(10).png

Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(9).png

Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(8).png

Attachment 6: Screenshot_2021-05-22_Statistics_aidas-gsi(9).png

Attachment 7: Screenshot_2021-05-22_Statistics_aidas-gsi(8).png

Attachment 8: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(3).png

Attachment 9: 805.png

Attachment 10: 806.png

Attachment 11: R15_13
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 102550190 ( 10129.0 Hz) Other data format: 159369822 ( 15741.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 69714 ( 6.9 Hz) WR48-63: 69714 ( 6.9 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 159230394 ( 15727.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 10124.450 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 0.000 2 0.000 67.951 3 0.000 0.000 4 0.000 0.000 5 0.000 9971.491 6 0.000 0.000 7 0.000 10069.837 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 213 0 0 0 0 0 0 0 0 0 0 0 1 1159 2 0 0 0 0 1 1 0 0 0 0 2 279 79660137 0 0 0 0 22470 22470 0 79615197 0 0 3 101219504 79708933 0 0 0 0 46868 46868 0 79615197 0 0 4 246 0 0 0 0 0 0 0 0 0 0 0 5 588847 366 0 0 0 0 183 183 0 0 0 0 6 280 0 0 0 0 0 0 0 0 0 0 0 7 737827 382 0 0 0 0 191 191 0 0 0 0 8 262 0 0 0 0 0 0 0 0 0 0 0 9 221 0 0 0 0 0 0 0 0 0 0 0 10 242 2 0 0 0 0 1 1 0 0 0 0 11 210 0 0 0 0 0 0 0 0 0 0 0 12 273 0 0 0 0 0 0 0 0 0 0 0 13 193 0 0 0 0 0 0 0 0 0 0 0 14 250 0 0 0 0 0 0 0 0 0 0 0 15 184 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 45.816s ( 698.440 blocks/s, 43.652 Mb/s)
Attachment 12: R15_19
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 55871232 ( 17633.8 Hz) Other data format: 206048770 ( 65032.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 56 ( 0.0 Hz) RESUME: 56 ( 0.0 Hz) SYNC100: 43803 ( 13.8 Hz) WR48-63: 43803 ( 13.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 205961052 ( 65004.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 3168.420 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 0.000 2865.072 2 0.000 0.000 3 0.419 0.000 4 0.000 0.000 5 0.000 2166.184 6 0.000 0.000 7 0.000 1949.169 8 0.000 0.000 9 0.000 0.000 10 0.000 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 55 0 0 0 0 0 0 0 0 0 0 0 1 133932 56 0 0 0 0 28 28 0 0 0 0 2 84 103025372 0 0 0 0 18982 18982 0 102987408 0 0 3 42389344 103017790 0 0 56 56 22017 22017 0 102973644 0 0 4 56 0 0 0 0 0 0 0 0 0 0 0 5 6040914 2574 0 0 0 0 1287 1287 0 0 0 0 6 103 0 0 0 0 0 0 0 0 0 0 0 7 7306081 2976 0 0 0 0 1488 1488 0 0 0 0 8 78 0 0 0 0 0 0 0 0 0 0 0 9 86 0 0 0 0 0 0 0 0 0 0 0 10 68 0 0 0 0 0 0 0 0 0 0 0 11 75 0 0 0 0 0 0 0 0 0 0 0 12 116 2 0 0 0 0 1 1 0 0 0 0 13 80 0 0 0 0 0 0 0 0 0 0 0 14 80 0 0 0 0 0 0 0 0 0 0 0 15 80 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.406s ( 737.221 blocks/s, 46.076 Mb/s)
Attachment 13: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(4).png

Attachment 14: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(3).png

Attachment 15: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(2).png

Attachment 16: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(19).png

Attachment 17: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(18).png

Attachment 18: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(17).png

Attachment 19: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(16).png

Attachment 20: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(15).png

Attachment 21: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(14).png

Attachment 22: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(13).png

Attachment 23: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(12).png

343	Fri May 21 23:19:23 2021	TD	Saturday 22 May 00:00-08:00
00.16 413Gb remaining /TapeData filesystem estimate 413/2 x 177s = 10.15h to filesystem full - OK 00.18 All histograms, statistics and merger statistics zero'd 00.57 All system wide checks OK except Base Current Difference aida05 fault 0x500 : 0x572 : 114 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 9 3 4 1 5 2 2 3 3 6 : 36356 aida02 : 4 8 7 2 4 3 3 3 2 3 6 : 35968 aida03 : 12 9 6 2 5 3 2 3 3 3 6 : 36824 aida04 : 11 5 9 2 2 2 2 4 3 3 6 : 37028 aida05 : 24 9 2 1 2 3 2 3 4 3 6 : 37608 aida06 : 25 6 3 4 2 4 3 3 3 3 6 : 37060 aida07 : 3 3 3 1 3 3 2 2 2 2 7 : 37044 aida08 : 20 4 2 3 1 3 2 2 3 3 6 : 36016 aida09 : 15 8 2 5 0 3 3 2 3 3 6 : 36284 aida10 : 16 6 4 1 2 3 3 2 2 4 6 : 37328 aida11 : 20 2 2 1 1 3 2 2 3 3 6 : 35936 aida12 : 1 2 1 2 3 4 1 2 4 3 6 : 36900 aida13 : 25 10 4 1 0 3 2 3 3 3 6 : 36500 aida14 : 20 6 9 2 2 5 1 2 2 4 6 : 37200 aida15 : 31 9 5 2 3 3 1 3 3 3 6 : 36500 aida16 : 24 4 2 3 1 4 2 3 3 3 6 : 36672 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 01.12 Lost activity monitor - attachment 1 1.8.H spectra - attachments 2 & 3 1.8.L spectra - attachments 4-6 aida01 pulser peak width 120 ch FWHM Rate specttra - attachments 7 & 8 01.06 Lost activity monitor - attachment 9 ucesb - attachment 10 Grafana - DSSSD bias & leakage currents past 7 days - attachment 11 statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17 FEE64 temperatures OK - attachment 18 DSSSD bias & leakage currents OK - attachment 19 Merger/Tape Server/Merger stats - attachment 20 no merger warning/error messages since last restart 01.26 Analysis file S496/R14_1798 - attachment 21 aida08 5.5% deadtime 01.17 Disk space remaining /TapeData filesystem [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 706M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.4T 395G 95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 395/2 x 177s = 9.7h to filesystem full - OK 01.30 despec shift reports no data reported by ucesb aida stats & tape server ok - no data being forwarded to MBS by data relay 01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811 04.35 System wide checks all OK except Base Current Difference aida05 fault 0x500 : 0x573 : 115 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 661M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.5T 337G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 337/2 x 177s = 8.3h to filesystem full - OK FEE64 temperatures OK - attachment 22 ADC data item statistics OK - attachment 24 DSSSD bias & leakage current OK - attachment 26 analysis of file S496/R14_1920 - attachment 28 aida04 deadtime 5.0% 07.10 Ssystem wide checks all OK except Base Current Difference aida05 fault 0x500 : 0x573 : 115 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR [npg@aidas-gsi S496]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_aidasgsi-lv_root 50G 18G 30G 37% / tmpfs 7.9G 662M 7.2G 9% /dev/shm /dev/sda2 477M 42M 410M 10% /boot /dev/sda1 200M 264K 200M 1% /boot/efi /dev/mapper/vg_aidasgsi-lv_home 401G 90G 292G 24% /home /dev/mapper/vg_aidas2-lv_home 7.2T 6.5T 287G 96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7 /dev/sdd1 7.2T 5.6T 1.3T 82% /media/SecondDrive /dev/sdc1 7.2T 6.6T 259G 97% /media/ThirdDrive estimate 287/2 x 178s = 7.1h to filesystem full - OK FEE64 temperatures OK - attachment 23 ADC data item statistics OK - attachment 25 DSSSD bias & leakage current OK - attachment 27 analysis of file S496/R14_1920 - attachment 29 aida04 deadtime 4.4% 07.35 Found MBS data monitor kill'd Restarted MBS data relay and data monitor OK 08.00 beam off - end of S496 08.28 Tape Server -> no storage mode DAQ continues 09.58 All slow comparators -> 0x64 BNC PB-5 OFF file S496/R15 alpha background Tape server 200kb/s, merger 260k data items/s
Attachment 1: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(1).png

Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(7).png

Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(6).png

Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(5).png

Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(2).png

Attachment 7: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(1).png

Attachment 8: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi.png

Attachment 9: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi.png

Attachment 10: Screenshot_2021-05-22_ucesb.png

Attachment 11: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana.png

Attachment 12: Screenshot_2021-05-22_Statistics_aidas-gsi(5).png

Attachment 13: Screenshot_2021-05-22_Statistics_aidas-gsi(4).png

Attachment 14: Screenshot_2021-05-22_Statistics_aidas-gsi(3).png

Attachment 15: Screenshot_2021-05-22_Statistics_aidas-gsi(2).png

Attachment 16: Screenshot_2021-05-22_Statistics_aidas-gsi(1).png

Attachment 17: Screenshot_2021-05-22_Statistics_aidas-gsi.png

Attachment 18: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi.png

Attachment 19: 800.png

Attachment 20: 801.png

Attachment 21: R14_1798
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254662088 ( 1433466.5 Hz) Other data format: 7257912 ( 40854.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 256 ( 1.4 Hz) RESUME: 257 ( 1.4 Hz) SYNC100: 32661 ( 183.8 Hz) WR48-63: 32661 ( 183.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7192077 ( 40483.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 365795 ( 2059.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 177.655 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.112 0.000 1 5.898 0.000 2 0.060 0.000 3 7.669 0.000 4 0.073 7.902 5 7.648 0.000 6 0.356 0.000 7 9.852 0.000 8 1.513 0.000 9 0.000 15.513 10 2.639 0.000 11 0.903 0.000 12 0.687 0.000 13 0.000 6.517 14 0.073 0.000 15 0.212 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5664289 2546 0 0 5 5 737 737 0 1062 0 19821 1 24002581 6376 0 0 39 39 3149 3149 0 0 0 40581 2 12140787 1878978 0 0 3 3 1758 1758 0 1875456 0 32794 3 29094115 1803103 0 0 54 54 3917 3917 0 1795161 0 46229 4 3034023 1008057 0 0 6 6 480 480 0 1007085 0 8526 5 29014047 561174 0 0 54 54 3636 3636 0 553794 0 24714 6 11191428 1180757 0 0 3 3 1528 1528 0 1177695 0 17521 7 34103710 790567 0 0 51 52 4320 4320 0 781824 0 29424 8 17182670 4348 0 0 9 9 2165 2165 0 0 0 41538 9 2132311 520 0 0 0 0 260 260 0 0 0 3304 10 25432181 6308 0 0 12 12 3142 3142 0 0 0 25046 11 16063496 3832 0 0 8 8 1908 1908 0 0 0 7738 12 11072472 2756 0 0 4 4 1374 1374 0 0 0 35884 13 2789016 664 0 0 0 0 332 332 0 0 0 2683 14 11165568 2934 0 0 4 4 1463 1463 0 0 0 24254 15 20579394 4992 0 0 4 4 2492 2492 0 0 0 5738 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 41.474s ( 771.565 blocks/s, 48.223 Mb/s)
Attachment 22: Screenshot_2021-05-22_Statistics_aidas-gsi(7).png

Attachment 23: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 24: Screenshot_2021-05-22_Statistics_aidas-gsi(6).png

Attachment 25: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 26: 802.png

Attachment 27: 804.png

Attachment 28: R14_1867
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254778017 ( 1461534.6 Hz) Other data format: 7141983 ( 40970.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 228 ( 1.3 Hz) RESUME: 228 ( 1.3 Hz) SYNC100: 32649 ( 187.3 Hz) WR48-63: 32649 ( 187.3 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7076229 ( 40592.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 358653 ( 2057.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 174.322 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.023 0.000 1 8.071 0.000 2 0.956 0.000 3 6.628 0.000 4 0.186 12.353 5 5.668 0.000 6 0.255 0.000 7 8.669 0.000 8 1.570 0.000 9 0.000 28.420 10 2.125 0.000 11 0.164 0.000 12 0.103 0.000 13 0.000 11.688 14 0.094 0.000 15 1.340 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5294176 2413 0 0 1 1 685 685 0 1041 0 18715 1 22878067 6136 0 0 44 44 3024 3024 0 0 0 38998 2 11959613 1840531 0 0 7 7 1681 1681 0 1837155 0 31909 3 28914032 1781740 0 0 51 51 3845 3845 0 1773948 0 44969 4 2970776 981532 0 0 7 7 454 454 0 980610 0 8409 5 29112496 551278 0 0 40 40 3688 3688 0 543822 0 24330 6 11438278 1149690 0 0 4 4 1643 1643 0 1146396 0 17361 7 34470364 802023 0 0 41 41 4342 4342 0 793257 0 30254 8 17110979 4312 0 0 6 6 2150 2150 0 0 0 40990 9 2129568 524 0 0 0 0 262 262 0 0 0 3068 10 25946722 6422 0 0 9 9 3202 3202 0 0 0 25071 11 16268948 3960 0 0 3 3 1977 1977 0 0 0 7119 12 11320269 2746 0 0 3 3 1370 1370 0 0 0 35684 13 2799924 716 0 0 0 0 358 358 0 0 0 2469 14 11389376 2772 0 0 2 2 1384 1384 0 0 0 23994 15 20774429 5188 0 0 10 10 2584 2584 0 0 0 5313 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.000s ( 744.186 blocks/s, 46.512 Mb/s)
Attachment 29: R14_1920
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254781729 ( 1428365.9 Hz) Other data format: 7138271 ( 40018.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 226 ( 1.3 Hz) RESUME: 227 ( 1.3 Hz) SYNC100: 32665 ( 183.1 Hz) WR48-63: 32665 ( 183.1 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7072488 ( 39650.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 352885 ( 1978.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 178.373 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.017 2.836 1 6.610 0.000 2 0.573 0.000 3 7.826 0.000 4 0.024 14.831 5 7.215 0.000 6 0.482 0.000 7 6.876 0.000 8 0.394 0.000 9 0.000 17.832 10 1.157 0.000 11 1.395 0.000 12 0.037 0.000 13 0.004 9.016 14 0.101 0.000 15 1.578 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5306415 2397 0 0 1 1 665 665 0 1065 0 18998 1 22746335 6008 0 0 36 36 2968 2968 0 0 0 37812 2 11931383 1875609 0 0 5 5 1756 1756 0 1872087 0 31542 3 28584441 1795513 0 0 57 57 3773 3773 0 1787853 0 43417 4 2954259 971818 0 0 3 3 494 494 0 970824 0 8522 5 28608195 543853 0 0 51 52 3567 3567 0 536616 0 24072 6 11355091 1133078 0 0 6 6 1573 1573 0 1129920 0 16979 7 34582740 783003 0 0 34 34 4406 4406 0 774123 0 29465 8 17425265 4332 0 0 7 7 2159 2159 0 0 0 40811 9 2130016 550 0 0 0 0 275 275 0 0 0 3176 10 26903024 6616 0 0 4 4 3304 3304 0 0 0 24219 11 16400506 4036 0 0 5 5 2013 2013 0 0 0 7163 12 11085724 2934 0 0 1 1 1466 1466 0 0 0 35361 13 2664218 650 0 0 1 1 324 324 0 0 0 2574 14 11252748 2738 0 0 4 4 1365 1365 0 0 0 23421 15 20851369 5136 0 0 11 11 2557 2557 0 0 0 5353 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.672s ( 732.737 blocks/s, 45.796 Mb/s)
342	Fri May 21 15:03:48 2021	CA	Friday 21st May 16:00 - 00:00 shift
16:00 CA takes over DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4 DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4 PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 16:14 system wide checks all ok except Base Current Difference aida05 fault 0x500 : 0x570 : 112 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 16:17 temperatures ok - attachment 1 statistics ok - attachment 2 detector bias / leakage currents ok - attachment 3 analysis of R14_1625 - FEE7 deadtime at 4.5% 18:06 system wide checks all ok except Base Current Difference aida05 fault 0x500 : 0x571 : 113 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:07 temperatures ok - attachment 4 statistics ok - attachment 5 detector bias / leakage currents ok - attachment 6 analysis of R14_1662 - FEE7 deadtime at 4.3% 19:07 DAQ continues ok - writing to file R14_1682 20:07 system wide checks all ok except Base Current Difference aida05 fault 0x500 : 0x572 : 114 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 20:10 temperatures ok - attachment 7 statistics ok - attachment 8 detector bias / leakage currents ok - attachment 9 analysis of R14_1703 - max deadtime at 4.0% - attachment 10 20:35 rate spectra - attachment 11 HEC spectra - attachment 12 & 13 20:36 all histograms zeroed 20:37 DAQ continues ok - writing to file R14_1712 20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use. ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files 21:12 no beam - writing to file R14_1723 21:17 beam is back 22:07 system wide checks all ok - WR decoder fault same as entry at 20:07 22:08 temperatures ok - attachment 14 statistics ok - attachment 15 detector bias / leakage currents ok - attachment 16 analysis of R14_1740 - max deadtime at 4.2% - attachment 17
Attachment 1: Screenshot_from_2021-05-21_15-14-59.png

Attachment 2: Screenshot_from_2021-05-21_15-15-38.png

Attachment 3: Screenshot_from_2021-05-21_15-15-54.png

Attachment 4: Screenshot_from_2021-05-21_17-06-56.png

Attachment 5: Screenshot_from_2021-05-21_17-07-18.png

Attachment 6: Screenshot_from_2021-05-21_17-07-47.png

Attachment 7: Screenshot_from_2021-05-21_19-09-19.png

Attachment 8: Screenshot_from_2021-05-21_19-09-59.png

Attachment 9: Screenshot_from_2021-05-21_19-10-15.png

Attachment 10: Screenshot_from_2021-05-21_19-14-45.png

Attachment 11: Screenshot_from_2021-05-21_19-25-51.png

Attachment 12: Screenshot_from_2021-05-21_19-30-14.png

Attachment 13: Screenshot_from_2021-05-21_19-32-13.png

Attachment 14: Screenshot_from_2021-05-21_21-08-28.png

Attachment 15: Screenshot_from_2021-05-21_21-08-51.png

Attachment 16: Screenshot_from_2021-05-21_21-09-06.png

Attachment 17: Screenshot_from_2021-05-21_21-12-03.png

341	Fri May 21 07:03:50 2021	OH, TD	Friday 21 May 08:00-16:00
08:00 OH and TD take over Current settings p+n sides at 0xc DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4 DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4 PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 08:29 System wide checks ok - WR difference at 111 Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1 Temperature ok - attachment 2 Bias and leakage current ok - attachment 3 09:58 System wide checks ok - WR difference at 111 Statistics ok - attachment 4 Temperature ok - attachment 5 Bias and leakage current ok - attachment 6 Max deadtime this morning has been around 5.5% on FEE8 12:32 System wide checks ok - WR difference at 111 Statistics ok - attachment 7 Temperature ok - attachment 9 Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png

Attachment 2: 210521_0829_Temp.png

Attachment 3: 210521_0830_Bias.png

Attachment 4: 210521_0957.png

Attachment 5: 210521_0957_Temp.png

Attachment 6: 210521_0958_Bias.png

Attachment 7: 210521_1232_Stats.png

Attachment 8: 210521_1233_Bias.png

Attachment 9: 210521_1233_Temp.png

340	Fri May 21 06:09:28 2021	ML	system wide checks
System wide checks at 7am CET All seems fine, nothing significant to report Slight increase of dead time noted in FEE 3, 5 and 7. ----- Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ----- ADC calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module ----- White Rabbit: Base Current Difference aida05 fault 0x500 : 0x56e : 110 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR ----- FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last ----- Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 23 5 6 5 0 4 1 3 3 3 6 : 36484 aida02 : 10 3 3 2 3 2 2 2 3 3 6 : 35952 aida03 : 2 5 1 4 5 2 2 3 3 3 6 : 36608 aida04 : 1 3 4 0 4 3 2 3 3 3 6 : 36572 aida05 : 19 9 3 2 2 3 1 3 4 3 6 : 37380 aida06 : 13 11 1 5 3 4 4 2 3 3 6 : 36860 aida07 : 12 2 4 3 3 2 1 3 2 2 7 : 37280 aida08 : 25 5 7 1 1 3 3 3 3 3 6 : 36828 aida09 : 22 5 6 4 1 3 2 3 3 3 6 : 36640 aida10 : 23 8 6 3 2 3 2 2 2 4 6 : 37212 aida11 : 24 14 7 3 3 2 3 3 3 3 6 : 36960 aida12 : 16 10 5 3 1 2 1 2 4 3 6 : 36736 aida13 : 23 6 4 2 2 3 2 3 3 3 6 : 36620 aida14 : 22 7 8 4 3 4 1 2 2 4 6 : 37200 aida15 : 10 12 9 2 1 3 1 3 3 3 6 : 36376 aida16 : 18 5 6 2 0 3 2 4 3 3 6 : 37008
Attachment 1: 2021-05-21_7h00_LeakCurrent.png

Attachment 2: 2021-05-21_7h00_Stats_GoodEvents.png

Attachment 3: 2021-05-21_7h00_Stats_ADCitems.png

Attachment 4: 2021-05-21_7h00_stats_Correlation.png

Attachment 5: 2021-05-21_7h00_Stats_PAUSE.png

Attachment 6: 2021-05-21_7h00_Stats_RESUME.png

Attachment 7: 2021-05-21_7h00_temperature.png

Attachment 8: 2021-05-21_7h00_ucesb.png

339	Fri May 21 02:17:47 2021	ML	system wide checks
Wide check at 3am - All fine. Same output as at the beginning of the shift (see my previous entry) and attachments.
Attachment 1: 2021-05-21_3h05_Leakcurrent.png

Attachment 2: 2021-05-21_3h05_stats_adcItem.png

Attachment 3: 2021-05-21_3h05_stats_Correlation.png

Attachment 4: 2021-05-21_3h05_stats_GoodEvents.png

Attachment 5: 2021-05-21_3h05_stats_PAUSE.png

Attachment 6: 2021-05-21_3h05_stats_RESUME.png

Attachment 7: 2021-05-21_3h05_temperature.png

Attachment 8: 2021-05-21_3h05_ucesb.png

Attachment 9: 2021-05-21_3h05_aidaGrafana.png

338	Thu May 20 23:26:11 2021	Marc Labiche (ML)	system wide checks
System Wide check at begining of the night shift: ----- Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ----- ADC calibration: FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module ----- White rabbit status: Base Current Difference aida05 fault 0x500 : 0x56d : 109 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR ----- FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last ----- Memory info from FEE cards Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 7 7 6 1 5 2 2 3 3 6 : 36472 aida02 : 19 5 4 4 4 3 3 3 2 3 6 : 36020 aida03 : 22 10 2 2 5 2 2 3 3 3 6 : 36680 aida04 : 12 7 1 0 4 3 1 4 3 3 6 : 36856 aida05 : 16 7 5 3 1 3 2 3 3 3 6 : 36584 aida06 : 20 7 5 6 2 4 4 2 3 3 6 : 36888 aida07 : 13 7 5 2 3 2 1 3 2 2 7 : 37308 aida08 : 18 7 5 3 0 2 3 3 3 3 6 : 36656 aida09 : 21 6 4 4 1 4 2 3 3 3 6 : 36740 aida10 : 21 10 2 3 2 3 2 2 2 4 6 : 37156 aida11 : 23 2 4 4 3 2 2 3 2 4 6 : 37612 aida12 : 24 1 4 3 2 2 1 2 4 3 6 : 36744 aida13 : 26 9 4 2 1 4 3 2 3 3 6 : 36464 aida14 : 19 9 9 2 3 4 1 2 2 4 6 : 37156 aida15 : 10 7 10 0 1 3 1 3 3 3 6 : 36288 aida16 : 28 7 3 2 1 3 3 3 3 3 6 : 36824
Attachment 1: 2021-05-21_0h20_ucesb.png

Attachment 2: 2021-05-21_0h10_Leakcurrent.png

Attachment 3: 2021-05-21_0h10_Stat_adcItem.png

Attachment 4: 2021-05-21_0h10_Stats_Correlation.png

Attachment 5: 2021-05-21_0h10_Stats_GoodEvents.png

Attachment 6: 2021-05-21_0h10_Stats_PAUSE.png

Attachment 7: 2021-05-21_0h10_stats_RESUME.png

Attachment 8: 2021-05-21_0h10_temperature.png

Attachment 9: 2021-05-21_0h30_AIDAgrafana.png

337	Thu May 20 14:56:33 2021	OH	Thursday 20th May 16:00 - 24:00
16:00 p+n threshold 0xc (120keV) n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4 n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4 Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure. 17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector 18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now. The deadtimes in FEE6 and 8 have risen to around 6.25%. Looking at this mornings and yesterday evenings fill times this behavior isn't observed 18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago 18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2 This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file) 18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8 18:15 System wide checks ok except WR as usual Base Current Difference aida05 fault 0x500 : 0x56d : 109 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics increased at 18:02 still working on - attachment 2 FEE temp - attachment 3 Bias and leakage current - attachment 4 20:51 System wide checks ok - WR same as before Statistics still high - attachment 5 Temperatures ok - attachment 6 Bias and leakage currents ok - attachment 7 22:52 Statistics still high - attachment 8 Temp ok - attachment 9 Bias and leakage current ok - attachment 10 System wide checks ok - WR difference still 109
Attachment 1: 210520_18002_MBSRate.png

Attachment 2: 210520_1916_Stats.png

Attachment 3: 210520_1916_Temp.png

Attachment 4: 210520_1917_Bias.png

Attachment 5: 21050_2052_Stats.png

Attachment 6: 210520_2052_Temp.png

Attachment 7: 210520_2053_Bias.png

Attachment 8: 210520_2251_Stats.png

Attachment 9: 210520_2251_Temp.png

Attachment 10: 210520_2254_Bias.png

Goto page Previous 1, 2, 3 ... 17, 18, 19 ... 33, 34, 35 Next

ELOG V3.1.4-unknown