AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 15 of 36  ELOG logo
Entry  Sun Jul 14 16:52:28 2024, TD, S181 R7_38-44 7x
 
Entry  Sun Jul 14 14:37:53 2024, TD, S181 R6_375-380 6x
 
Entry  Thu Sep 8 12:31:25 2022, NH, Retrying AIDA DataAcq v10 AnyDeskMSI_2022-09-08_13-35-50.pngAnyDeskMSI_2022-09-08_13-35-58.pngAnyDeskMSI_2022-09-08_13-36-02.png
Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 
Entry  Fri Mar 22 13:47:32 2019, NH, Resolution Checks 20190322Resolutions.png20190323Waves.png20190322Rates.png

Pulser widths of AIDA modules, resolution is fairly typical. AIDA09 still noisier than other DSSD channels.

Figure 1: Picture of pulser peaks
Figure 2: Raw waves (AIDA01 did not calibrate, so AIDA02 used instead as "non DSSD module")
Figure 3: Good Events

Fast comparator threshold (LEC/MEC) changed to 0xFF

AIDA09 seems noiser than before(?) but is known to be noisy.

AIDA01:  24.22

AIDA09: 205.21

AIDA10: 88.60

AIDA11: 63.33

AIDA12: 102.69

Entry  Mon Nov 4 13:17:12 2019, NH, Report: aida10 database corruption db_working.txtdb_corrupt.txt

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

    Reply  Tue Nov 5 10:02:45 2019, NH PJCS, Report: aida10 database corruption 

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could well cause you problems with readout as it won't understand some of the new data.

Hope this helps.

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

    Reply  Tue Nov 5 10:32:21 2019, NH PJCS, Report: aida10 database corruption 

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

 

Quote:

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could welll cause you problems with readout as it won't understand some of the new data.

 

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

 

    Reply  Tue Nov 5 11:15:41 2019, NH PJCS, Report: aida10 database corruption 

If ypou now check the actual values in the aida10 Options screen are the correct ?

It can happen that corruptions once loaded are propagated when Options are generally saved. 

Looking at the dates the files were modified does that make sense to you. It doesn't look as if the corrupted database was changed except when they all were ? Can you check ? Have you checked the other Options files ? I seem to recall that RIKEN AIDA had a program to to this ?

Quote:

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

 

Quote:

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could welll cause you problems with readout as it won't understand some of the new data.

 

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

 

 

Entry  Mon Feb 17 09:48:34 2020, NH, Report: aida09 Kernel Panic & Lost WR  ttyUSB12GSI_WR.pngGSI_WR2.pngGSI_WR4.pngGSI_WR3.png
aida09 crashed over the weekend and automatically rebooted.
After the reboot the WR timestamp sent to the merger is in the future and hence incorrect

Reset/Setup did not fix issue
Sync ASICs did not fix issue
GSI White Rabbit control page shows a correct WR timestamp

Attach 1: ttyUSB12 (aida09 log with kernel panic)
Attach 2: GSI White Rabbit control page
Attach 3: "Collect All WR Timestamps"
Attach 4: RAW Display for aida09
 
WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000

WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
c.f. "GSI page" timestamp starting 0x15F427F5 

Attach 5: Timestamp shown by merger
    Reply  Wed Feb 19 10:58:29 2020, NH, PJCS, Report: aida09 Kernel Panic & Lost WR  
> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
> 
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
> 
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>  
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> 
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5 
> 
> Attach 5: Timestamp shown by merger

Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of 
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a 
logical "rollover" of the lower 14 bits of this 10nS timestamp.


Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?


It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.

The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be 
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.

If this occurs again then the system wide check results would be interesting.
Entry  Mon Apr 22 09:03:54 2024, TD, Report: aida04 stops producing data Screenshot_from_2024-04-22_09-49-30.png
09.14 aida04 stopped producing data - system console output - attachment 1

It is not clear (to me at least) whether the cause is radiation-induced single event upsets or
the start/stop of LN2 fills.

Nor is it clear why the FEE64 stops
- WR timestamp error (data link to Merger blocks)
- AIDAExecV10 fails
- ?

rebooted aida04

DAQ RESET/SETUP etc cycle for all FEE64s

10.03 successful restart 
      current data file R15
Entry  Mon Mar 9 12:47:18 2020, NH, Report: Unusual AIDA Timestamp Error  
Noted using my AIDA event builder code, a timewarp was reported and I noticed the ADC event had a different timestamp LSB to the "WR" markers preceding it, which I didn't think was possible in the merger as it stands

 aida event 875015fa 0f68098c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68098c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68098c
 aida event c1f6851b 0f68098c | ADC 08:54 L 851b
 event time: 15fa4b164f68098c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1ef8238 0f68115c | ADC 08:47 L 8238
 event time: 15fa4b164f68115c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1f884b0 0f68115c | ADC 08:56 L 84b0
 event time: 15fa4b164f68115c
 aida event 8a5015fa 0f681382 | INFO 5 SYNC6348 11 015fa
 aida event 8a44b164 0f681382 | INFO 4 SYNC4828 11 4b164
 event time: 15fa4b164f681382
 aida event c2817dc8 0f681382 | ADC 11:01 L 7dc8
 event time: 15fa4b164f681382
 aida event 875015fa 0f68192c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68192c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68192c
 aida event c1f9849c 0f68192c | ADC 08:57 L 849c
 event time: 15fa4b164f68192c
 aida event 815015fa 0f681f94 | INFO 5 SYNC6348 02 015fa
 aida event 8144b164 0f681f94 | INFO 4 SYNC4828 02 4b164
 event time: 15fa4b164f681f94
 aida event c0568084 04b6854a | ADC 02:22 L 8084
 event time: 15fa4b1644b6854a
AIDA Timewarp (15fa4b1644b6854a before 15fa4b164f68192c)
Event reported errors, skipping this file...
Events: 506459240   506459240             (0 errors)

Unsure if the cause, easily handled by ignoring such events but should be "impossible"
Entry  Wed Apr 3 12:42:57 2024, NH, Report aida02 WR errors Screenshot_from_2024-04-03_13-43-48.pngScreenshot_from_2024-04-03_13-43-38.pngScreenshot_from_2024-04-03_13-43-58.pngScreenshot_from_2024-04-03_13-44-14.pngScreenshot_from_2024-04-03_13-44-36.png

The WR error counter for aida02 seems to consantly rise

Tried reseating cable on both ends, no change

However clock status passed, aida02 has a correct WR timestamp and no FIFO/PLL errors seen

Edit to add: aida02 has the faulty ASIC temperature readout as well, related or coincidence?

Entry  Thu Dec 19 10:11:59 2019, TD, Report - medium - FEE64 panics during boot 300.png
Some of the FEE64s aida01 .. aida12 panic during boot 

Frequencies of panics for each FEE64 can be seen in attachment 1

Below is an example of an aida04 panic following a power cycle and an automatic reboot

pi@raspberrypi:~/logs $ ./tail_aida aida04
aida04
ttyUSB7
19/11:07:04|LR [203f5680] 0x203f5680
19/11:07:04|Call Trace:
19/11:07:04|Kernel panic - not syncing: Fatal exception in interrupt
19/11:07:04|Call Trace:
19/11:07:04|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/11:07:04|[c6941e20] [c00345bc] panic+0x94/0x168
19/11:07:04|[c6941e70] [c000bd44] die+0x178/0x18c
19/11:07:04|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
19/11:07:04|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
19/11:07:04|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
19/11:10:05|
Found 0 errors checking kernel image
19/11:10:06|VHDL version number 0X18430701 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
19/11:10:06|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/11:10:06|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/11:10:06|SPIconfig done now to set up the LMK3200 registers
19/11:10:06|LMK #0 : regInit[0]=0x80000000
19/11:10:07|LMK #0 : regInit[1]=0x10070600
19/11:10:07|LMK #0 : regInit[2]=0x60601
19/11:10:07|LMK #0 : regInit[3]=0x60602
19/11:10:07|LMK #0 : regInit[4]=0x60603
19/11:10:07|LMK #0 : regInit[5]=0x70624
19/11:10:07|LMK #0 : regInit[6]=0x70605
19/11:10:07|LMK #0 : regInit[7]=0x70606
19/11:10:07|LMK #0 : regInit[8]=0x70627
19/11:10:07|LMK #0 : regInit[9]=0x10000908
19/11:10:07|LMK #0 : regInit[10]=0xA0022A09
19/11:10:07|LMK #0 : regInit[11]=0x82800B
19/11:10:07|LMK #0 : regInit[12]=0x28C800D
19/11:10:07|LMK #0 : regInit[13]=0x830020E
19/11:10:07|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/11:10:07|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
19/11:10:07|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/11:10:08|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
19/11:10:08|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/11:10:11|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xb7 
19/11:10:17|Using Xilinx Virtex440 machine description
19/11:10:18|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/11:10:18|Zone PFN ranges:
19/11:10:18|  DMA      0x00000000 -> 0x00007000
19/11:10:18|  Normal   0x00007000 -> 0x00007000
19/11:10:18|Movable zone start PFN for each node
19/11:10:18|early_node_map[1] active PFN ranges
19/11:10:18|    0: 0x00000000 -> 0x00007000
19/11:10:18|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/11:10:18|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/11:10:18|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/11:10:18|PID hash table entries: 512 (order: 9, 2048 bytes)
19/11:10:19|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/11:10:19|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/11:10:19|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/11:10:19|Kernel virtual memory layout:
19/11:10:19|  * 0xffffe000..0xfffff000  : fixmap
19/11:10:19|  * 0xfde00000..0xfe000000  : consistent mem
19/11:10:19|  * 0xfde00000..0xfde00000  : early ioremap
19/11:10:19|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/11:10:19|NR_IRQS:512
19/11:10:19|clocksource: timebase mult[a00000] shift[22] registered
19/11:10:19|Console: colour dummy device 80x25
19/11:10:19|Mount-cache hash table entries: 512
19/11:10:19|NET: Registered protocol family 16
19/11:10:19|PCI: Probing PCI hardware
19/11:10:19|bio: create slab <bio-0> at 0
19/11:10:19|NET: Registered protocol family 2
19/11:10:19|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/11:10:19|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/11:10:19|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/11:10:19|TCP: Hash tables configured (established 4096 bind 4096)
19/11:10:20|TCP reno registered
19/11:10:20|NET: Registered protocol family 1
19/11:10:20|ROMFS MTD (C) 2007 Red Hat, Inc.
19/11:10:20|msgmni has been set to 214
19/11:10:20|io scheduler noop registered
19/11:10:20|io scheduler anticipatory registered
19/11:10:20|io scheduler deadline registered
19/11:10:20|io scheduler cfq registered (default)
19/11:10:20|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/11:10:20|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/11:10:20|console [ttyS0] enabled
19/11:10:20|brd: module loaded
19/11:10:20|loop: module loaded
19/11:10:20|Device Tree Probing 'ethernet'
19/11:10:20|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:b7
19/11:10:20|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/11:10:20|XLlTemac: DCR address: 0x80
19/11:10:20|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/11:10:20|XLlTemac: Allocating DMA descriptors with kmalloc
19/11:10:20|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/11:10:20|XTemac: PHY detected at address 7.
19/11:10:20|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/11:10:21|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21|Using buffer write method
19/11:10:21|cfi_cmdset_0001: Erase suspend on write enabled
19/11:10:21|cmdlinepart partition parsing not available
19/11:10:21|RedBoot partition parsing not available
19/11:10:21|Creating 5 MTD partitions on "fc000000.flash":
19/11:10:21|0x000000000000-0x000000500000 : "golden_firmware"
19/11:10:21|0x000000500000-0x000000800000 : "golden_kernel"
19/11:10:21|0x000000800000-0x000000d00000 : "user_firmware"
19/11:10:21|0x000000d00000-0x000000fe0000 : "user_kernel"
19/11:10:21|0x000000fe0000-0x000001000000 : "env_variables"
19/11:10:21|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/11:10:21|SPI: XIlinx spi: bus number now 32766
19/11:10:21|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/11:10:22|SPI: XIlinx spi: bus number now 32765
19/11:10:22|mice: PS/2 mouse device common for all mice
19/11:10:22|Device Tree Probing 'i2c'
19/11:10:22| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/11:10:22|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/11:10:22|TCP cubic registered
19/11:10:22|NET: Registered protocol family 17
19/11:10:22|RPC: Registered udp transport module.
19/11:10:22|RPC: Registered tcp transport module.
19/11:10:22|eth0: XLlTemac: Options: 0x3fa
19/11:10:22|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/11:10:23|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/11:10:23|eth0: XLlTemac: speed set to 1000Mb/s
19/11:10:25|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/11:10:25|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/11:10:25|Sending DHCP requests ., OK
19/11:10:26|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
19/11:10:26|IP-Config: Complete:
19/11:10:26|     device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
19/11:10:26|     host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/11:10:26|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
19/11:10:26|Looking up port of RPC 100003/2 on 192.168.11.99
19/11:10:26|Looking up port of RPC 100005/1 on 192.168.11.99
19/11:10:26|VFS: Mounted root (nfs filesystem) on device 0:12.
19/11:10:26|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/11:10:27|Starting sysinit...
19/11:10:27|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/11:10:27|		Press 'I' to enter interactive startup.
19/11:10:27|Setting clock  (utc): Thu Dec 19 10:10:28 GMT 2019 [  OK  ]
19/11:10:28|Building the cache [  OK  ]
19/11:10:28|Setting hostname aida04:  [  OK  ]
19/11:10:29|Mounting local filesystems:  [  OK  ]
19/11:10:30|Enabling /etc/fstab swaps:  [  OK  ]
19/11:10:32|Finishing sysinit...
INIT: Entering runlevel: 3
19/11:10:35|Entering non-interactive startup
19/11:10:36|FATAL: Module ipv6 not found.
19/11:10:37|Bringing up loopback interface:  [  OK  ]
19/11:10:39|FATAL: Module ipv6 not found.
19/11:10:39|Starting system logger: [  OK  ]
19/11:10:40|Starting kernel logger: [  OK  ]
19/11:10:40|Starting rpcbind: [  OK  ]
19/11:10:41|Mounting NFS filesystems:  [  OK  ]
19/11:10:42|Mounting other filesystems:  [  OK  ]
19/11:10:42|Starting xinetd: [  OK  ]
19/11:10:43|Starting midas:  Starting MIDAS Data Acquisition for aida04
19/11:10:43|xaida: device parameters: base=0x81000000 size=0x200000
19/11:10:48|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/11:10:49|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/11:10:49|xaida: driver assigned major number 254
19/11:10:49|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/11:10:54|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/11:10:54|AIDAMEM: aidamem: driver assigned major number 253
19/11:10:54|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/11:11:01|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/11:11:01|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/11:11:01|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/11:11:01|Computer Name = aida04; Temp Directory = /tmp/tcl361
19/11:11:05|package limit is not available: can't find package limit
19/11:11:07|Running with default file descriptor limit
19/11:11:07|package setuid is not available: can't find package setuid
19/11:11:09|Running as user 0 group 0
19/11:11:09|
19/11:11:10|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
19/11:11:10|
19/11:11:10|Built without pthreaxaida: open:
19/11:11:10|ds
19/11:11:10|
19/11:11:10|Creating NetAIDAMEM: aidamem_open:
19/11:11:10|Vars
19/11:11:10|Output buffer length = 65504; format option = 4; transfer option = 3
19/11:11:11|EB transfer option = 3
19/11:11:11|NetVars created and initialised
19/11:11:11|Statistics thread starting
19/11:11:11|Statistics thread created
19/11:11:11|Stat/Rate creation thread starting
19/11:11:11|Data Acquisition task has PID 375
19/11:11:11|Stat/Rate creation thread created
19/11:11:11|Hit/Rate creation thread starting
19/11:11:11|Hit/Rate creation thread created
19/11:11:11|AIDA Heartbeat thread starting
19/11:11:11|Heartbeat thread created
19/11:11:11|Installing signal handlers
19/11:11:11|Done
19/11:11:11|ModuleNum = 0
19/11:11:11|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 18430701 
19/11:11:11|Spectra table initialised
19/11:11:11|AIDA Data Acquisition now all ready to start
19/11:11:11|SIGBUS, SIGSEGV and SIGPIPE traps setup
19/11:11:11|/debug user "debug" password "-f9x7ruru8cg"
19/11:11:17|httpd started on port 8015
19/11:11:18|
19/11:11:18|Cannot use /MIDAS/config/TclHttpd/aida04@8015/startup.tcl
19/11:11:18|Custom startup from /MIDAS/config/TclHttpd/aida04/startup.tcl
19/11:11:18|XAIDA Access package 1.0
19/11:11:19|/XAIDAAccessServer
19/11:11:19|XAD9252 Access package 1.0
19/11:11:20|/XAD9252AccessServer
19/11:11:20|/DataBaseAccessServer
19/11:11:20|/NetVarService
19/11:11:20|/SigTaskService
19/11:11:20|Loaded MemSasAccess
19/11:11:20|/SpectrumService
19/11:11:20|loading tcl/AIDARunControl.tcl for namespace ::
19/11:11:20|/DataAcquisitionControlServer
19/11:11:20|DefineMessage unknown
19/11:11:20|Run Control Server Implementation for AIDA
19/11:11:21|RunControlServer loaded
19/11:11:21|loading Html/RunControl/implementation.tcl
19/11:11:21|[  OK  ]
19/11:11:21|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
19/11:11:21|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
19/11:11:21|
19/11:11:21|DENX ELDK version 4.2 build 2008-04-01
19/11:11:21|Linux 2.6.31 on a ppc
19/11:11:21|
19/11:11:21|aida04 login: Created UI registers
19/11:11:22|RunControl loaded
19/11:11:22|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
19/11:11:22|AIDA RunControl loaded
19/11:11:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
Entry  Thu Dec 19 14:07:38 2019, TD, Report - low - aida08 ADC spectrum size 32k cf. 64k channels expcted using layout 6x
Note that aida08 spectrum 1.8.L size is (shown as) 32k channels cf. 64k channels expected using layout - attachment 1 - 5
Layout configuration - attachment 6
Why?

Current AIDA Options are shown below

[npg@aidas-gsi Options]$ pwd
/MIDAS/DB/EXPERIMENTS/AIDA/Options
[npg@aidas-gsi Options]$ csh
[npg@aidas-gsi Options]$ foreach f (`ls`)
foreach? echo $f
foreach? cat "$f"/CONTENTS
foreach? end
aida01
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
Rate.channels string 512
RunNumber string 12
Aida_GroupBase string 1
WAVE_DMA_HWM string 0x0007ffff
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
Aida_Hist_D_Enable string 1
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida02
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&ASIC.settings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable 
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida03
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida04
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Aida_GroupBase string 1
Rate.channels string 512
Stat.channels string 512
RunNumber string 12
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
MERGE.LinksAvailable string 4
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida05
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida06
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida07
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida08
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida09
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida10
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida11
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&ASIC.settings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida12
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
Entry  Fri Feb 21 12:16:06 2020, TD, NH, Report - high - unusual FEE64 crash during startup ttyUSB4
 
Entry  Wed Apr 3 12:09:47 2024, NH, Report - aida06 frequently fails to boot first time (PHY error) ttyUSB15

When booting up AIDA aida06 usually crashes the first time, it fails to get IP from DHCP
After 180 seconds it reboots and seems to connect fine

Log file attached, key part (to me) is this:

27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|
Entry  Wed Jan 8 10:46:50 2025, TD, Report - aida06 -boot fail due to network issue 
8:01:25/11:42:48|0x000000d00000-0x000000fe0000 : "user_kernel"
08:01:25/11:42:48|0x000000fe0000-0x000001000000 : "env_variables"
08:01:25/11:42:48|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32766
08:01:25/11:42:48|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32765
08:01:25/11:42:48|mice: PS/2 mouse device common for all mice
08:01:25/11:42:48|Device Tree Probing 'i2c'
08:01:25/11:42:48| #0 at 0x81600000 mapped to 0xD1030000, irq=22
08:01:25/11:42:48|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
08:01:25/11:42:48|TCP cubic registered
08:01:25/11:42:48|NET: Registered protocol family 17
08:01:25/11:42:48|RPC: Registered udp transport module.
08:01:25/11:42:49|RPC: Registered tcp transport module.
08:01:25/11:42:49|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:42:49|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:42:51|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:42:51|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:42:51|Sending DHCP requests .
08:01:25/11:42:53|eth0: XLlTemac: PHY Link carrier lost.
08:01:25/11:42:53|..... timed out!
08:01:25/11:44:15|IP-Config: Reopening network devices...
08:01:25/11:44:15|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:44:16|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:44:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:44:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:44:18|Sending DHCP requests ...... timed out!
08:01:25/11:45:35|IP-Config: Auto-configuration of network failed.
08:01:25/11:45:35|Root-NFS: No NFS server available, giving up.
08:01:25/11:45:35|VFS: Unable to mount root fs via NFS, trying floppy.
08:01:25/11:45:35|VFS: Cannot open root device "nfs" or unknown-block(2,0)
08:01:25/11:45:35|Please append a correct "root=" boot option; here are the available partitions:
08:01:25/11:45:35|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
08:01:25/11:45:35|Call Trace:
08:01:25/11:45:35|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
08:01:25/11:45:35|[c6827f10] [c00345bc] panic+0x94/0x168
08:01:25/11:45:35|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
08:01:25/11:45:36|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
08:01:25/11:45:36|[c6827fd0] [c0341220] kernel_init+0x104/0x130
08:01:25/11:45:36|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
08:01:25/11:45:36|Rebooting in 180 seconds..
Entry  Wed Apr 10 14:53:50 2019, NH, Report - FEE stops sending data 6x

it seems a FEE somtimes enters a confusing state and stops sending data
the current merger requires all FEEs to be active and so this stops the entire system from proceeding.

On MIDAS the page reports the module is "undefined"

On the TTY console (PUTTY) it returns: do_GetState returned z=0 and 8

Resetting the DAQ in question via MIDAS works (Putty logs of the stages shown) and then the merger resumes without trouble.

    Reply  Fri Apr 12 15:15:33 2019, NH, Report - FEE Kernel Panics (Update on 48) aida01_log.txt

Update on issue #48 - the "confusing state" is that the FEE has restarted and hence is undefined again.

An attached TTY log from the pi shows that the module is kernel panicking.
I have seen a couple of FEEs panic with the same error now.

(NB. The Day/time of the logs is wrong as the pi does not have the correct time - pis dont have a RTC or Internet access so the time isn't corrected)

Temperature of the modules is fine.

Aida is currently powered off (and I am away from GSI)

Quote:

it seems a FEE somtimes enters a confusing state and stops sending data
the current merger requires all FEEs to be active and so this stops the entire system from proceeding.

On MIDAS the page reports the module is "undefined"

On the TTY console (PUTTY) it returns: do_GetState returned z=0 and 8

Resetting the DAQ in question via MIDAS works (Putty logs of the stages shown) and then the merger resumes without trouble.

 

ELOG V3.1.3-7933898