ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 21 of 35

Full | Summary | Threaded | Hide attachments

694 Entries

Goto page Previous 1, 2, 3 ... 20, 21, 22 ... 33, 34, 35 Next

ID	Date	Author	Subject
303	Mon May 10 08:49:37 2021	TD	Monday 10 May
09.51 All system wide checks OK except Base Current Difference aida05 fault 0x8441 : 0x8442 : 1 aida06 fault 0xc5c5 : 0xc5c6 : 1 aida07 fault 0x6f07 : 0x6f08 : 1 aida08 fault 0x4814 : 0x4815 : 1 White Rabbit error counter test result: Passed 12, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 4 6 3 5 5 2 3 2 3 6 : 36004 aida02 : 4 1 3 6 3 4 3 3 2 3 6 : 36040 aida03 : 12 8 5 2 3 3 3 4 2 3 6 : 36416 aida04 : 1 2 5 4 1 4 3 4 2 3 6 : 36388 aida05 : 22 8 4 2 4 5 1 4 1 4 6 : 37272 aida06 : 5 4 0 6 3 6 3 3 2 3 6 : 36276 aida07 : 38 16 7 7 1 6 1 3 3 3 6 : 37032 aida08 : 16 4 2 3 1 4 1 3 3 3 6 : 36384 aida09 : 2 3 7 6 5 5 2 3 2 3 6 : 36112 aida10 : 42 15 4 6 2 6 2 4 2 3 6 : 36768 aida11 : 1 5 13 7 3 3 3 3 2 3 6 : 36124 aida12 : 23 9 3 4 7 3 2 3 2 3 6 : 35988 aida13 : 2 6 7 3 4 4 3 2 2 3 6 : 35592 aida14 : 17 11 3 4 3 5 2 3 2 3 6 : 35980 aida15 : 4 7 5 2 1 6 2 3 2 3 6 : 35864 aida16 : 4 1 3 3 4 2 3 4 2 3 6 : 36264 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 09 15:10:52 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 09.52 All histograms, stats and merger stats zero'd FEE64 temps OK - attachment 1 ADC data, disc/WR 28-27/pause/correlation stats OK - attachments 2-6 Rate spectra - attachment 7 ASIC check load Rate spectra - attachment 8 1.8.W spectra 20us FSR - attachments 9 & 10 Grafana - most recent 7 days DSSSD bias and leakage current - attachment 11 Merger stats/tape server/merger - attachment 12 DSSSD bias and leakage current OK - attachment 13 10.08 1.8.W spectra 20us, 200us, 2ms & 20ms FSR - attachments 14-21 we are not missing extrinsic noise features at longer time ranges cf. 20us FSR 12.18 Cannot connect to port 8015, MIDAS terminal says "Killed" Restart MIDAS 8015 Collect FPGA baseline to avoid TCL error FEEs unaffected and running as before 13.25 FEEs power cycled to see if any change in weird noise/rates situation No change ASICs synchronised and back to normal 14.34 Start a pulser walkthrough in May21/R10 Attenuation 5x, Rep Rate 25 Hz Amplitudes 10..9...1 14.45 Back to 2 Hz @ 2V Pulser, walkthrough complete. No Storage again. PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 (NB. actually it is 12 MeV) Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 OK
Attachment 1: Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-05-10_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-10_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-10_Statistics_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-10_Statistics_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-10_Statistics_aidas-gsi(4).png

Attachment 7: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png

Attachment 8: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png

Attachment 9: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png

Attachment 10: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png

Attachment 11: Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png

Attachment 12: 510.png

Attachment 13: 511.png

Attachment 14: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png

Attachment 15: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png

Attachment 16: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png

Attachment 17: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png

Attachment 18: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png

Attachment 19: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png

Attachment 20: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png

Attachment 21: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png

302	Sun May 9 19:29:23 2021	TD	Sunday 9 May

Attachment 1: Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-05-09_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-09_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-09_Statistics_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-09_Statistics_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png

Attachment 7: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png

Attachment 8: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png

Attachment 9: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png

Attachment 10: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png

Attachment 11: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png

Attachment 12: Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png

Attachment 13: 503.png

Attachment 14: 504.png

301	Sat May 8 09:05:04 2021	TD	Saturday 8 May
09.54 FEE64 temperatures OK - attachment 1 ADC data item stats OK - attachment 2 p+n FEE64s 11x <100k, 1x <200k Rate spectra - attachments 3 & 4 before and after ASIC check load 1.8.L spectra - attachments 5 & 6 1.8.W spectra 20us FSR - attachments 7 & 8 Disc, WR-28-47, pause, resume info data items - attachments 9-12 note large number pause/resume for aida13 - legacy numbers since last restart/zero? attachment 16 for pause stats since stats zero at 10.02 - looks OK Grafana DSSSD bias & leakage currents for most recent 48h OK - atatchment 13 DSSSD bias & leakage currents - attachment 14 Merger stats/merger/tape server messages - attachment 15 10.02 All histograms and stats zero'd 10.06 All system wide checks OK except Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 06 15:29:51 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 10.25 Pulser peak width aida01 97 ch FWHM, aida02 350 ch FWHM peak widths are increasing cf. earlier this week - see https://elog.ph.ed.ac.uk/DESPEC/291 , https://elog.ph.ed.ac.uk/DESPEC/298
Attachment 1: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-05-08_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(1).png

Attachment 5: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(2).png

Attachment 6: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(3).png

Attachment 7: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(4).png

Attachment 8: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(5).png

Attachment 9: Screenshot_2021-05-08_Statistics_aidas-gsi(1).png

Attachment 10: Screenshot_2021-05-08_Statistics_aidas-gsi(2).png

Attachment 11: Screenshot_2021-05-08_Statistics_aidas-gsi(3).png

Attachment 12: Screenshot_2021-05-08_Statistics_aidas-gsi(4).png

Attachment 13: Screenshot_2021-05-08_AIDA_Alerting_-_Grafana.png

Attachment 14: 500.png

Attachment 15: 501.png

Attachment 16: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png

Attachment 17: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png

Attachment 18: Screenshot_2021-05-08_Statistics_aidas-gsi(6).png

Attachment 19: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 20: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(6).png

Attachment 21: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(7).png

Attachment 22: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(8).png

Attachment 23: Screenshot_2021-05-08_Statistics_aidas-gsi(7).png

Attachment 24: Screenshot_2021-05-08_Statistics_aidas-gsi(8).png

Attachment 25: Screenshot_2021-05-08_Statistics_aidas-gsi(9).png

Attachment 26: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(9).png

Attachment 27: Screenshot_2021-05-08_Statistics_aidas-gsi(10).png

300	Fri May 7 19:47:54 2021	PJCS	HowTo mitigate excessive temperature in an FEE64
After tests in the Daresbury T9 system. Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees. To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected. Alternatively STOP acquisition, set the ADC Control register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.
299	Fri May 7 13:11:22 2021	NH	Friday 7th May
14:11 - Alpha has been running most of morning Just saw rates in tape spike to 6 MB/s... Stop output to tape and look: ASIC check & load... all rates except aida04 back to 0 Start output to tape Back to ~300 KB/s R6_49 will be affected by this. Others seem OK System wide check failures: Base Current Difference aida01 fault 0xb405 : 0xb406 : 1 aida02 fault 0xefc7 : 0xefc8 : 1 aida03 fault 0xdaab : 0xdaac : 1 aida04 fault 0x8f7c : 0x8f7d : 1 aida05 fault 0xb5bd : 0xb5be : 1 aida06 fault 0xeff1 : 0xeff2 : 1 aida07 fault 0x8f57 : 0x8f58 : 1 aida08 fault 0xbef7 : 0xbef8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x200 : 512 aida13 fault 0x0 : 0x61 : 97 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page) Temps & Rates OK Will now stop Alpha run and check noise situation Run stopped at R6_49 Switching to nostorage (NOTAPE/R2) Set ASICs to threshold 0xa, Rates OK Rate to Tape/MBS ca 9 MB/s No change to situation since yesterday... good sign During MBS test aida13 rebooted... crash log from ttyUSB5 attached It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again... cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :) A power cycle of all FEEs was performed after aida13's reboot to make sure things are good. All systems check OK
Attachment 1: elog_locks.png

Attachment 2: elog_temps.png

Attachment 3: elog_rates.png

Attachment 4: elog_rates1.png

Attachment 5: aida13.txt
06/19:25:39\|Data Acquisition Statistics counters now cleared^M 06/19:25:55\|Clear Statistics (1)^M 06/19:25:55\|------------[ cut here ]------------^M 07/15:23:58\|kernel BUG at mm/slab.c:2974!^M 07/15:23:58\|Oops: Exception in kernel mode, sig: 5 [#1]^M 07/15:23:58\|PREEMPT Xilinx Virtex440^M 07/15:23:59\|Modules linked in: aidamem xdriver xh_spidev_register^M 07/15:23:59\|NIP: c009211c LR: c00920b0 CTR: 00000007^M 07/15:23:59\|REGS: c0391bf0 TRAP: 0700 Not tainted (2.6.31)^M 07/15:23:59\|MSR: 00021000 <ME,CE> CR: 24022048 XER: 00000000^M 07/15:23:59\|TASK = c036e318[0] 'swapper' THREAD: c0390000^M 07/15:23:59\|GPR00: 00000001 c0391ca0 c036e318 c680daf0 c694003c 0000000a c6940020 00000009 ^M 07/15:23:59\|GPR08: 0000001b c680dae0 00000cf0 c680dae0 24008042 00005aa8 c03a0000 00000020 ^M 07/15:23:59\|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 ^M 07/15:23:59\|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000006 c680e400 ^M 07/15:23:59\|NIP [c009211c] cache_alloc_refill+0x130/0x608^M 07/15:23:59\|LR [c00920b0] cache_alloc_refill+0xc4/0x608^M 07/15:23:59\|Call Trace:^M 07/15:23:59\|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)^M 07/15:23:59\|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M 07/15:23:59\|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M 07/15:23:59\|[c0391d40] [c0042938] __send_signal+0x78/0x260^M 07/15:23:59\|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M 07/15:24:00\|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M 07/15:24:00\|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M 07/15:24:00\|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M 07/15:24:00\|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M 07/15:24:00\|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M 07/15:24:00\|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M 07/15:24:00\|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M 07/15:24:00\|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M 07/15:24:00\|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M 07/15:24:00\|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M 07/15:24:00\|[c0391fb0] [c000172c] rest_init+0x70/0x84^M 07/15:24:00\|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M 07/15:24:00\|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M 07/15:24:00\|Instruction dump:^M 07/15:24:00\|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 ^M 07/15:24:00\|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 ^M 07/15:24:00\|Kernel panic - not syncing: Fatal exception in interrupt^M 07/15:24:00\|Call Trace:^M 07/15:24:00\|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)^M 07/15:24:00\|[c0391a80] [c00345bc] panic+0x94/0x168^M 07/15:24:01\|[c0391ad0] [c000bd44] die+0x178/0x18c^M 07/15:24:01\|[c0391af0] [c000c000] _exception+0x164/0x1b4^M 07/15:24:01\|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c^M 07/15:24:01\|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608^M 07/15:24:01\|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M 07/15:24:01\|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M 07/15:24:01\|[c0391d40] [c0042938] __send_signal+0x78/0x260^M 07/15:24:01\|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M 07/15:24:01\|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M 07/15:24:01\|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M 07/15:24:01\|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M 07/15:24:01\|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M 07/15:24:01\|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M 07/15:24:01\|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M 07/15:24:01\|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M 07/15:24:01\|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M 07/15:24:01\|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M 07/15:24:01\|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M 07/15:24:01\|[c0391fb0] [c000172c] rest_init+0x70/0x84^M 07/15:24:01\|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M 07/15:24:02\|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M 07/15:24:02\|Rebooting in 180 seconds.. 07/15:27:02\|^MISOL Version 1.00 Date 9th January 2017
298	Thu May 6 16:59:20 2021	TD, OH	Wednesday 6 May
18.09 NH has installed jumpers LK2-4 for all adaptor PCBs per FEE64 rate spectra - attachments 1-4 attachment 4 with BNC PB-5 pulser 10Hz ADC data items stats - attachment 5 1.8.W spectra - attachments 6 & 7 Observe significant reduction in DSSSD/ribbon cable boundary channel rates - status quo ante Tuesday 4 May https://elog.ph.ed.ac.uk/DESPEC/291 ? 1.8.L spectra - attachments 8 & 9 pulser peak width aida01 85 ch FWHM, aida2 295 ch FWHM 19:21 Alpha run started May21/R6 Statistics dominated by aida04 with a rate of 6k System wide checks all ok except: Base Current Difference aida13 fault 0x0 : 0x17 : 23 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Writing to file at ~300kB/s Rate spectra for aida04 shows dominated by first asic - attachment 10 1*L spectra for aida04 asic 1 - attachment 11 20.25 per FEE64 rate spectra - attachment 12 handful of hot channels aida06, 09 & 10 ADC data item stats - attachment 13
Attachment 1: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(4).png

Attachment 2: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(5).png

Attachment 3: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(6).png

Attachment 4: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(7).png

Attachment 5: Screenshot_2021-05-06_Statistics_aidas-gsi(1).png

Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(8).png

Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(9).png

Attachment 8: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png

Attachment 9: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png

Attachment 10: 210506_1924_aida04Rate.png

Attachment 11: 210506_1927_1*L.png

Attachment 12: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(2).png

Attachment 13: Screenshot_2021-05-06_Statistics_aidas-gsi.png

297	Wed May 5 20:10:16 2021	OH	Wednesday 5 May Alpha run
21:10 After all of the works in S4 today the noise in the system was considerably worse It was decided that before starting a new alpha run a powercycle would be performed. The powercycle has had no effect on the rates. They are still terrible. The difference between yesterday and today is: - Adaptor cards for FEE9 and 13 were re-aligned as they were previously misaligned - bPlast has been plugged in and biased All system wide checks ok Statistics threshold at 0xa - attachment 1 Rate spectra (FEE5 does have events in all channels just noise is dominated by a few) - attachment 2 Waveforms - attachments 3 and 4 Pulser turned off and threshold raised to 0x64 Statistics - attachment 5 21:19 Alpha run started to May21/R4* 08:51 DAQ continues OK - file May21/R4_23 FEE64 temps OK - attachment 8 ADC data items OK - attachment 9 1.8.W spectra - attachments 6 & 7 no change cf. yesterday pm Grafana - most recent 12h DSSSD bias & leakage current - attachment 10 DSSSD biases & leakage currents - attachment 11 Tape Server/Merger - attachment 12 09:44 R4 stopped
Attachment 1: 210504_2105_Stats.png

Attachment 2: 210504_2107_Layout1.png

Attachment 3: 210504_2108_layout7.png

Attachment 4: 210504_2109_Layout8.png

Attachment 5: 210504_2115_stats.png

Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png

Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png

Attachment 8: Screenshot_2021-05-06_Temperature_and_status_scan_aidas-gsi.png

Attachment 9: Screenshot_2021-05-06_Statistics_aidas-gsi.png

Attachment 10: Screenshot_2021-05-06_AIDA_Alerting_-_Grafana.png

Attachment 11: 401.png

Attachment 12: 402.png

296	Wed May 5 16:57:48 2021	TD	Tuesday 5 May
17.55 Grafana DSSSD bias & leakage current - attachment 1 Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37 DSSSD bias & leakage currents - attachment 2
Attachment 1: Screenshot_2021-05-05_AIDA_Alerting_-_Grafana(1).png

Attachment 2: 401.png

295	Tue May 4 21:26:43 2021	OH	Alpha run
22:26 Tape server set up to write to directory May21 DAQ stopped Slow comparator set to 0x64 for all FEE64 and an ASIC check performed Pulser turned off DAQ started ~300k in merger - 38kB per second to file Statistics good (all FEE show 0 bar FEE4 at 3k) - attachment 1 All system wide checks ok apart from FPGA check Base Current Difference aida13 fault 0x0 : 0x62 : 98 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Histograms zeroed. DSSD1 at 130V DSSD2 at 100V 08:47 Found DAQ running at c. 3M data items/s / 13MB/s Switched Tape Server to no storage mode [npg@aidas-gsi ~]$ ls -lrth /TapeData/May21 total 113G -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_0 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_1 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_2 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_3 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_4 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_5 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_6 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_7 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_8 -rw-rw-r--. 1 npg npg 64K May 4 22:24 R1_9 -rw-rw-r--. 1 npg npg 1.3G May 5 06:15 R1_10 -rw-rw-r--. 1 npg npg 2.0G May 5 06:17 R1_11 -rw-rw-r--. 1 npg npg 2.0G May 5 06:20 R1_12 -rw-rw-r--. 1 npg npg 2.0G May 5 06:23 R1_13 -rw-rw-r--. 1 npg npg 2.0G May 5 06:25 R1_14 -rw-rw-r--. 1 npg npg 2.0G May 5 06:28 R1_15 -rw-rw-r--. 1 npg npg 2.0G May 5 06:30 R1_16 -rw-rw-r--. 1 npg npg 2.0G May 5 06:33 R1_17 -rw-rw-r--. 1 npg npg 2.0G May 5 06:36 R1_18 -rw-rw-r--. 1 npg npg 2.0G May 5 06:38 R1_19 -rw-rw-r--. 1 npg npg 2.0G May 5 06:41 R1_20 -rw-rw-r--. 1 npg npg 2.0G May 5 06:43 R1_21 -rw-rw-r--. 1 npg npg 2.0G May 5 06:46 R1_22 -rw-rw-r--. 1 npg npg 2.0G May 5 06:49 R1_23 -rw-rw-r--. 1 npg npg 2.0G May 5 06:51 R1_24 -rw-rw-r--. 1 npg npg 2.0G May 5 06:54 R1_25 -rw-rw-r--. 1 npg npg 2.0G May 5 06:57 R1_26 -rw-rw-r--. 1 npg npg 2.0G May 5 06:59 R1_27 -rw-rw-r--. 1 npg npg 2.0G May 5 07:02 R1_28 -rw-rw-r--. 1 npg npg 2.0G May 5 07:04 R1_29 -rw-rw-r--. 1 npg npg 2.0G May 5 07:07 R1_30 -rw-rw-r--. 1 npg npg 2.0G May 5 07:10 R1_31 -rw-rw-r--. 1 npg npg 2.0G May 5 07:12 R1_32 -rw-rw-r--. 1 npg npg 2.0G May 5 07:15 R1_33 -rw-rw-r--. 1 npg npg 2.0G May 5 07:18 R1_34 -rw-rw-r--. 1 npg npg 2.0G May 5 07:20 R1_35 -rw-rw-r--. 1 npg npg 2.0G May 5 07:23 R1_36 -rw-rw-r--. 1 npg npg 2.0G May 5 07:25 R1_37 -rw-rw-r--. 1 npg npg 2.0G May 5 07:28 R1_38 -rw-rw-r--. 1 npg npg 2.0G May 5 07:31 R1_39 -rw-rw-r--. 1 npg npg 2.0G May 5 07:33 R1_40 -rw-rw-r--. 1 npg npg 2.0G May 5 07:36 R1_41 -rw-rw-r--. 1 npg npg 2.0G May 5 07:39 R1_42 -rw-rw-r--. 1 npg npg 2.0G May 5 07:41 R1_43 -rw-rw-r--. 1 npg npg 2.0G May 5 07:44 R1_44 -rw-rw-r--. 1 npg npg 2.0G May 5 07:46 R1_45 -rw-rw-r--. 1 npg npg 2.0G May 5 07:49 R1_46 -rw-rw-r--. 1 npg npg 2.0G May 5 07:52 R1_47 -rw-rw-r--. 1 npg npg 2.0G May 5 07:54 R1_48 -rw-rw-r--. 1 npg npg 2.0G May 5 07:57 R1_49 -rw-rw-r--. 1 npg npg 2.0G May 5 08:00 R1_50 -rw-rw-r--. 1 npg npg 2.0G May 5 08:02 R1_51 -rw-rw-r--. 1 npg npg 2.0G May 5 08:05 R1_52 -rw-rw-r--. 1 npg npg 2.0G May 5 08:07 R1_53 -rw-rw-r--. 1 npg npg 2.0G May 5 08:10 R1_54 -rw-rw-r--. 1 npg npg 2.0G May 5 08:13 R1_55 -rw-rw-r--. 1 npg npg 2.0G May 5 08:15 R1_56 -rw-rw-r--. 1 npg npg 2.0G May 5 08:18 R1_57 -rw-rw-r--. 1 npg npg 2.0G May 5 08:20 R1_58 -rw-rw-r--. 1 npg npg 2.0G May 5 08:23 R1_59 -rw-rw-r--. 1 npg npg 2.0G May 5 08:26 R1_60 -rw-rw-r--. 1 npg npg 2.0G May 5 08:28 R1_61 -rw-rw-r--. 1 npg npg 2.0G May 5 08:31 R1_62 -rw-rw-r--. 1 npg npg 2.0G May 5 08:34 R1_63 -rw-rw-r--. 1 npg npg 2.0G May 5 08:36 R1_64 -rw-rw-r--. 1 npg npg 2.0G May 5 08:39 R1_65 -rw-rw-r--. 1 npg npg 2.0G May 5 08:41 R1_66 -rw-rw-r--. 1 npg npg 1.6G May 5 08:44 R1_67 [npg@aidas-gsi ~]$ Rate increased c. 06.15 multiple ASICs in multiple FEE64s hot HEC channels ASIC check load restores rates to those at beginning of May21/R1 Switched Tape Server to storage mode - file May21/R1_67 08.53 All system wide checks OK except Base Current Difference aida07 fault 0x0 : 0x1 : 1 aida08 fault 0x0 : 0x1 : 1 aida13 fault 0x62 : 0x9380 : 37662 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Most recent merger errors MERGE Data Link (2497): bad timestamp 7 3 0xd1fc813c 0x073033d6 0x00001083773033d6 0x167c1083773033d6 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1ce7ff9 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1dd806e 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1ed7fc1 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1fd803c 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1cf8066 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1de8110 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1ee8086 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1fe7ff7 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46 MERGE Data Link (2497): bad timestamp 7 3 0xd1c38041 0x0e60a6d6 0x000010832e60a6d6 0x167c10832e60a6d6 0x167c108377304b46 09.04 FEE64 temps OK - attachment 2 ADC data item stats - attachment 3 1.8.L spectra - attachments 4 & 5 1.8.W spectra - attachments 6 & 7 Grafana DSSSD bias & Leakage current most recent 24h - attachment 8 DSSSD bias & leakage current - attacment 9 11:32 R1 stopped to realign boards and check situation in S4
Attachment 1: 210504_2228_Stats.png

Attachment 2: Screenshot_2021-05-05_Temperature_and_status_scan_aidas-gsi.png

Attachment 3: Screenshot_2021-05-05_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(1).png

Attachment 6: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(2).png

Attachment 7: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(3).png

Attachment 8: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(4).png

Attachment 9: Screenshot_2021-05-05_AIDA_Alerting_-_Grafana.png

Attachment 10: 400.png

293	Tue May 4 14:26:50 2021	NH	FEE64 Adapter wiring diagram and layout
A figure showing the FEE64 numbers (a previous diagram had 2/6 and 4/8 on the wrong side of the DSSD) and the wiring of pulser/HV Also the jumpers connected to the boards As always diagram is looking with direction of beam Edit: Figure as signals (HV/Pulser) come from the beam right not beam left Otherwise OK
Attachment 1: AIDA_Adapter_Configuration(1).png

292	Tue May 4 14:05:48 2021	NH	MACB Cables
During the installation of the 4 new HDMI cables for aidas13-16 the HDMI cables between the MACB Root and the MACB leaves were looped through the cable guide at the top to support them more This may improve their connection and reduce the chance of WR errors seen in S460/S452 The connectors were all reseated a few times to clean the contacts too
Attachment 1: macb_front.jpg

Attachment 2: macb_side.jpg

291	Tue May 4 13:53:22 2021	OH, TD	Tuesday 4 May
NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290 14.40 Rate spectra - attachment 1 adc data items - attachment 2 slow comparator 0xa stats for p+n much improved, n+n remain poor 1.8.W spectra - attachments 3 & 4 1.8.L spectra - attachments 5 & 6 pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM 17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7
Attachment 1: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png

Attachment 2: Screenshot_2021-05-04_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png

Attachment 7: 210504_Leakage.png

290	Tue May 4 12:11:39 2021	OH TD	Bias test of triple
13:11 System wide checks all ok except aida02 fails ADC calibration Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1025 Last changed Tue May 04 10:57:38 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Thu Apr 29 14:44:05 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 ASIC clocks have been synchronised and ADC re-calibrated Rate spectra - attachment 1 Layout7 - attachment 2 Layout8 - attachment 3 Detectors at 130V Statistics - attachment 4 V-I Curve - attachment 5 Voltage Ch0 Ch1 10 1.345 1.49 20 2.01 2.285 30 2.405 2.8 40 2.635 3.13 50 2.82 3.37 60 2.95 3.57 70 3.08 3.725 80 3.185 3.865 90 3.26 3.98 100 3.33 4.105 110 3.42 4.22 120 3.525 4.35 130 3.64 4.46 140 3.79 4.37 150 3.98 4.37 150+10min 3.9 4.485
Attachment 1: 210504_1325_Rate.png

Attachment 2: 210504_1326_Layout7.png

Attachment 3: 210504_1327_Layout08.png

Attachment 4: 210504_1334_Stats.png

Attachment 5: VICurve.png

289	Sat May 1 08:45:19 2021	TD	AIDA 24cm x 8cm DSSSD + triPlast assembly for S496
Photos provided by Nic Hubbard and Helena Albers from the assembly of 2x triPlast plus 2x MSL type BB18(DS)-1000 24cm x 8cm detector assembly - attachments 1-3. Spacers installed between MSL type BB18(DS)-1000 24cm x 8cm PCBs - attachment 4. AIDA support stand in 24cm x 8cm DSSSD mode ready for installation of snout assembly - attachments 5 & 6 AIDA 24cm x 8cm DSSSD plus 2x TriPlast installed in S4 - view downstream towards upstream window of AIDA detctor assembly - attachment 7 AIDA 24cm x 8cm DSSSD plus 2x TriPlast snout assembly installed at S4 - attachment 8 Side view of AIDA + triPLast + FATIMA + DEGAS - attachment 9
Attachment 1: MicrosoftTeams-image.png

Attachment 2: IMG_7649.JPG

Attachment 3: IMG_7650.JPG

Attachment 4: MicrosoftTeams-image_(1).png

Attachment 5: image-7becce0a-c5bb-431b-8af9-597e2b8d3829.jpg

Attachment 6: image-3ba4cdee-a8f4-48e4-b087-03ddf07504e5.jpg

Attachment 7: image-beb082d4-a965-453f-8804-394ce65a2f70.jpg

Attachment 8: IMG_7697.JPG

Attachment 9: IMG_7708.JPG

Attachment 10: IMG_7682.JPG

Attachment 11: IMG_7681.JPG

Attachment 12: FATIMA_Triple_AIDA.png

288	Fri Apr 30 08:38:12 2021	TD	Friday 30 April
09.40 All system wide checks OK except FEE64 module aida07 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1025 Last changed Thu Apr 29 17:50:06 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Thu Apr 29 14:44:05 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 09.44 DSSSD bias & leakage currents Ok - attachment 1 Merger time seq errors - attachment 2 no merger errors since restart yesterday FEE64 temps OK - attachment 3 good event stats - attachment 4 Merger/Tape Server server consoles - attachment 5 Grafana DSSSD bias \|& leakage currents for previous 7 days - attachment 6 DAQ continues OK - no data storage
Attachment 1: 310.png

Attachment 2: 311.png

Attachment 3: Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png

Attachment 4: Screenshot_2021-04-30_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png

287	Thu Apr 29 16:56:42 2021	OH	Config changes for 16FEE
New set of ASIC settings produced 2021Apr29-13-16-00 - Note no longer odd/even for polarity of signal New set of Options file for FEEs - Points to the new ASIC settings New layouts for histograms grouped by side of the detector. Heavily weighted to the front side in terms of number of his /MIDAS/Linux/startup/NewMerger Updated line from "./master64 -i 12 -l 16 -p 11001 &" to "./master64 -i 16 -l 16 -p 11001 &" /MIDAS/DB/EXPERIMENTS/MERGE/Options/aidas-gsi/CONTENTS updated the number of links to 16. System wide checks passed except aida03 ADC calibration System left running to no storage.
286	Wed Apr 28 17:39:35 2021	TD	Preparations for setup of aida13-aida16
Updated /etc/dhcpd.conf cd /etc cp dhcpd.conf dhcpd.conf.BAK-280421 Add MAC addresses https://elog.ph.ed.ac.uk/DESPEC/285 aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15 aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed New version of dhcpd.conf (and backup) - attachments 1 & 2 Cannot find dhcpd (and therefore re-read config and restart) in System - Administration -> Services - has the dhcpd service been re-named or amalgamated with another service? Updated /MIDAS/config/TclHttpd/aidas-gsi@8015/startup.tcl New version of startup.tcl (and backup) - attachments 3 & 4 Created rfs filesytems for aida13-aida24 [root@aidas-gsi]# cd /home/Embedded/XilinxLinux/ppc_4xx/rfs [root@aidas-gsi rfs]# cd aida01 [root@aidas-gsi rfs]# tar cvf ../rfs.tar . Create directories aida13-aida24, uid=10101, gid=npg, mode 777 [root@aidas-gsi rfs]# cd aida13; tar xvf ../rfs.tar . : : etc [root@aidas-gsi rfs]# ls -l total 1491880 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida01 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida02 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida03 drwxrwxrwx. 19 10101 npg 4096 Apr 24 03:51 aida04 drwxrwxrwx. 19 10101 npg 4096 Apr 24 03:51 aida05 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida06 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida07 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida08 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:35 aida09 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida10 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:34 aida11 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida12 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida13 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida14 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida15 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida16 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida17 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida18 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida19 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida20 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida21 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida22 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida23 drwxrwxrwx. 19 10101 npg 4096 Apr 28 16:32 aida24 -rw-r--r--. 1 root root 1527582720 Apr 28 18:26 rfs.tar Once dhcpd has been restarted it should be possible to power cycle all FEE64s, start system and update firmware of newly installed FEE64s.
Attachment 1: dhcpd.conf
# # DHCP Server Configuration file. # see /usr/share/doc/dhcp*/dhcpd.conf.sample # # Date of last update Jan 12 2015 # authoritative; ddns-update-style none; ddns-updates off; # 2 days #default-lease-time 172800; # 4 days default-lease-time 345600; # 8 days max-lease-time 691200; option domain-search code 119 = string; option domain-name "dl.ac.uk"; option domain-name-servers 193.62.115.16, 148.79.80.78; option netbios-name-servers 148.79.160.89; option netbios-node-type 8; option nis-domain "nuclear.physics"; option nis-servers 193.62.115.77; subnet 192.168.11.0 netmask 255.255.255.0 { option subnet-mask 255.255.255.0; option broadcast-address 192.168.11.255; pool { range 192.168.11.118 192.168.11.199; } } group { use-host-decl-names true; default-lease-time 3600; max-lease-time 14400; server-name "192.168.11.99"; next-server 192.168.11.99; host nnrpi1 { hardware ethernet b8:27:eb:bb:46:7b; fixed-address 192.168.11.251; } host nnrpi2 { hardware ethernet b8:27:eb:40:53:e8; fixed-address 192.168.11.117; } host aida01 { hardware ethernet d8:80:39:41:ba:8a; fixed-address 192.168.11.1; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01"; } host aida02 { hardware ethernet d8:80:39:41:ba:22; fixed-address 192.168.11.2; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02"; } host aida03 { hardware ethernet d8:80:39:41:d8:21; fixed-address 192.168.11.3; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03"; } host aida04 { hardware ethernet d8:80:39:41:f6:b7; fixed-address 192.168.11.4; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04"; } host aida05 { hardware ethernet d8:80:39:41:d7:cc; fixed-address 192.168.11.5; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05"; } host aida06 { hardware ethernet d8:80:39:41:ee:72; fixed-address 192.168.11.6; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06"; } host aida07 { hardware ethernet d8:80:39:41:b4:0c; fixed-address 192.168.11.7; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07"; } host aida08 { hardware ethernet d8:80:39:41:ba:2b; fixed-address 192.168.11.8; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08"; } host aida09 { hardware ethernet d8:80:39:41:f6:ee; fixed-address 192.168.11.9; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09"; } host aida10 { hardware ethernet d8:80:39:41:ba:89; fixed-address 192.168.11.10; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10"; } host aida11 { hardware ethernet d8:80:39:41:f6:5a; fixed-address 192.168.11.11; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11"; } host aida12 { hardware ethernet d8:80:39:41:cf:ac; fixed-address 192.168.11.12; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12"; } host aida13 { hardware ethernet d8:80:39:42:d:15; fixed-address 192.168.11.13; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13"; } host aida14 { hardware ethernet d8:80:39:42:d:b; fixed-address 192.168.11.14; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14"; } host aida15 { hardware ethernet d8:80:39:41:ee:10; fixed-address 192.168.11.15 option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15"; } host aida16 { hardware ethernet d8:80:39:41:f6:ed; fixed-address 192.168.11.16; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida16"; } } subnet 192.168.12.0 netmask 255.255.255.0 { option subnet-mask 255.255.255.0; option broadcast-address 192.168.12.255; pool { range 192.168.12.100 192.168.12.199; } } group { use-host-decl-names true; default-lease-time 3600; max-lease-time 14400; server-name "192.168.12.99"; next-server 192.168.12.99; host aida21 { hardware ethernet d8:80:39:41:d8:2a; fixed-address 192.168.12.1; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21"; } host aida22 { hardware ethernet 00:04:a3:2a:d0:26; fixed-address 192.168.12.2; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22"; } host aida23 { hardware ethernet 00:04:a3:2b:09:ce; fixed-address 192.168.12.3; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23"; } host aida24 { hardware ethernet 00:04:a3:2b:09:e8; fixed-address 192.168.12.4; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24"; } }
Attachment 2: dhcpd.conf.BAK-280421
# # DHCP Server Configuration file. # see /usr/share/doc/dhcp*/dhcpd.conf.sample # # Date of last update Jan 12 2015 # authoritative; ddns-update-style none; ddns-updates off; # 2 days #default-lease-time 172800; # 4 days default-lease-time 345600; # 8 days max-lease-time 691200; option domain-search code 119 = string; option domain-name "dl.ac.uk"; option domain-name-servers 193.62.115.16, 148.79.80.78; option netbios-name-servers 148.79.160.89; option netbios-node-type 8; option nis-domain "nuclear.physics"; option nis-servers 193.62.115.77; subnet 192.168.11.0 netmask 255.255.255.0 { option subnet-mask 255.255.255.0; option broadcast-address 192.168.11.255; pool { range 192.168.11.118 192.168.11.199; } } group { use-host-decl-names true; default-lease-time 3600; max-lease-time 14400; server-name "192.168.11.99"; next-server 192.168.11.99; host nnrpi1 { hardware ethernet b8:27:eb:bb:46:7b; fixed-address 192.168.11.251; } host nnrpi2 { hardware ethernet b8:27:eb:40:53:e8; fixed-address 192.168.11.117; } host aida01 { hardware ethernet d8:80:39:41:ba:8a; fixed-address 192.168.11.1; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01"; } host aida02 { hardware ethernet d8:80:39:41:ba:22; fixed-address 192.168.11.2; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02"; } host aida03 { hardware ethernet d8:80:39:41:d8:21; fixed-address 192.168.11.3; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03"; } host aida04 { hardware ethernet d8:80:39:41:f6:b7; fixed-address 192.168.11.4; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04"; } host aida05 { hardware ethernet d8:80:39:41:d7:cc; fixed-address 192.168.11.5; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05"; } host aida06 { hardware ethernet d8:80:39:41:ee:72; fixed-address 192.168.11.6; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06"; } host aida07 { hardware ethernet d8:80:39:41:b4:0c; fixed-address 192.168.11.7; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07"; } host aida08 { hardware ethernet d8:80:39:41:ba:2b; fixed-address 192.168.11.8; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08"; } host aida09 { hardware ethernet d8:80:39:41:f6:ee; fixed-address 192.168.11.9; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09"; } host aida10 { hardware ethernet d8:80:39:41:ba:89; fixed-address 192.168.11.10; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10"; } host aida11 { hardware ethernet d8:80:39:41:f6:5a; fixed-address 192.168.11.11; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11"; } host aida12 { hardware ethernet d8:80:39:41:cf:ac; fixed-address 192.168.11.12; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12"; } } subnet 192.168.12.0 netmask 255.255.255.0 { option subnet-mask 255.255.255.0; option broadcast-address 192.168.12.255; pool { range 192.168.12.100 192.168.12.199; } } group { use-host-decl-names true; default-lease-time 3600; max-lease-time 14400; server-name "192.168.12.99"; next-server 192.168.12.99; host aida21 { hardware ethernet d8:80:39:41:d8:2a; fixed-address 192.168.12.1; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21"; } host aida22 { hardware ethernet 00:04:a3:2a:d0:26; fixed-address 192.168.12.2; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22"; } host aida23 { hardware ethernet 00:04:a3:2b:09:ce; fixed-address 192.168.12.3; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23"; } host aida24 { hardware ethernet 00:04:a3:2b:09:e8; fixed-address 192.168.12.4; option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24"; } }
Attachment 3: startup.tcl
Setting BGColor "BlanchedAlmond" Setting TextColor "red" Setting ProjectName AIDA Setting ProjectImage AIDA.gif Setting ProjectURL "http://npg.dl.ac.uk/NPG/Projects/Fair.html" Setting Option1Name STFC Setting Option1URL "stfc.ac.uk" Setting Option1Image stfc.jpg # global MessageLoggerServerHost # set MessageLoggerServerHost nndhcp077.dl.ac.uk # package require MessageLoggerClient # Setting Option3Name "Elog Server" # Setting Option3URL "http://nndhcp077.dl.ac.uk:8080/DataAcq/" # Install required services if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}} # set ::AIDA::TestMode 1 global SpectrumServerHost set SpectrumServerHost aidas package require XAIDAAccessClient ;# for Client end - ie AIDA server workstation sets MIDAS_XAIDAAccessClient_Exists (SOAP) package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation sets MIDAS_XAD9252AccessClient_Exists (SOAP) namespace eval DataAcquisition { variable PROJECT AIDA variable ACQSERVERS set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12 aida13 aida14 aida15 aida16] # set ACQSERVERS [list aida01 aida02 aida03 aida04] # set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12] # set ACQSERVERS [list aida01 aida02 aida03 aida04] variable ACQSERVER [first $ACQSERVERS] variable DataTransferStreams; set DataTransferStreams 2 # set ::DataAcquisition::EnableMerger 1 # set ::DataAcquisition::EnableTape 1 variable RDOGo 0 } namespace eval AIDA { variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS variable TSMaster; set TSMaster aida01 variable FEE64 [first $ModuleMenu] variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd] } Setting DataBaseRoot "/MIDAS/DB" package require DataBaseAccessServer package require NetVarsServer 1.1 package require SigTaskServer # package require MemSasServer # Install services for Run Control namespace eval $::DataAcquisition::PROJECT { variable DBRootName [file join EXPERIMENTS AIDA] variable DBNodeBase 2012 } package require AIDARunControl # package require TapeServer # package require MergerControl # package require MERGERunControl Setting Option7Name "Project Home Page" Setting Option7URL "http://npg.dl.ac.uk/NPG/Projects/Fair.html" Setting Option8Name "Server Software Home Page" Setting Option8URL "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"
Attachment 4: startup.tcl.BAK-280421
Setting BGColor "BlanchedAlmond" Setting TextColor "red" Setting ProjectName AIDA Setting ProjectImage AIDA.gif Setting ProjectURL "http://npg.dl.ac.uk/NPG/Projects/Fair.html" Setting Option1Name STFC Setting Option1URL "stfc.ac.uk" Setting Option1Image stfc.jpg # global MessageLoggerServerHost # set MessageLoggerServerHost nndhcp077.dl.ac.uk # package require MessageLoggerClient # Setting Option3Name "Elog Server" # Setting Option3URL "http://nndhcp077.dl.ac.uk:8080/DataAcq/" # Install required services if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}} # set ::AIDA::TestMode 1 global SpectrumServerHost set SpectrumServerHost aidas package require XAIDAAccessClient ;# for Client end - ie AIDA server workstation sets MIDAS_XAIDAAccessClient_Exists (SOAP) package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation sets MIDAS_XAD9252AccessClient_Exists (SOAP) namespace eval DataAcquisition { variable PROJECT AIDA variable ACQSERVERS # set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12 aida13 aida14 aida15 aida16] # set ACQSERVERS [list aida01 aida02 aida03 aida04] set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12] # set ACQSERVERS [list aida01 aida02 aida03 aida04] variable ACQSERVER [first $ACQSERVERS] variable DataTransferStreams; set DataTransferStreams 2 # set ::DataAcquisition::EnableMerger 1 # set ::DataAcquisition::EnableTape 1 variable RDOGo 0 } namespace eval AIDA { variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS variable TSMaster; set TSMaster aida01 variable FEE64 [first $ModuleMenu] variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd] } Setting DataBaseRoot "/MIDAS/DB" package require DataBaseAccessServer package require NetVarsServer 1.1 package require SigTaskServer # package require MemSasServer # Install services for Run Control namespace eval $::DataAcquisition::PROJECT { variable DBRootName [file join EXPERIMENTS AIDA] variable DBNodeBase 2012 } package require AIDARunControl # package require TapeServer # package require MergerControl # package require MERGERunControl Setting Option7Name "Project Home Page" Setting Option7URL "http://npg.dl.ac.uk/NPG/Projects/Fair.html" Setting Option8Name "Server Software Home Page" Setting Option8URL "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"
285	Wed Apr 28 10:06:11 2021	NH	Wednesday 28 April
11.05 All system wide checks OK except WR Base Current Difference aida01 fault 0xc960 : 0xc971 : 17 aida02 fault 0x8bbf : 0x8bd0 : 17 aida03 fault 0xae9e : 0xaeaf : 17 aida04 fault 0x6e3 : 0x6f4 : 17 aida05 fault 0x31a4 : 0x31bd : 25 aida05 : WR status 0x10 aida06 fault 0x22ea : 0x2303 : 25 aida07 fault 0x438f : 0x43a8 : 25 aida08 fault 0x61ed : 0x6206 : 25 aida09 fault 0x1c32 : 0x1c40 : 14 aida10 fault 0x9332 : 0x933f : 13 aida11 fault 0x227b : 0x2288 : 13 aida12 fault 0xb2fc : 0xb309 : 13 White Rabbit error counter test result: Passed 0, Failed 12 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Base Current Difference aida01 fault 0x0 : 0x1 : 1 aida02 fault 0x0 : 0x3 : 3 aida03 fault 0x0 : 0x1 : 1 aida04 fault 0x0 : 0x2 : 2 aida06 fault 0x0 : 0x2 : 2 aida08 fault 0x0 : 0x1 : 1 aida11 fault 0x0 : 0x3 : 3 aida12 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 4, Failed 8 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Looks like a big WR glitch happened at some point (master in issue?) Memory Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 26 20 26 15 3 1 3 2 2 3 4 : 27848 aida02 : 25 7 1 4 1 4 2 2 3 3 6 : 36204 aida03 : 30 5 3 3 3 3 1 3 3 3 6 : 36464 aida04 : 5 10 0 5 2 3 1 3 3 3 6 : 36356 aida05 : 34 40 16 13 4 3 3 2 3 3 4 : 29160 aida06 : 16 8 4 2 1 4 1 3 3 3 6 : 36416 aida07 : 28 23 23 10 4 3 2 4 3 4 3 : 27736 aida08 : 26 9 3 3 1 3 1 3 3 3 6 : 36352 aida09 : 54 42 10 17 3 4 3 4 3 3 4 : 30376 aida10 : 17 5 7 4 3 3 3 3 2 3 6 : 35996 aida11 : 19 10 11 2 1 1 2 4 2 3 6 : 35916 aida12 : 82 42 12 7 4 2 3 2 2 3 4 : 27960 Options unchanged from ELOG 284 DAQ continues OK but New Merger window has disappeared... clearly still running in background! DSSSD leakage currents decreasing as expected - Plan later today: Prepare AIDA Kapton Cables for triple install 4 new FEE64s into S4 (13,14,15,16) Proposed layout of FEE64s (perspective with beam) in fig 5 Cards 9-12 will be moved on Friday when triple is installed When ready MAC addressed will be given to OH/TD who will do the PC configuration including: - Network IP allocation - MIDAS registration - Firmware updating - Checks 17.20: The four new FEEs have been installed and connected to Ethernet, Power, HDMI and Water No front-end adapter is connected yet The MAC addresss are as follows aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15 aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10 aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed
Attachment 1: elog_temps.png

Attachment 2: elog_gswr.png

Attachment 3: elog_tse.png

Attachment 4: elog_hv.png

Attachment 5: AIDA_FEE.png.jpeg

284	Tue Apr 27 08:26:01 2021	TD	Tuesday 27 April - aida09 system console reports 'received undefined information code: 1' and 'code: 9'
09.25 All system wide checks OK except Base Current Difference aida01 fault 0xc960 : 0xc963 : 3 aida02 fault 0x8bbf : 0x8bc2 : 3 aida03 fault 0xae9e : 0xaea1 : 3 aida04 fault 0x6e3 : 0x6e6 : 3 aida05 fault 0x31a4 : 0x31ac : 8 aida05 : WR status 0x10 aida06 fault 0x22ea : 0x22f2 : 8 aida07 fault 0x438f : 0x4397 : 8 aida08 fault 0x61ed : 0x61f5 : 8 aida09 fault 0x1c32 : 0x1c33 : 1 White Rabbit error counter test result: Passed 3, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 16 18 24 16 3 1 3 2 2 3 4 : 27792 aida02 : 25 6 3 5 1 4 2 2 3 3 6 : 36260 aida03 : 19 10 3 6 3 3 1 3 3 3 6 : 36556 aida04 : 21 7 3 4 2 4 2 2 3 3 6 : 36284 aida05 : 40 37 16 13 4 3 2 2 3 3 4 : 28904 aida06 : 32 4 4 3 0 4 1 3 3 3 6 : 36416 aida07 : 26 22 18 13 3 3 2 4 3 4 3 : 27672 aida08 : 15 9 4 1 2 3 1 3 3 3 6 : 36324 aida09 : 29 41 11 19 3 4 3 4 3 3 4 : 30348 aida10 : 17 5 8 4 3 3 3 3 2 3 6 : 36012 aida11 : 25 9 3 5 1 1 2 4 2 3 6 : 35900 aida12 : 80 43 12 7 4 2 3 2 2 3 4 : 27960 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Apr 24 03:57:22 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021 09.25 DSSSD bias \|& leakage currents Ok - attachment 1 FEE64 temps OK - attachment 2 good event stats - attachment 3 aida09 zero rate merger time seq errors - attachment 4 Grafana DSSSD bias \|& leakage currents for previous 7 days - attachment 5 DAQ continues OK except aida09 - no data storage 10.00 from aida09 system console 24/03:04:39\|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout 24/03:04:39\|Also in RDOGo_Operate: Enabled waveform readout 24/03:04:39\| 24/03:04:39\|get_WAVEBlk (C) : which = 0 , Status = 00001031 , Count = 0 24/03:04:39\|get_WAVEBlk (A) which = 0 ---- status = 0540, State machines = 050A 24/03:04:39\|WV:0:Buffer info. Length = 131072 : Offset = 0 24/03:04:40\|WV:1:Buffer info. Length = 131070 : Offset = 2 24/03:04:40\|WV:2:Buffer info. Length = 131068 : Offset = 4 24/03:04:40\|WV:3:Buffer info. Length = 131066 : Offset = 6 24/03:04:40\|WV:4:Buffer info. Length = 131064 : Offset = 8 24/03:04:40\|WV:5:Buffer info. Length = 131062 : Offset = 10 24/03:04:40\|WV:6:Buffer info. Length = 131060 : Offset = 12 24/03:04:40\|WV:7:Buffer info. Length = 131058 : Offset = 14 24/03:04:40\|WV:8:Buffer info. Length = 131056 : Offset = 16 24/03:04:40\|WV:9:Buffer info. Length = 131054 : Offset = 18 24/03:04:40\|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908 24/03:04:40\|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 804441A5 : 1 => 09CD0000 24/03:04:40\|Last Data : Databuffer 274139128 : 0 => C01B7F88 : 1 => 0C05B135 24/03:04:40\| 1FA61FB2 : 1F951F66 \| 1F361F5D : 1F751F0F \| 1F191EF0 : 1EF31EE9 \| 1E701E8E : 1EEB1F51 24/03:04:40\| 1F521F99 : 1FF52005 \| 1FE52027 : 20171FCC \| 1FAD1F3D : 1ED41E7B \| 1EB11E77 : 3FAB373A 24/03:04:40\| 8050023F : 0AE0C000 \| 804441A5 : 0AE0C000 \| 400003FC : 0AE0C4D0 \| 1E581F6F : 1E451D26 24/03:04:41\| 1FB01E60 : 1D2E1F80 \| 1E251D88 : 1FA81F4A \| 1DD51F1F : 1E1D1DF7 \| 1EBC1DF0 : 1DF41E7E 24/03:04:41\| 1DEA1DF5 : 1E9E1E83 \| 1E231E9C : 1E661E60 \| 1EA41E6F : 1E101D9C \| 1D751D92 : 1DFC1DF2 24/03:04:41\| 1E201E47 : 1E7C1E6C \| 1E351E00 : 1DD91DFE \| 1DD81DB5 : 1DE91E1C \| 1E261DD6 : 1E1A1E0B 24/03:04:41\| 1DFB1DCC : 1DEA1E11 \| 1E201E39 : 1E351E7A \| 1E7B1DBE : 1E7A1E81 \| 1E261DFC : 1DEA1DAC 24/03:04:41\| 1D831DAC : 1DF51E1F \| 1E4E1E48 : 1E421E50 \| 1E861DDB : 1E361E18 \| 1D551DB9 : 1DC51D8E 24/03:04:41\|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C03A7F13 : 1 => 0C05B455 24/03:04:41\|Last Data : Databuffer 274139128 : 0 => C0127D9D : 1 => 0DB9DEED 24/03:04:41\| 1FA61FB2 : 1F951F66 \| 1F361F5D : 1F751F0F \| 1F191EF0 : 1EF31EE9 \| 1E701E8E : 1EEB1F51 24/03:04:41\| 1F521F99 : 1FF52005 \| 1FE52027 : 20171FCC \| 1FAD1F3D : 1ED41E7B \| 1EB11E77 : 3FAB373A 24/03:04:41\| 8050023F : 0AE0C000 \| 804441A5 : 0AE0C000 \| 400003FC : 0AE0C4D0 \| 1E581F6F : 1E451D26 24/03:04:42\| 1FB01E60 : 1D2E1F80 \| 1E251D88 : 1FA81F4A \| 1DD51F1F : 1E1D1DF7 \| 1EBC1DF0 : 1DF41E7E 24/03:04:42\| 1DEA1DF5 : 1E9E1E83 \| 1E231E9C : 1E661E60 \| 1EA41E6F : 1E101D9C \| 1D751D92 : 1DFC1DF2 24/03:04:42\| 1E201E47 : 1E7C1E6C \| 1E351E00 : 1DD91DFE \| 1DD81DB5 : 1DE91E1C \| 1E261DD6 : 1E1A1E0B 24/03:04:42\| 1DFB1DCC : 1DEA1E11 \| 1E201E39 : 1E351E7A \| 1E7B1DBE : 1E7A1E81 \| 1E261DFC : 1DEA1DAC 24/03:04:42\| 1D831DAC : 1DF51E1F \| 1E4E1E48 : 1E421E50 \| 1E861DDB : 1E361E18 \| 1D551DB9 : 1DC51D8E 24/03:04:42\|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908 24/03:04:42\|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C02D7E26 : 1 => 0DB9DEED 24/03:04:42\|Last Data : Databuffer 274139128 : 0 => C0217F73 : 1 => 0F816B3D 24/03:04:42\| 1EFB1F2D : 1F311F1E \| 1F3B1F1C : 1EDD1EC4 \| 1ED71EE6 : 1F301F09 \| 1F4D1FAA : 1F1E1F81 24/03:04:42\| 1F491F0C : 1F481F31 \| 1F6C1F7E : 1F601F8E \| 1FC71FA9 : 1F451F9F \| 1F8C1F1D : 1F551FA9 24/03:04:42\| 1F671E98 : 1F421F47 \| 1ED91F67 : 1F911F70 \| 1DFD1F98 : 224A1D5B \| 1CF821BC : 1F771E94 24/03:04:43\| 213F1EE7 : 1DBC20F2 \| 1F961D9F : 1FC81FE3 \| 1EA0206E : 1F9D1DD9 \| 20042040 : 1E6A1ECA 24/03:04:43\| 1F7E1FD0 : 1F981EC7 \| 1E7A1F1B : 1F5F1F27 \| 1F121F10 : 1F531F92 \| 1F5B1EF8 : 1FA12003 24/03:04:43\| 1F7A1F97 : 1F561F26 \| 200C1F80 : 1FC3201A \| 1FA81FA4 : 1FEB1F99 \| 1F5C1F9D : 1F8F1F98 24/03:04:43\| 1FCC1FC5 : 1FB01F92 \| 1FA21F6D : 1F261F0A \| 1F141F1C : 1F461F24 \| 1EEE1FDB : 20221F2F 24/03:04:43\| 1F741FF1 : 1F891F8A \| 1FC71F5E : 1F961FF2 \| 1F6B1F6C : 1FB11F74 \| 1F821FA8 : 1F841F54 24/03:04:43\|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C02D7EB4 : 1 => 0F816D95 24/03:04:43\|Last Data : Databuffer 274139128 : 0 => C0167F53 : 1 => 0148F2DD 24/03:04:43\| 1EFB1F2D : 1F311F1E \| 1F3B1F1C : 1EDD1EC4 \| 1ED71EE6 : 1F301F09 \| 1F4D1FAA : 1F1E1F81 24/03:04:43\| 1F491F0C : 1F481F31 \| 1F6C1F7E : 1F601F8E \| 1FC71FA9 : 1F451F9F \| 1F8C1F1D : 1F551FA9 24/03:04:43\| 1F671E98 : 1F421F47 \| 1ED91F67 : 1F911F70 \| 1DFD1F98 : 224A1D5B \| 1CF821BC : 1F771E94 24/03:04:44\| 213F1EE7 : 1DBC20F2 \| 1F961D9F : 1FC81FE3 \| 1EA0206E : 1F9D1DD9 \| 20042040 : 1E6A1ECA 24/03:04:44\| 1F7E1FD0 : 1F981EC7 \| 1E7A1F1B : 1F5F1F27 \| 1F121F10 : 1F531F92 \| 1F5B1EF8 : 1FA12003 24/03:04:44\| 1F7A1F97 : 1F561F26 \| 200C1F80 : 1FC3201A \| 1FA81FA4 : 1FEB1F99 \| 1F5C1F9D : 1F8F1F98 24/03:04:44\| 1FCC1FC5 : 1FB01F92 \| 1FA21F6D : 1F261F0A \| 1F141F1C : 1F461F24 \| 1EEE1FDB : 20221F2F 24/03:04:44\| 1F741FF1 : 1F891F8A \| 1FC71F5E : 1F961FF2 \| 1F6B1F6C : 1FB11F74 \| 1F821FA8 : 1F841F54 24/03:04:44\|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0207E17 : 1 => 0148F2DD 24/03:04:44\|Last Data : Databuffer 274139128 : 0 => C0177F46 : 1 => 047FF0CD 24/03:04:44\| 1EFB1F2D : 1F311F1E \| 1F3B1F1C : 1EDD1EC4 \| 1ED71EE6 : 1F301F09 \| 1F4D1FAA : 1F1E1F81 24/03:04:44\| 1F491F0C : 1F481F31 \| 1F6C1F7E : 1F601F8E \| 1FC71FA9 : 1F451F9F \| 1F8C1F1D : 1F551FA9 24/03:04:44\| 1F671E98 : 1F421F47 \| 1ED91F67 : 1F911F70 \| 1DFD1F98 : 224A1D5B \| 1CF821BC : 1F771E94 24/03:04:44\| 213F1EE7 : 1DBC20F2 \| 1F961D9F : 1FC81FE3 \| 1EA0206E : 1F9D1DD9 \| 20042040 : 1E6A1ECA 24/03:04:45\| 1F7E1FD0 : 1F981EC7 \| 1E7A1F1B : 1F5F1F27 \| 1F121F10 : 1F531F92 \| 1F5B1EF8 : 1FA12003 24/03:04:45\| 1F7A1F97 : 1F561F26 \| 200C1F80 : 1FC3201A \| 1FA81FA4 : 1FEB1F99 \| 1F5C1F9D : 1F8F1F98 24/03:04:45\| 1FCC1FC5 : 1FB01F92 \| 1FA21F6D : 1F261F0A \| 1F141F1C : 1F461F24 \| 1EEE1FDB : 20221F2F 24/03:04:45\| 1F741FF1 : 1F891F8A \| 1FC71F5E : 1F961FF2 \| 1F6B1F6C : 1FB11F74 \| 1F821FA8 : 1F841F54 24/03:04:45\|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908 24/03:04:45\|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908 24/03:04:45\|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908 24/03:04:47\|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908 24/03:04:56\|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908 24/03:05:05\|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908 24/03:05:15\|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908 24/03:05:24\|197 spectra zeroed 25/10:21:10\|do_GetState returned z=0 and 1 25/10:21:11\|Data Acquisition Statistics counters now cleared 25/10:24:01\|Clear Statistics (1) 25/10:24:01\|bad event data (0x45000034 0x84e64000) 26/16:52:50\|bad event data (0x40061e21 0xc0a80b63) 26/16:52:50\|received undefined information code: 1 26/16:52:50\|bad event data (0x0101080a 0xe9b68def) 26/16:52:50\|bad event data (0x03527827 0x0101050a) 26/16:52:50\|received undefined information code: 9 From system console Apr 24 02:52:42 aida09 kernel: IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9 Apr 24 02:52:42 aida09 kernel: IP-Config: Complete: Apr 24 02:52:42 aida09 kernel: device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255, Apr 24 02:52:42 aida09 kernel: host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics, Apr 24 02:52:42 aida09 kernel: bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100003/2 on 192.168.11.99 Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100005/1 on 192.168.11.99 Apr 24 02:52:43 aida09 kernel: VFS: Mounted root (nfs filesystem) on device 0:12. Apr 24 02:52:43 aida09 kernel: Freeing unused kernel memory: 168k init Apr 24 02:52:43 aida09 kernel: xaida: device parameters: base=0x81000000 size=0x200000 Apr 24 02:52:44 aida09 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff> Apr 24 02:52:44 aida09 kernel: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000 Apr 24 02:52:44 aida09 kernel: xaida: driver assigned major number 254 Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service telnet, trying IPv4 Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service ftp, trying IPv4 Apr 24 02:52:44 aida09 xinetd[273]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in. Apr 24 02:52:44 aida09 xinetd[273]: Started working: 2 available services Apr 24 02:52:48 aida09 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff> Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000 Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: driver assigned major number 253 Apr 24 02:53:04 aida09 kernel: xaida: open: Apr 24 02:53:04 aida09 kernel: AIDAMEM: aidamem_open: Apr 24 02:57:33 aida09 kernel: xaida: open: Apr 24 13:39:36 aida09 xinetd[273]: START: telnet pid=396 from=192.168.11.99 Apr 24 13:42:05 aida09 xinetd[273]: EXIT: telnet status=0 pid=396 duration=149(sec) Apr 27 09:01:50 aida09 xinetd[273]: START: telnet pid=422 from=192.168.11.99
Attachment 1: 303.png

Attachment 2: Screenshot_2021-04-27_Temperature_and_status_scan_aidas-gsi.png

Attachment 3: Screenshot_2021-04-27_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-04-27_GSI_White_Rabbit_Timestamp_Control_aidas-gsi.png

Attachment 5: Screenshot_2021-04-27_NewMerger_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-04-27_AIDA_Alerting_-_Grafana.png

283	Mon Apr 26 09:41:21 2021	TD	Monday 26 April
10.42 All system wide checks OK except Base Current Difference aida01 fault 0xc960 : 0xc962 : 2 aida02 fault 0x8bbf : 0x8bc1 : 2 aida03 fault 0xae9e : 0xaea0 : 2 aida04 fault 0x6e3 : 0x6e5 : 2 aida05 fault 0x31a4 : 0x31ab : 7 aida05 : WR status 0x10 aida06 fault 0x22ea : 0x22f1 : 7 aida07 fault 0x438f : 0x4396 : 7 aida08 fault 0x61ed : 0x61f4 : 7 aida09 fault 0x1c32 : 0x1c33 : 1 White Rabbit error counter test result: Passed 3, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 25 8 28 8 3 2 2 3 2 3 4 : 27940 aida02 : 24 9 3 5 1 3 1 3 3 3 6 : 36408 aida03 : 24 6 5 4 3 3 1 3 3 3 6 : 36512 aida04 : 23 6 6 1 3 3 1 3 3 3 6 : 36428 aida05 : 15 21 27 7 4 2 2 3 3 3 4 : 29044 aida06 : 24 8 2 4 1 3 1 3 3 3 6 : 36352 aida07 : 29 24 28 9 3 3 2 2 2 3 4 : 27732 aida08 : 17 10 1 1 1 4 1 3 3 3 6 : 36356 aida09 : 41 49 18 14 3 4 3 4 2 3 4 : 29388 aida10 : 3 1 6 4 2 3 3 3 2 3 6 : 35828 aida11 : 22 7 5 4 2 1 2 4 2 3 6 : 35936 aida12 : 65 30 23 9 3 1 2 3 2 3 4 : 28100 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Apr 24 03:57:22 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021 10.47 DSSSD bias \|& leakage currents Ok - attachment 1 FEE64 temps OK - attachment 2 good event stats - attachment 3 merger time seq errors - attachment 4 Grafana DSSSD bias \|& leakage currents for previous 7 days - attachment 5 DAQ continues OK - no data storage
Attachment 1: 302.png

Attachment 2: Screenshot_2021-04-26_Temperature_and_status_scan_aidas-gsi.png

Attachment 3: Screenshot_2021-04-26_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-04-26_NewMerger_Statistics_aidas-gsi.png

Attachment 5: Screenshot_2021-04-26_AIDA_Alerting_-_Grafana.png

Goto page Previous 1, 2, 3 ... 20, 21, 22 ... 33, 34, 35 Next

ELOG V3.1.4-unknown