ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 7 of 35

Full | Summary | Threaded | Hide attachments

687 Entries

Goto page Previous 1, 2, 3 ... 6, 7, 8 ... 33, 34, 35 Next

ID	Date	Author	Subject
464	Tue May 17 06:58:51 2022	OH	Tuesday 17th May
08:00 Still have beam for the time being. Currently on file R4_2049 FEE Stats - attachment 1 FEE Temps ok - attachment 2 Bias and leakage currents - attachment 3 System wide checks all ok except aida09 fails clock check with value 6 which is ok Plan for the morning is once beam is lost is Ge group will enter and place source on the snout to perform efficiency measurement. Once they have left the area and are taking data a pulser walkthrough will be performed. Patrick can then have access to the system 08:16 Beam taken away Data taken stopped on file R4_2055 Will perform a pulser walkthrough now Pulser walkthrough started on R5 1V->0.1V in 0.1V steps Pulser setting - attachment 4 p+n pulser peaks - attachment 5 n+n pulser "peaks" - attachment 6
Attachment 1: Screenshot_from_2022-05-17_08-00-50.png

Attachment 2: Screenshot_from_2022-05-17_08-01-13.png

Attachment 3: Screenshot_from_2022-05-17_08-01-25.png

Attachment 4: Screenshot_from_2022-05-17_08-49-02.png

Attachment 5: Screenshot_from_2022-05-17_08-51-19.png

Attachment 6: Screenshot_from_2022-05-17_08-51-52.png

465	Tue May 17 16:53:20 2022	TD	Tuesday 17 May 16.00-00.00
17.35 zero all histograms zero statistics zero NewMerger statistics ASIC check control - all FEE64s, all ASICs file no storage mode, no MBS data transfer ADC control register 0xff all disc outputs disabled ASIC settings 2021Apr29-13-16-00 slow comparator 0x80 BNC PB-5 settings amplitude 1.000V attenuation x1 tau_d 1ms frequency 2Hz polarity + Attachment 1 - grafana DSSSD bias, leakage current & temp - OK Attachments 2-10 - NewMerger stats Attachments 11-13 - iptraf, TapeServer, NewMerger Attachment 14 - per Fee64 rate spectra Attachment 15 - lost activity monitor Attachment 16 - ASIC settings aida01 Attachments 17-22 - adc, pause, resume & correlation scaler data items, push, flush, aida01 & aida06 stats Attachment 23 - FEE64 temperatures OK Attachment 24 - DSSSD bias & leakage current
Attachment 1: Screenshot_from_2022-05-17_17-52-15.png

Attachment 2: Screenshot_from_2022-05-17_17-50-50.png

Attachment 3: Screenshot_from_2022-05-17_17-49-18.png

Attachment 4: Screenshot_from_2022-05-17_17-48-24.png

Attachment 5: Screenshot_from_2022-05-17_17-47-35.png

Attachment 6: Screenshot_from_2022-05-17_17-47-12.png

Attachment 7: Screenshot_from_2022-05-17_17-46-19.png

Attachment 8: Screenshot_from_2022-05-17_17-45-34.png

Attachment 9: Screenshot_from_2022-05-17_17-45-10.png

Attachment 10: Screenshot_from_2022-05-17_17-44-34.png

Attachment 11: Screenshot_from_2022-05-17_17-44-03.png

Attachment 12: Screenshot_from_2022-05-17_17-43-37.png

Attachment 13: Screenshot_from_2022-05-17_17-42-58.png

Attachment 14: Screenshot_from_2022-05-17_17-42-40.png

Attachment 15: Screenshot_from_2022-05-17_17-40-53.png

Attachment 16: Screenshot_from_2022-05-17_17-40-08.png

Attachment 17: Screenshot_from_2022-05-17_17-39-20.png

Attachment 18: Screenshot_from_2022-05-17_17-38-52.png

Attachment 19: Screenshot_from_2022-05-17_17-38-20.png

Attachment 20: Screenshot_from_2022-05-17_17-37-47.png

Attachment 21: Screenshot_from_2022-05-17_17-37-19.png

Attachment 22: Screenshot_from_2022-05-17_17-36-50.png

Attachment 23: Screenshot_from_2022-05-17_17-36-14.png

Attachment 24: Screenshot_from_2022-05-17_17-35-42.png

202	Tue Mar 16 09:26:47 2021	TD	Tuesday 16 March
10.25 DAQ continues OK - no storage ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 10.26 System wide checks FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida07 failed FEE64 module aida09 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x6acb : 0x6acc : 1 aida06 fault 0x8394 : 0x8395 : 1 aida07 fault 0x8c77 : 0x8c7a : 3 aida08 fault 0x1692 : 0x1693 : 1 White Rabbit error counter test result: Passed 8, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Timestamp error counter test result: Passed 12, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 19 5 5 1 3 2 2 3 4 10 : 54660 aida02 : 20 13 14 11 5 5 2 4 2 3 6 : 37048 aida03 : 2 3 7 3 2 3 2 4 2 3 6 : 36080 aida04 : 14 10 4 3 3 2 2 4 1 4 15 : 73960 aida05 : 97 51 19 3 3 34 21 20 6 4 7 : 64364 aida06 : 24 7 5 9 0 1 1 3 4 2 5 : 31112 aida07 : 21 13 9 3 3 2 3 4 3 3 6 : 37484 aida08 : 38 10 3 5 2 3 1 4 3 3 6 : 37048 aida09 : 24 11 4 3 2 3 2 3 3 3 6 : 36696 aida10 : 26 10 1 3 1 2 1 4 3 3 6 : 36712 aida11 : 0 7 7 1 1 4 2 3 2 3 6 : 35592 aida12 : 6 8 3 2 2 1 2 3 3 3 6 : 36296 FEE64 Temperatures OK - attachment 1 Good event statistics OK - attachment 2 Detector bias & leakage currents OK - attachment 3 Merger OK - 4.5M data items/s TapeServer OK - 13Mb/s No merger server error messages since most recent check yesterday 15.50 Merger error messages : : MERGE Data Link (20503): bad timestamp 3 3 0x838153cb 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0x83820000 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0d481de 0x090bff32 0x0000c2c4f90bff32 0x166cc2c4f90bff32 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0c181f8 0x090c0ed2 0x0000c2c4f90c0ed2 0x166cc2c4f90c0ed2 0x166cc2c4fefc5fd2 MERGE Actor (20516): Working with 0 from 12 data sources MERGE Data Link (20503): bad timestamp 3 3 0xc0c47f9e 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0d08127 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0c681ee 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0d181c9 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0c98204 0x090c2642 0x0000c2c4f90c2642 0x166cc2c4f90c2642 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0cc8075 0x090c2e12 0x0000c2c4f90c2e12 0x166cc2c4f90c2e12 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0cf812f 0x090c35e2 0x0000c2c4f90c35e2 0x166cc2c4f90c35e2 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0c780ef 0x090c4d52 0x0000c2c4f90c4d52 0x166cc2c4f90c4d52 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0f7816c 0x090c8bd2 0x0000c2c4f90c8bd2 0x166cc2c4f90c8bd2 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0d480f1 0x090c93a2 0x0000c2c4f90c93a2 0x166cc2c4f90c93a2 0x166cc2c4fefc5fd2 MERGE Data Link (20503): bad timestamp 3 3 0xc0c48039 0x090ca342 0x0000c2c4f90ca342 0x166cc2c4f90ca342 0x166cc2c4fefc5fd2 : : MERGE Data Link (20502): bad timestamp 2 3 0xc08d7cc1 0x085c7e10 0x0000c429685c7e10 0x166cc429685c7e10 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc08e281c 0x085c85e0 0x0000c429685c85e0 0x166cc429685c85e0 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc08f7cc1 0x085c8db0 0x0000c429685c8db0 0x166cc429685c8db0 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0927e8c 0x085c9580 0x0000c429685c9580 0x166cc429685c9580 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc08c414b 0x085ca520 0x0000c429685ca520 0x166cc429685ca520 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0857dee 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0987d2c 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07d90 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0877fd6 0x085d02e0 0x0000c429685d02e0 0x166cc429685d02e0 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0927e7c 0x085d2220 0x0000c429685d2220 0x166cc429685d2220 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0857d7f 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0987cc0 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07ccd 0x085d5100 0x0000c429685d5100 0x166cc429685d5100 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc08e80fd 0x085d7fe0 0x0000c429685d7fe0 0x166cc429685d7fe0 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0977e77 0x085d8f80 0x0000c429685d8f80 0x166cc429685d8f80 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0817ec8 0x085d9750 0x0000c429685d9750 0x166cc429685d9750 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0847fca 0x085d9f20 0x0000c429685d9f20 0x166cc429685d9f20 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc08c40e8 0x085da6f0 0x0000c429685da6f0 0x166cc429685da6f0 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0927f3e 0x085db690 0x0000c429685db690 0x166cc429685db690 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0x82614000 0x085dbb36 0x0000c429685dbb36 0x166cc429685dbb36 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0857ebc 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380 MERGE Data Link (20502): bad timestamp 2 3 0xc0987eec 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380 21.04 Merger errors since 15.50 MERGE Data Link (20502): bad timestamp 2 3 0xc0907de6 0x03087130 0x0000ea6805fd8d80 0x166cea6803087130 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0878007 0x030880d0 0x0000ea68030880d0 0x166cea68030880d0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0937ec9 0x030888a0 0x0000ea68030888a0 0x166cea68030888a0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4e2d 0x0308a7e0 0x0000ea680308a7e0 0x166cea680308a7e0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0998001 0x0308bf50 0x0000ea680308bf50 0x166cea680308bf50 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4abc 0x0308fdd0 0x0000ea680308fdd0 0x166cea680308fdd0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0907fb1 0x030905a0 0x0000ea68030905a0 0x166cea68030905a0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0937fd8 0x03091d10 0x0000ea6803091d10 0x166cea6803091d10 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0807df1 0x03093480 0x0000ea6803093480 0x166cea6803093480 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4d14 0x03093c50 0x0000ea6803093c50 0x166cea6803093c50 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0x8280dbb0 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0x82816a12 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0x82820000 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0837e07 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0967c2a 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0887d0f 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0997dad 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc08f7bfc 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc09a7c98 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc09d7f7e 0x03096b30 0x0000ea6803096b30 0x166cea6803096b30 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4e10 0x03098a70 0x0000ea6803098a70 0x166cea6803098a70 0x166cea6805fd8d80 MERGE Data Link (20502): bad timestamp 2 3 0xc0957f91 0x048638a0 0x0000ea6bf48638a0 0x166cea6bf48638a0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0967e3d 0x04864070 0x0000ea6bf4864070 0x166cea6bf4864070 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4dd3 0x04865fb0 0x0000ea6bf4865fb0 0x166cea6bf4865fb0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0998027 0x04866780 0x0000ea6bf4866780 0x166cea6bf4866780 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08d7dd9 0x0486a600 0x0000ea6bf486a600 0x166cea6bf486a600 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08e4d45 0x0486add0 0x0000ea6bf486add0 0x166cea6bf486add0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08f7e98 0x0486b5a0 0x0000ea6bf486b5a0 0x166cea6bf486b5a0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0967d19 0x0486cd10 0x0000ea6bf486cd10 0x166cea6bf486cd10 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4e95 0x0486ec50 0x0000ea6bf486ec50 0x166cea6bf486ec50 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0997dba 0x0486f420 0x0000ea6bf486f420 0x166cea6bf486f420 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0x8280cc32 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0x82816a15 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0x82820000 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08e4c2e 0x04874240 0x0000ea6bf4874240 0x166cea6bf4874240 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08f7ea0 0x04874a10 0x0000ea6bf4874a10 0x166cea6bf4874a10 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4b7e 0x048780c0 0x0000ea6bf48780c0 0x166cea6bf48780c0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0857dd8 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0947ed6 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08d7f9f 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0967f8e 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0a07d73 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0977d81 0x0487b770 0x0000ea6bf487b770 0x166cea6bf487b770 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc09b7d98 0x0487bf40 0x0000ea6bf487bf40 0x166cea6bf487bf40 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc09d803b 0x0487c710 0x0000ea6bf487c710 0x166cea6bf487c710 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08e4ca6 0x0487d6b0 0x0000ea6bf487d6b0 0x166cea6bf487d6b0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0x8280cc32 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0x82816a15 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0x82820000 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0b27dba 0x04880d60 0x0000ea6bf4880d60 0x166cea6bf4880d60 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4996 0x04881d00 0x0000ea6bf4881d00 0x166cea6bf4881d00 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0857f80 0x04883470 0x0000ea6bf4883470 0x166cea6bf4883470 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0947f50 0x04883c40 0x0000ea6bf4883c40 0x166cea6bf4883c40 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0a07e3b 0x04884410 0x0000ea6bf4884410 0x166cea6bf4884410 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0977f31 0x048853b0 0x0000ea6bf48853b0 0x166cea6bf48853b0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc09b7db2 0x04885b80 0x0000ea6bf4885b80 0x166cea6bf4885b80 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08e4951 0x04886b20 0x0000ea6bf4886b20 0x166cea6bf4886b20 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0b27e54 0x0488a1d0 0x0000ea6bf488a1d0 0x166cea6bf488a1d0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4bc5 0x0488b170 0x0000ea6bf488b170 0x166cea6bf488b170 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0947d43 0x0488d0b0 0x0000ea6bf488d0b0 0x166cea6bf488d0b0 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0877f7a 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0977e13 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08e4985 0x0488ff90 0x0000ea6bf488ff90 0x166cea6bf488ff90 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08b7f0e 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc0b27dc7 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4db1 0x04893e10 0x0000ea6bf4893e10 0x166cea6bf4893e10 0x166cea6bfa7c5dac MERGE Data Link (20502): bad timestamp 2 3 0xc08c4ea2 0x07d8b160 0x0000ea6d67d8b160 0x166cea6d67d8b160 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08d7eda 0x07d8b930 0x0000ea6d67d8b930 0x166cea6d67d8b930 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc09b7c0e 0x07d8d0a0 0x0000ea6d67d8d0a0 0x166cea6d67d8d0a0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4bdb 0x07d8efe0 0x0000ea6d67d8efe0 0x166cea6d67d8efe0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0947de3 0x07d90750 0x0000ea6d67d90750 0x166cea6d67d90750 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0877fed 0x07d945d0 0x0000ea6d67d945d0 0x166cea6d67d945d0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4f9a 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07f5e 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08d7e2b 0x07d95570 0x0000ea6d67d95570 0x166cea6d67d95570 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4c6a 0x07d98c20 0x0000ea6d67d98c20 0x166cea6d67d98c20 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0947f27 0x07d99bc0 0x0000ea6d67d99bc0 0x166cea6d67d99bc0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc09a7d09 0x07d9a390 0x0000ea6d67d9a390 0x166cea6d67d9a390 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0937d1a 0x07d9bb00 0x0000ea6d67d9bb00 0x166cea6d67d9bb00 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0x8280e345 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0x82816a16 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0x82820000 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0878187 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07ce8 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08c38a0 0x07d9e9e0 0x0000ea6d67d9e9e0 0x166cea6d67d9e9e0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4b25 0x07da2090 0x0000ea6d67da2090 0x166cea6d67da2090 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc09a7c55 0x07da3030 0x0000ea6d67da3030 0x166cea6d67da3030 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0937e6c 0x07da47a0 0x0000ea6d67da47a0 0x166cea6d67da47a0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08c36aa 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07c8e 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0837f38 0x07da9d90 0x0000ea6d67da9d90 0x166cea6d67da9d90 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0917d84 0x07daad30 0x0000ea6d67daad30 0x166cea6d67daad30 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0947f39 0x07dab500 0x0000ea6d67dab500 0x166cea6d67dab500 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0977d15 0x07dabcd0 0x0000ea6d67dabcd0 0x166cea6d67dabcd0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4d6c 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc09a7f77 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0x8280e346 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0x82816a16 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0x82820000 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0937f84 0x07dadc10 0x0000ea6d67dadc10 0x166cea6d67dadc10 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4bb4 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07cee 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0837e94 0x07db3200 0x0000ea6d67db3200 0x166cea6d67db3200 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4d16 0x07db5910 0x0000ea6d67db5910 0x166cea6d67db5910 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0937dd1 0x07db7080 0x0000ea6d67db7080 0x166cea6d67db7080 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc09a7e42 0x07db87f0 0x0000ea6d67db87f0 0x166cea6d67db87f0 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4ed0 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0a07e15 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0977f69 0x07dbaf00 0x0000ea6d67dbaf00 0x166cea6d67dbaf00 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0837e60 0x07dbc670 0x0000ea6d67dbc670 0x166cea6d67dbc670 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4bfb 0x07dbed80 0x0000ea6d67dbed80 0x166cea6d67dbed80 0x166cea6d6938be60 MERGE Data Link (20502): bad timestamp 2 3 0xc0837eb4 0x055a8c50 0x0000ea6d79a8c844 0x166cea6d755a8c50 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc09c7dba 0x055a8c50 0x0000ea6d755a8c50 0x166cea6d755a8c50 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4e6d 0x055a9420 0x0000ea6d755a9420 0x166cea6d755a9420 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc0907bef 0x055aa3c0 0x0000ea6d755aa3c0 0x166cea6d755aa3c0 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc08b7ed5 0x055acad0 0x0000ea6d755acad0 0x166cea6d755acad0 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc0887eb6 0x055ae240 0x0000ea6d755ae240 0x166cea6d755ae240 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc08e4da2 0x055aea10 0x0000ea6d755aea10 0x166cea6d755aea10 0x166cea6d79a8c844 MERGE Data Link (20502): bad timestamp 2 3 0xc08f7ff1 0x055b0180 0x0000ea6d755b0180 0x166cea6d755b0180 0x166cea6d79a8c844 : : MERGE Data Link (20503): bad timestamp 3 3 0xc0da81a0 0x0bb28852 0x0000ee992bb28852 0x166cee992bb28852 0x166cee992d6fc8b2 MERGE Data Link (20503): bad timestamp 3 3 0xc0f4811f 0x0bb29022 0x0000ee992bb29022 0x166cee992bb29022 0x166cee992d6fc8b2 MERGE Data Link (20503): bad timestamp 3 3 0xc0cc805b 0x0bb297f2 0x0000ee992bb297f2 0x166cee992bb297f2 0x166cee992d6fc8b2 MERGE Data Link (20503): bad timestamp 3 3 0xc0d38174 0x0bb2af62 0x0000ee992bb2af62 0x166cee992bb2af62 0x166cee992d6fc8b2 MERGE Data Link (20503): bad timestamp 3 3 0xc0c38233 0x0bb2ede2 0x0000ee992bb2ede2 0x166cee992bb2ede2 0x166cee992d6fc8b2 MERGE Data Link (20503): bad timestamp 3 3 0xc0f68019 0x0bb2f5b2 0x0000ee992bb2f5b2 0x166cee992bb2f5b2 0x166cee992d6fc8b2 MERGE Data Link (20503): bad timestamp 3 3 0x83807863 0x0bb30fca 0x0000ee992bb30fca 0x166cee992bb30fca 0x166cee992d6fc8b2 MERGE Data Link (20502): bad timestamp 2 3 0xc0877f3c 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770 MERGE Data Link (20502): bad timestamp 2 3 0xc09a7bc6 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770 MERGE Data Link (20502): bad timestamp 2 3 0xc08c4e12 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770 MERGE Data Link (20502): bad timestamp 2 3 0xc09c7df4 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770 Earliest & latest timestamps from latest batch = Tuesday, March 16, 2021 7:48:18.767 PM - Tuesday, March 16, 2021 9:06:38.184 PM this evening 21.17 FEE64 Temperatures OK - attachment 4 Good event statistics OK - attachment 5 Detector bias & leakage currents OK - attachment 6 Merger OK - 4.5M data items/s TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-03-16_Statistics_aidas-gsi.png

Attachment 3: 82.png

Attachment 4: Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 5: Screenshot_2021-03-16_Statistics_aidas-gsi(1).png

Attachment 6: 83.png

575	Tue Apr 16 10:49:37 2024	TD, JB, CC	Tuesday 16 April
11.45 DSSSD bias & leakage current OK - attachments 9-10 FEE64 temps OK - attachment 8 except aida02 ASIC temp which is known to be u/s All system wide checks OK except aida02 and aida04 WR decoder/FPGA status errors - attachments 6-7 HDMI cabling disturbed during platform move? WR timestamps OK - attachment 5 ADC data item stats - attachment 4 10/16 < 20k, max 115k per FEE64 Rate spectra - attachments 3 per 1.8.W spectra - 20us FSR - attachments 1-2 11.58 BB7 preamps unplugged. Noise conditions OK. Did not change considerably. 11/16 < 20k (aida09 has 1 hot channel), max 114k Attachments 11-12. 12:34 Detector powered down to bring to safe state to check HDMI cables of aida02 and aida03. 15:06 Reseated aida02 and aida03. No more FPGA timestamp error. See attachment 13-14. Rates and histograms look good for pulser walk through. Test + and test - connected. Currently, Attachment 16-17. Temp OK Attachment 15. Bias OK Attachment 18. Data links aida02 and aida08 missing (?) 16:30 All slow comparator thresholds set to 0x64 for alpha background run during platform freeze. Initial rates and histograms attachment 19 - 20. aida04 noisy in ASIC 1. Temps OK. Attachment 21. MBS disc and tape setup in atttachment 22-23.
Attachment 1: Screenshot_from_2024-04-16_11-49-17.png

Attachment 2: Screenshot_from_2024-04-16_11-48-28.png

Attachment 3: Screenshot_from_2024-04-16_11-47-40.png

Attachment 4: Screenshot_from_2024-04-16_11-46-48.png

Attachment 5: Screenshot_from_2024-04-16_11-46-01.png

Attachment 6: Screenshot_from_2024-04-16_11-45-15.png

Attachment 7: Screenshot_from_2024-04-16_11-44-37.png

Attachment 8: Screenshot_from_2024-04-16_11-44-06.png

Attachment 9: Screenshot_from_2024-04-16_11-51-17.png

Attachment 10: Screenshot_from_2024-04-16_11-50-22.png

Attachment 11: Screenshot_from_2024-04-16_11-58-26.png

Attachment 12: Screenshot_from_2024-04-16_11-58-08.png

Attachment 13: Screenshot_from_2024-04-16_15-04-13.png

Attachment 14: Screenshot_from_2024-04-16_15-04-07.png

Attachment 15: Screenshot_from_2024-04-16_15-06-31.png

Attachment 16: Screenshot_from_2024-04-16_15-06-12.png

Attachment 17: Screenshot_from_2024-04-16_15-05-33.png

Attachment 18: Screenshot_from_2024-04-16_15-08-04.png

Attachment 19: Screenshot_from_2024-04-16_15-10-46.png

Attachment 20: Screenshot_from_2024-04-16_17-17-35.png

Attachment 21: Screenshot_from_2024-04-16_17-17-32.png

Attachment 22: Screenshot_from_2024-04-16_17-14-40.png

Attachment 23: Screenshot_from_2024-04-16_17-17-24.png

Attachment 24: Screenshot_from_2024-04-16_17-17-11.png

369	Tue Jun 15 12:35:48 2021	TD	Tuesday 15 June
13.35 DAQ continues OK - no storage mode All system wide checks OK except Base Current Difference aida01 fault 0x8944 : 0x8949 : 5 aida02 fault 0x9b36 : 0x9b3b : 5 aida03 fault 0xf26b : 0xf270 : 5 aida04 fault 0x57a0 : 0x57a5 : 5 aida05 fault 0x4d3c : 0x4d3e : 2 aida05 : WR status 0x10 aida06 fault 0x640c : 0x640e : 2 aida07 fault 0x255e : 0x2560 : 2 aida08 fault 0xffbd : 0xffbf : 2 aida15 fault 0x4e4 : 0x1267 : 3459 White Rabbit error counter test result: Passed 7, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x3e01 : 15873 aida13 fault 0x0 : 0x32d2 : 13010 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 37 10 8 5 4 4 2 2 2 4 6 : 37636 aida02 : 37 10 10 3 4 4 3 2 2 4 6 : 37860 aida03 : 41 8 7 7 5 3 1 3 2 4 6 : 37876 aida04 : 33 9 4 4 2 5 2 3 3 3 6 : 37004 aida05 : 37 9 9 3 5 4 2 2 2 4 6 : 37644 aida06 : 39 9 6 5 4 2 2 3 4 3 6 : 37860 aida07 : 29 12 6 5 4 2 3 2 2 4 6 : 37588 aida08 : 25 7 9 5 4 3 3 2 2 4 6 : 37708 aida09 : 34 5 9 6 2 3 3 2 2 4 6 : 37632 aida10 : 34 12 11 5 4 3 3 3 3 3 6 : 37304 aida11 : 38 8 7 6 3 4 3 3 1 4 6 : 37320 aida12 : 34 10 8 6 3 4 3 3 1 4 6 : 37336 aida13 : 33 11 5 6 4 4 3 3 1 4 6 : 37356 aida14 : 31 10 5 6 5 3 3 3 3 3 6 : 37276 aida15 : 14 6 7 4 1 2 3 2 1 4 7 : 40344 aida16 : 33 8 6 7 5 3 3 3 1 4 6 : 37316 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1 Lost activity monitor - attachment 2 1.8.W spectra - 20us FSR - attachments 3 & 4 ADC data items - attachment 5 FEE64 temperatures OK - attachment 6 DSSSD bias & leakage currents OK - attachment 7 Merger/Tape Server/Merger statistics - attachment 8 no merger errors reported since previous restart 13.40 All histograms, stats & merger stats zero'd
Attachment 1: Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png

Attachment 2: Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png

Attachment 3: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png

Attachment 4: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-06-15_Statistics_aidas-gsi.png

Attachment 6: Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png

Attachment 7: 1002.png

Attachment 8: 1003.png

218	Tue Apr 13 16:49:41 2021	TD	Tuesday 13 April
Rename Options directory mv Options Options.BAK-130421 Copy Options.PRISTINE to Options cp -r Options.PRISTINE Options chmod -R 777 Options Start Options directory monitor (will check directory every 5 minutes) Python ~/oh/OptionsCheck 17.32 Power cycle FEE64s FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Calibration test result: Passed 12, Failed 0 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit error counter test result: Passed 12, Failed 0 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Timestamp error counter test result: Passed 12, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 3 5 2 2 1 3 3 2 2 3 8 : 43348 aida02 : 2 3 0 2 1 2 2 3 2 3 8 : 43424 aida03 : 3 4 0 1 1 2 2 3 2 3 8 : 43404 aida04 : 1 4 2 0 3 3 3 2 2 3 8 : 43396 aida05 : 1 4 2 2 2 3 3 2 2 3 8 : 43396 aida06 : 2 3 2 1 1 2 2 3 2 3 8 : 43424 aida07 : 4 1 3 3 1 3 1 3 2 3 8 : 43368 aida08 : 1 4 2 2 2 3 3 2 2 3 8 : 43396 aida09 : 2 3 3 3 2 4 2 2 2 3 8 : 43312 aida10 : 1 3 1 2 2 3 2 2 2 3 8 : 43116 aida11 : 1 3 3 1 2 3 2 2 2 3 8 : 43116 aida12 : 3 2 1 0 1 2 1 3 2 3 8 : 43116 Note that system wide checks identify 1x file of different size immediately after boot/reset/setup Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida02 => Options file size is 1025 Last changed Tue Apr 13 17:59:04 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Tue Apr 13 17:59:04 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Tue Apr 13 17:59:04 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Tue Apr 13 17:59:05 CEST 2021 06:25:00 PM FAULTS! Unknown option in FEE2 Options database Unknown entry starts: da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable 06:30:00 PM FAULTS! Unknown option in FEE2 Options database Unknown entry starts: da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable Unknown option in FEE2 Options database Unknown entry starts: et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable FEE64 module aida01 => 7 FEE64 module aida02 => 7 FEE64 module aida03 => 7 FEE64 module aida04 => 7 FEE64 module aida05 => 7 FEE64 module aida06 => 7 FEE64 module aida07 => 7 FEE64 module aida08 => 7 FEE64 module aida09 => 7 FEE64 module aida10 => 7 FEE64 module aida11 => 7 FEE64 module aida12 => 7 ASIC Clock Timestamp check test result: Passed 12, Failed 0 If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status aida01 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida02 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida03 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida04 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida05 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida06 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida07 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida08 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida09 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida10 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida11 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 aida12 timestamp memory No stop found : 0x0 0 0 0 0 0 4 0 0 0 0 8 0 0 0 0 12 0 0 0 0 16 0 0 0 0 20 0 0 0 0 24 0 0 0 0 28 0 0 0 0 17.14 rate spectra - attachment 1 1.8.L spectra - attachments 2 & 3 pulser peak width aida06 c.75 ch FWHM, aida07 c.85 ch FWHM 1.8.W spectra - attachments 4 & 5 good events statistics (slow comparator 0xa) - attachment 6 FEE64 temperatures OK - attachment 7 DSSSD bias & leakage currents OK - attachment 8 17.38 DAQ starts file NULL/R14 slow comparator 0xa -> 0x64 BNC PB-5 pulser OFF alpha background 10:40 CET 14.04.2021 - NH Close file NULL/R14
Attachment 1: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi.png

Attachment 2: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(1).png

Attachment 3: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(2).png

Attachment 4: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(3).png

Attachment 5: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(4).png

Attachment 6: Screenshot_2021-04-13_Statistics_aidas-gsi.png

Attachment 7: Screenshot_2021-04-13_Temperature_and_status_scan_aidas-gsi.png

Attachment 8: 3.png

219	Wed Apr 14 09:55:12 2021	TD	Tuesday 13 April
Two further comments 1) The 'pristine' Options files produced by OH fixed all of the undefined spectra issues, i.e. we now have L, H, Rate, Stat, Hit, HitRate spectra for all FEE64s 2) The system has undegone 1x power cycle/reset/setup and 1x reset/setup cycle since the new Options files were installed and we currently have 2x errors reported in aida02. There is an open Options tab and we have viewed the Options settings for aida01 and aida02 (currently viewing aida02) - no changes/saves/restores requested. > > Rename Options directory > > mv Options Options.BAK-130421 > > Copy Options.PRISTINE to Options > > cp -r Options.PRISTINE Options > chmod -R 777 Options > > Start Options directory monitor (will check directory every 5 minutes) > > Python ~/oh/OptionsCheck > > > 17.32 Power cycle FEE64s > > > FEE64 module aida09 global clocks failed, 6 > Clock status test result: Passed 11, Failed 1 > > Understand status as follows > Status bit 3 : firmware PLL that creates clocks from external clock not locked > Status bit 2 : always logic '1' > Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock > Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock > If all these bits are not set then the operation of the firmware is unreliable > > Calibration test result: Passed 12, Failed 0 > > If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module > > White Rabbit error counter test result: Passed 12, Failed 0 > > Understand the status reports as follows:- > Status bit 3 : White Rabbit decoder detected an error in the received data > Status bit 2 : Firmware registered WR error, no reload of Timestamp > Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR > > > FPGA Timestamp error counter test result: Passed 12, Failed 0 > If any of these counts are reported as in error > The ASIC readout system has detected a timeslip. > That is the timestamp read from the time FIFO is not younger than the last > > > > Returned 0 0 0 0 0 0 0 0 0 0 0 0 > Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k > aida01 : 3 5 2 2 1 3 3 2 2 3 8 : 43348 > aida02 : 2 3 0 2 1 2 2 3 2 3 8 : 43424 > aida03 : 3 4 0 1 1 2 2 3 2 3 8 : 43404 > aida04 : 1 4 2 0 3 3 3 2 2 3 8 : 43396 > aida05 : 1 4 2 2 2 3 3 2 2 3 8 : 43396 > aida06 : 2 3 2 1 1 2 2 3 2 3 8 : 43424 > aida07 : 4 1 3 3 1 3 1 3 2 3 8 : 43368 > aida08 : 1 4 2 2 2 3 3 2 2 3 8 : 43396 > aida09 : 2 3 3 3 2 4 2 2 2 3 8 : 43312 > aida10 : 1 3 1 2 2 3 2 2 2 3 8 : 43116 > aida11 : 1 3 3 1 2 3 2 2 2 3 8 : 43116 > aida12 : 3 2 1 0 1 2 1 3 2 3 8 : 43116 > > > Note that system wide checks identify 1x file of different size immediately after boot/reset/setup > > Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same > > FEE : aida01 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida02 => Options file size is 1025 Last changed Tue Apr 13 17:59:04 CEST 2021 > FEE : aida03 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida04 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida05 => Options file size is 1014 Last changed Tue Apr 13 17:59:04 CEST 2021 > FEE : aida06 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida07 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida08 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida09 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida10 => Options file size is 1014 Last changed Tue Apr 13 17:59:04 CEST 2021 > FEE : aida11 => Options file size is 1014 Last changed Tue Apr 13 17:59:03 CEST 2021 > FEE : aida12 => Options file size is 1014 Last changed Tue Apr 13 17:59:05 CEST 2021 > > 06:25:00 PM > FAULTS! > > Unknown option in FEE2 Options database > Unknown entry starts: > da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV E_DMA_HWM&&Ai > da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable > > 06:30:00 PM > FAULTS! > > Unknown option in FEE2 Options database > Unknown entry starts: > da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV E_DMA_HWM&&Ai > da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable > > Unknown option in FEE2 Options database > Unknown entry starts: > et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.setting s&&WAVE_DMA_H > WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_En able > > > > > > FEE64 module aida01 => 7 > FEE64 module aida02 => 7 > FEE64 module aida03 => 7 > FEE64 module aida04 => 7 > FEE64 module aida05 => 7 > FEE64 module aida06 => 7 > FEE64 module aida07 => 7 > FEE64 module aida08 => 7 > FEE64 module aida09 => 7 > FEE64 module aida10 => 7 > FEE64 module aida11 => 7 > FEE64 module aida12 => 7 > > > ASIC Clock Timestamp check test result: Passed 12, Failed 0 > > > If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status > > > > > aida01 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida02 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida03 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida04 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida05 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida06 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida07 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida08 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida09 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida10 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida11 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > aida12 timestamp memory No stop found : 0x0 > 0 0 0 0 0 > 4 0 0 0 0 > 8 0 0 0 0 > 12 0 0 0 0 > 16 0 0 0 0 > 20 0 0 0 0 > 24 0 0 0 0 > 28 0 0 0 0 > > 17.14 rate spectra - attachment 1 > > 1.8.L spectra - attachments 2 & 3 > pulser peak width aida06 c.75 ch FWHM, aida07 c.85 ch FWHM > > 1.8.W spectra - attachments 4 & 5 > > good events statistics (slow comparator 0xa) - attachment 6 > > FEE64 temperatures OK - attachment 7 > > DSSSD bias & leakage currents OK - attachment 8 > > > > > 17.38 DAQ starts file NULL/R14 > slow comparator 0xa -> 0x64 > BNC PB-5 pulser OFF > alpha background > > 10:40 CET 14.04.2021 - NH > Close file NULL/R14
305	Tue May 11 07:21:24 2021	OH	Tuesday 11 May
08:20 System wide checks all ok except Base Current Difference aida01 fault 0xc10c : 0xc10e : 2 aida02 fault 0xbe2a : 0xbe2c : 2 aida03 fault 0xf665 : 0xf667 : 2 aida04 fault 0xeabd : 0xeabf : 2 aida05 fault 0x36c8 : 0x36ca : 2 aida06 fault 0x2ea2 : 0x2ea4 : 2 aida07 fault 0x9186 : 0x9188 : 2 aida08 fault 0x5443 : 0x5445 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR and Base Current Difference aida01 fault 0xc10c : 0xc10e : 2 aida02 fault 0xbe2a : 0xbe2c : 2 aida03 fault 0xf665 : 0xf667 : 2 aida04 fault 0xeabd : 0xeabf : 2 aida05 fault 0x36c8 : 0x36ca : 2 aida06 fault 0x2ea2 : 0x2ea4 : 2 aida07 fault 0x9186 : 0x9188 : 2 aida08 fault 0x5443 : 0x5445 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics still high - attachment 1 Temperatures ok - attachment 2 Bias and leakage currents ok - attachment 3
Attachment 1: 210511_0823_Stats.png

Attachment 2: 210511_0824_Temp.png

Attachment 3: 210511_0825_Bias.png

427	Tue May 10 07:41:28 2022	NH TD	Tuesday 10 May
8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam Reseat aida05 ribbon cable on adapter board while in... No difference to rates, aida05 still shows hot asic01 aida14 seems seated OK, still asic02 basically empty Noise level still high, threshold set to 0x20 (320 keV) which has decent rate Start forwarding to MBS with 0x20 thresholds 10.12 slow comparator 0x20 -> 0xa Attachments 1-5 Attachment 6 ADC data items c. 150-180k per FEE64, c. 21Mb to disk Attachment 7 FEE64 temps OK Attachment 8 DSSSD bias & leakage currents OK 13.30 system checks N.B. aida05 ASIC temp low and pause/resume data rate high Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK Attachments 10-15 spectra Attacahments 16-19 system wide checks OK except WR error counter Attachment 20 ADC data items c. 150-180k per FEE64, c. 21Mb to disk Attachment 21 FEE64 temps OK Attachment 22 DSSSD bias & leakage currents OK Attachments 23-29 newMerger stats 20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable 22:26 - No implantation test although we had some brief beam through AIDA. - FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow Will keep DAQ on 0x64 to eliminate deadtime issues for implantation
Attachment 1: Screenshot_from_2022-05-10_10-11-54.png

Attachment 2: Screenshot_from_2022-05-10_10-10-09.png

Attachment 3: Screenshot_from_2022-05-10_10-08-55.png

Attachment 4: Screenshot_from_2022-05-10_10-08-02.png

Attachment 5: Screenshot_from_2022-05-10_10-07-01.png

Attachment 6: Screenshot_from_2022-05-10_10-06-30.png

Attachment 7: Screenshot_from_2022-05-10_10-06-20.png

Attachment 8: Screenshot_from_2022-05-10_10-06-15.png

Attachment 9: Screenshot_from_2022-05-10_13-40-43.png

Attachment 10: Screenshot_from_2022-05-10_13-40-01.png

Attachment 11: Screenshot_from_2022-05-10_13-37-40.png

Attachment 12: Screenshot_from_2022-05-10_13-37-33.png

Attachment 13: Screenshot_from_2022-05-10_13-36-39.png

Attachment 14: Screenshot_from_2022-05-10_13-35-06.png

Attachment 15: Screenshot_from_2022-05-10_13-34-44.png

Attachment 16: Screenshot_from_2022-05-10_13-31-05.png

Attachment 17: Screenshot_from_2022-05-10_13-30-59.png

Attachment 18: Screenshot_from_2022-05-10_13-30-52.png

Attachment 19: Screenshot_from_2022-05-10_13-29-58.png

Attachment 20: Screenshot_from_2022-05-10_13-29-42.png

Attachment 21: Screenshot_from_2022-05-10_13-29-20.png

Attachment 22: Screenshot_from_2022-05-10_13-28-48.png

Attachment 23: Screenshot_from_2022-05-10_13-51-23.png

Attachment 24: Screenshot_from_2022-05-10_13-51-05.png

Attachment 25: Screenshot_from_2022-05-10_13-49-53.png

Attachment 26: Screenshot_from_2022-05-10_13-49-41.png

Attachment 27: Screenshot_from_2022-05-10_13-49-32.png

Attachment 28: Screenshot_from_2022-05-10_13-49-12.png

Attachment 29: Screenshot_from_2022-05-10_13-48-56.png

137	Tue Mar 10 08:48:07 2020	TD, OH, NH, LS, CA, DK, PJW, RP	Tuesday 10 March 08.00-16.00
c 09.00 aida03 drops Histogram enable at DAQ GO NH determined this was due to Options DB corruption and fixed 09.55 DAQ currently running No Storage mode ASIC settings 2019Oct31-13.24.23 (different from the December ones yesterday which experienced corruption) slow comparator 0xa BNC PB-5 pulser amplitude 1.0V , attenuator x1 frequency 2Hz decay time 1ms DSSSD biases & leakage currents OK - attachment 1 FEE64 temps OK - attachment 2 good events statistics OK - attachment 3 system wide checks - attachments 4-11 aida06, 08, 10, 12 re-calibrated OK, all others OK aida03 failed WR decoder status - collect all WR errors for baseline - all FEE64s pass WR decoder status sync all ASIC clocks using ReSync merger & TapeServer OK - attachments 12-14 13.29 DAQ to disk file S480/R6 ASIC settings 2019Oct31-13.24.23 slow comparator 0xa aida01 LEC/MEC fast comparator 0x64, all other FEE64s 0xff 14.57 fast comparator thresholds LEC/MEC changed to 0xff for all FEEs 15.01 FEE temperatures normal, aida1 virtex temperature still slightly over 65 degrees (attachment 15) leakage currents recorded to spreadsheet (attachment 16) good events rate similar to previous, high in DSSD3 and seems to be returning to normal in DSSD1 except in one FEE (attachments 17 and 18) system wide checks OK data rate around 22000 KB per second (attachment 19) as of 15.12 on file R6_80 attachment 20 is total event stats shows events are being implanted attachment 21 and 22 show low energy spectrum left most peak is pulser and rightmost is central noise. middle peak in aida1 attachment 23 and 24 show high energy spectrum, aida4 histogram not being produced unsure why. attachment 25 and 26 show waveforms, only aida3 histogram working 16.00 checked ASIC control, all OK 16.35 spill is now changed to 10s with beam intensity increase by factor of 10. 3000 implants per spill. 16.42 leakage currents recorded to spreadsheet 17.10 system wide checks ok fee64 temperatures ok - attachment 27 detector bias/leakage currents ok - attachment 28 good event statistics ok - attachment 29 17.15 moved slit at S2 - rate drop to try remove some 96Pd 17.20 writing to file R6_166 17.44 merger/tapeserver ok. Data rate ~ 25000 kb/s 18:10 Fine tuning implantation profile Removing 0.25mm Al degrader (60 mg/cm^2) to get 96Pd slightly further in to AIDA. DESPEC is closing the current file, R6_203 was just finished on AIDA on typing this. (Maybe the end of run S480f004_ on DESPEC side but that elog was not updated since 16:40?) Decision based on assumption of 6 GeV 96Pd having a range 1 mm in silicon. 19.16 system wide checks ok good event statistics ok - attachment 30 detector bias/leakage currents ok - attachment 31 fee64 temperatures ok - attachment 32 19.39 TapeServer/Merger ok. Writing to file R6_265. Data rate ~ 25000 kb/s 20.07 1.8.H high energy spectra front and back side (attachments 33 + 34) 20.36 S4 timing change, increased by factor of 2 20.38 Merger/TapeServer ok. Data being forwarded to MBS. 20.38 writing to file R6_308 at rate of ~ 25000 kb/s 21:21 system wide checks ok good event statistics ok - attachment 35 fee64 temperatures ok - attachment 36 detector bias/leakage currents ok - attachment 37 21:57 TapeServer/merger ok data forwarding to MBS ok writing to file R6_368 at rate of ~26000 kb/s 23:10 system wide checks ok good event statistics ok - attachment 38 fee64 temperatures ok - attachment 39 detector bias/leakage currents ok - attachment 40 23:24 TapeServer/merger ok data forwarding to MBS ok writing to file R6_435 at rate of 27000 kb/s 23:42 rate spectra - attachment 41
Attachment 1: 1.png

Attachment 2: 2.png

Attachment 3: 3.png

Attachment 4: 4.png

Attachment 5: 5.png

Attachment 6: 6.png

Attachment 7: 7.png

Attachment 8: 8.png

Attachment 9: 9.png

Attachment 10: 10.png

Attachment 11: 11.png

Attachment 12: 12.png

Attachment 13: 13.png

Attachment 14: 14.png

Attachment 15: 200310_1419_feetemps.png

Attachment 16: 200310_1451_leakcurrents.png

Attachment 17: 200310_1450_goodevents.png

Attachment 18: 200309_1506goodevents.png

Attachment 19: 200309_1510_datarate.png

Attachment 20: 200310_1517_stathistograms.png

Attachment 21: 200310_1517_lowEoddfees.png

Attachment 22: 200310_1522_lowEevevfees.png

Attachment 23: 200310_1523_highEoddfees.png

Attachment 24: 200310_1525_highE_evenfees.png

Attachment 25: 200310_1529_waveformsoddfees.png

Attachment 26: 200310_1530_waveformsevenfees.png

Attachment 27: 200310_1707_temp.png

Attachment 28: 200310_1708_bias.png

Attachment 29: 200320_1708_stats.png

Attachment 30: 200310_1915_stats.png

Attachment 31: 200310_1916_bias.png

Attachment 32: 200310_1919_temp.png

Attachment 33: 200310_2000_18h_odd.png

Attachment 34: 200310_2005_18h_even.png

Attachment 35: 200310_2119_stats.png

Attachment 36: 200310_2119_temp.png

Attachment 37: 200310_2120_bias.png

Attachment 38: 200310_2307_temp.png

Attachment 39: 200310_2308_stats.png

Attachment 40: 200310_2309_bias.png

Attachment 41: 200310_2342_rates.png

638	Tue Jun 4 15:40:32 2024	NH	Tues 4 Jun
DEGAS Array over Snout again Situation similar to yesterday, most p+n rates good
Attachment 1: Screenshot_from_2024-06-04_16-39-28.png

Attachment 2: Screenshot_from_2024-06-04_16-39-47.png

507	Tue Jun 28 09:37:19 2022	NH	Tues 28 June 08:00-
Experiment over 10:37 - Stop DAQ & Tape S4 enters controlled access and they uncable bPlas Will dismount AIDA snout after
550	Tue Mar 26 10:32:05 2024	NH, JB	Tue 26 March
Taken 4 FEE64s from CRYRING (the 4 easiest to access) 41:d8:2b 41:f6:5a 41:ee:71 41:d0:0e In addition a FEE without rails marked only 20 was found in my old office for 5 - This probably came in a shipment to/for repairs DSSSD#1 unbiased DESPEC Platform moved out of beam and gamma platform moved from AIDA JB works on grounding the original 8 DSSSD#1 FEEs Connected grounding from ribbon cable drain to adapter PCB Ground loop connected AIDA rates unchanged bad still fig 1 Waves fig 2 (p+n 6000-9000) Waves fig 3 (n+n 8000-10000) Statistics fig 4
Attachment 1: Screenshot_from_2024-03-26_14-39-35.png

Attachment 2: Screenshot_from_2024-03-26_14-40-05.png

Attachment 3: Screenshot_from_2024-03-26_14-40-30.png

Attachment 4: Screenshot_from_2024-03-26_14-41-56.png

Attachment 5: 20240326_144539.jpg

Attachment 6: 20240326_144638.jpg

Attachment 7: 20240326_144528.jpg

Attachment 8: 20240326_144659.jpg

Attachment 9: 20240326_144627.jpg

472	Tue May 24 08:35:00 2022	NH	Tue 24 May
09:00 - Remove bias of DSSDs and pwoer of FEEs Remove snout from S4 so DEGAS frame can be mechnically removed Snout now in FRS Messhuette
383	Mon Oct 18 14:20:32 2021	OH	Top bar launchers
launcher1 TclHttpd @8015 for AIDA /MIDAS/Linux/startup/HTTPD launcher2 TclHttpd @8115 for AIDA /MIDAS/Linux/startup/HTTPD@8115 launcher3 TapeServer /MIDAS/Linux/startup/TapeServer launcher4 New Merger for AIDA /MIDAS/Linux/startup/NewMerger Launcher5 MBS Relay /MIDAS/Linux/startup/datarelaymbs
533	Mon Mar 11 15:19:44 2024	JB, NH, Priyanka, Michael Armstrong, Helena Albers	To Do: AIDA PCB tests
To do: 1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs - When the p+n ribbon cables were disconnected for both the upstream and downstream AIDA, the pins and ribbon cables did not appear to have any blemishes. - Both sets of adaptor PCBs were biased to 100 V and only had an apparent fluctuation in current at c. 0.01 uA. At low voltages (~10V) the current cycled between 0 and 1 uA. 2) Check that all ribbon cables are properly seated in the adaptor PCBs - When inspected all ribbon cables seemed to be properly seated. A decent press was applied to confirm the seating. The PCBs themselves were inspected and no noticeable damage was observed. 3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables - Done for the p+n side, can decide if this is needed for the n+n side. 4) *If/when you open check seating of all Kapton PCBs in the DSSSD connectors and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say. - Will have to decide further. 5) From email discussion: Check the SHV connector is grounded if one unplugs the cable from the back of the HV module. If yes: We are touching a ground somewhere. If no: it's via the HV (fine). - When HV#0 was disconnected from the HV module, we used a multimeter on the SHV connecter and frame and read zero resistance 0L on the meter. While when completing the circuit for HV#1 (still) plugged into the HV module we read a resistance of > 1.7 Ohm. - The HV is touching ground somewhere. This should not be a problem however as the snout is isolated, and this was observed by connecting the SHV to the snout and reading 0L. 6) Properly cover snout with black cloth and bias upstream.** - Snout was covered with black cloth and a black bag. The downstream detector p+n side was biased at 50V and charging/discharging was observed. This is probably due to a short connection somewhere. Summary: from the p+n side test the voltage-current break down appears to come from inside the snout.
510	Tue Jul 5 08:53:20 2022	TD	To Do
In no particular order 1) CAEN 83xx series NIM bin (Ortec 533A output noise issue) observe +/- 6V, 12V, 24V lines with/without load try new CAEN NIM bin and/or NIM bin of different type 2) Measure actual voltages at FEE64 power connector input OH suggests fab of power adaptor for safe observation - contact EW 3) rev B adaptor PCB invert 125 way ERNI - check for mech conflicts paired HV input (avoid Lemo-00 T pieces) consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination) straight jumpers shrouded Samtec headers - consider mech issues/consequences of using eject clips too re-visit HV filtering & separate trace ground 4) isolation transformer as practical matter may be necessary to operate all platform from isolation transformer consider hire of appropriate unit need method to measure isolation - will require permit to work or equiv 5) investigate S4 area ac mains NH discussing with GSI electricians 6) Systematic measurement of AIDA PSU noise Spec linear AIDA FEE64 PSU 7) Redesign of snout Return to 1mm welded box Al for lower stage of snout for added rigidity 8) Revisit calculation of cable lengths. Particularly for the triple 9)
685	Thu Jan 9 15:43:17 2025	JB, CC, MP	Timing test platform moved in
16:30 We moved the platform in. The system was still biased and the DAQ was still running. The system seems to be okay, temperatures OK. Bias voltage OK 685/1 and 685/2. Two FEEs - aida07 and aida12 are receiving a lot of trigger data items from the fast discriminator. This is already a bit worrying. 685/3
Attachment 1: Screenshot_from_2025-01-09_16-41-48.png

Attachment 2: Screenshot_from_2025-01-09_16-42-06.png

Attachment 3: Screenshot_from_2025-01-09_16-42-46.png

110	Thu Dec 19 09:22:52 2019	TD	Thusrday 19 December
10.15 Found DAQ running at c. 1M data items/s Rate spectra show aida09 1..H and one 1x LEC channel of aida08 asic running hot Check ASIC settings all* FEE64s all ASICs restores rates to normal /TapeData/171219/R2_* files indicates files R2_0 ... R2_1 are probably OK and problem started c. 08.00 - activity in S4? -rw-rw-r--. 1 npg npg 2097152000 Dec 19 01:56 R2_0 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 05:30 R2_1 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:05 R2_2 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:15 R2_3 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:29 R2_4 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:34 R2_5 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:38 R2_6 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:43 R2_7 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:47 R2_8 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:52 R2_9 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:56 R2_10 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:00 R2_11 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:05 R2_12 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:09 R2_13 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:14 R2_14 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:18 R2_15 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:23 R2_16 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:27 R2_17 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:31 R2_18 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:36 R2_19 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:40 R2_20 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:45 R2_21 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:49 R2_22 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:54 R2_23 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:58 R2_24 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:02 R2_25 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:07 R2_26 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:11 R2_27 -rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:16 R2_28 -rw-rw-r--. 1 npg npg 1582563328 Dec 19 10:19 R2_29 10.31 1.8.L and Stat spectra - attachments 1-3 10.32 DAQ stop file 171219/R2_30 zero all histograms Check ASIC settings all FEE64s all ASICs 10.48 system wide checks OK except aida07 calibration & master clock status fail - attachments 4-9 GSI WR timestamp control OK - attachments 10-11 DSSSD biases & leakage currents OK - attachment 12 FEE64 temperatures OK - attachment 13 Good event statistics OK - attachment 14 11.31 DAQ/Merger/TapeServer stopped merging/writing data at start of new run Stop DAQ/Merger/TapeServer, reboot FEE64s and restart from scratch DAQ start file 171219/R8 slow comparator 0x64 alpha background data rate c. 170kb/s 15.30 DAQ continues OK file 171219/R9_0 DSSSD biases & leakage currents OK - attachment 15 BNC Pb-5 settings - attachment 16 FEE64 temperatures OK - attachment 17 Good event statistics OK - attachment 18 system wide checks OK except master clock status fail GSI WR timestamp control OK - attachments 19-20 Merger & server consoles - attachment 21
Attachment 1: 100.png

Attachment 2: 101.png

Attachment 3: 102.png

Attachment 4: 110.png

Attachment 5: 111.png

Attachment 6: 112.png

Attachment 7: 113.png

Attachment 8: 114.png

Attachment 9: 115.png

Attachment 10: 116.png

Attachment 11: 117.png

Attachment 12: 200.png

Attachment 13: 201.png

Attachment 14: 202.png

Attachment 15: 1.png

Attachment 16: 2.png

Attachment 17: 3.png

Attachment 18: 4.png

Attachment 19: 5.png

Attachment 20: 6.png

Attachment 21: 8.png

188	Thu Mar 11 00:38:33 2021	TD	Thusrday 11 March 00.00-08.00
01.38 DAQ continues OK - file R22_388 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse System wide checks all OK except FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module FEE64 Temperatures OK - attachment 1 Good event statistics OK - attachment 2 Detector bias & leakage currents OK - attachment 3 Merger OK - 4.4M data items/s TapeServer OK - 14Mb/s 01.47 no merger server error/warning messages since last check by TD at start of shift 00.00 03.44 merger server error/warning messages since last check e.g. MERGE Data Link (30260): bad timestamp 6 3 0xc1bd7e9a 0x0ce9c9c6 0x0000292e0ce9c9c6 0x166b292e0ce9c9c6 0x166b292e0cea5486 03.46 DAQ continues OK - file R22_441 System wide checks all OK except FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module FPGA timestamp errors Base Current Difference aida07 fault 0x1 : 0x2 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last FEE64 Temperatures OK - attachment 4 Good event statistics OK - attachment 5 Detector bias & leakage currents OK - attachment 6 Merger OK - 4.6M data items/s TapeServer OK - 15Mb/s 02.51 Rate spectra - attachment 7 04.14 p+n junction HEC spectra - attachment 8 all spectra zero'd 06.22 merger server error/warning messages since last check e.g. MERGE Data Link (30265): bad timestamp 11 3 0xc2fc8067 0x02df391e 0x0000324e62df391e 0x166b324e62df391e 0x166b324e643c775e 06.23 DAQ continues OK - file R22_508 System wide checks all OK except FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit decoder status Base Current Difference aida01 fault 0xe67 : 0xe68 : 1 aida02 fault 0x6b22 : 0x6b23 : 1 aida03 fault 0x4c76 : 0x4c77 : 1 aida04 fault 0x7542 : 0x7543 : 1 aida05 fault 0x1597 : 0x1599 : 2 aida06 fault 0xe241 : 0xe243 : 2 aida07 fault 0x749 : 0x74c : 3 aida08 fault 0xf854 : 0xf856 : 2 White Rabbit error counter test result: Passed 4, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp errors Base Current Difference aida07 fault 0x1 : 0x2 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last FEE64 Temperatures OK - attachment 9 Good event statistics OK - attachment 10 Detector bias & leakage currents OK - attachment 11 Merger OK - 4.8M data items/s TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-03-11_Statistics_aidas-gsi.png

Attachment 3: 20.png

Attachment 4: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 5: Screenshot_2021-03-11_Statistics_aidas-gsi(1).png

Attachment 6: 21.png

Attachment 7: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi.png

Attachment 8: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi(1).png

Attachment 9: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 10: Screenshot_2021-03-11_Statistics_aidas-gsi(2).png

Attachment 11: 22.png

Goto page Previous 1, 2, 3 ... 6, 7, 8 ... 33, 34, 35 Next

ELOG V3.1.4-unknown