ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 19 of 35

Full | Summary | Threaded | Show attachments

687 Entries

Goto page Previous 1, 2, 3 ... 18, 19, 20 ... 33, 34, 35 Next

Selected entries:

	ID	Date	Author	Subject
	336	Thu May 20 07:03:02 2021	CA	Thursday May 20th 08:00 - 16:00 shift
08:00 CA takes over 08:04 DAQ continues ok - writing to file R14_989 09:30 all system wide checks ok except WR decoder status; Base Current Difference aida05 fault 0x500 : 0x56a : 106 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 09:34 temperatures ok - attachment 1 stats ok - attachment 2 bias/leakage currents ok - attachment 3 R14_1016 analysis - max deadtime ~4.3% - attachment 4 11:30 all system wide checks ok except WR decoder status; Base Current Difference aida05 fault 0x500 : 0x56b : 107 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 11:34 temperatures ok - attachment 5 stats ok - attachment 6 bias/leakage currents ok - attachment 7 R14_1052 analysis - max deadtime ~3.6% - attachment 8 13:05 beam optimisation currently taking place DAQ continues ok and writing to file R14_1085 13:17 no beam 13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30. 13:31 temperatures ok - attachment 9 stats ok - attachment 10 bias/leakage currents ok - attachment 11 R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12 13:37 beam is back - rate spectra - attachment 13 DAQ ok - writing to file R14_1088 13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14 14:21 beam off - issue with UNILAC DESPEC also report issues with their DAQ AIDA continues to forward data to MBS ok - lower rates while beam off 14.30 1.8.H spectra - attachments 16-18 1.8.L spectra - attachments 19-22 aida01 pulser peak width 99 ch FWHM Rate spectra - attachment 23 Grafana - DSSSD bias & leakage currents past 7 days - attachment 24 DSSSD bias & leakage currents - attachment 25 Merger/Tape Server/Merger stats - attachment 26 14.41 All histograms & stats zero'd 14.55 Lost Activity Monitors - attachments 27, 28 & 35 Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler 15:00 beam is back 15:05 AIDA writing to file R14_1108 ADC data items - attachment 36 15:27 system wide checks ok except WR decoder status Base Current Difference aida05 fault 0x500 : 0x56c : 108 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 15:30 temperatures ok - attachment 37 stats ok - attachment 38 bias/leakage currents ok - attachment 39 R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40
	335	Thu May 20 01:00:30 2021	Muneerah	Thu May 20 00:00-08:00
02:00 System check The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA! The rates on ucesb got back to work by around 01:12 Deat Time check ok R14_870, attachment 1 Statistics are ok attachment 2 Temperatures are ok attacment 3 Voltages are ok attachment 4 system wide check All ok except ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbite 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 8 8 5 0 6 1 2 2 2 7 : 37296 aida02 : 19 8 11 2 3 4 2 2 3 3 6 : 36412 aida03 : 14 6 3 4 1 3 1 3 3 3 6 : 36312 aida04 : 25 12 12 4 3 4 1 4 3 3 6 : 37316 aida05 : 4 8 3 4 1 2 2 4 3 3 6 : 36928 aida06 : 42 12 8 7 2 3 3 4 3 3 6 : 37736 aida07 : 24 6 7 3 3 3 1 3 2 2 7 : 37536 aida08 : 10 15 7 2 1 4 2 3 3 3 6 : 36752 aida09 : 29 13 6 6 4 3 3 3 3 3 6 : 37244 aida10 : 18 12 7 2 2 3 3 2 2 4 6 : 37464 aida11 : 20 8 0 5 5 4 2 3 3 3 6 : 36976 aida12 : 7 2 4 3 2 3 2 2 4 3 6 : 37068 aida13 : 2 5 5 4 2 3 2 3 3 3 6 : 36608 aida14 : 9 13 8 2 2 2 3 3 2 4 6 : 37836 aida15 : 24 12 12 1 0 3 2 3 3 3 6 : 36640 aida16 : 12 9 9 2 1 3 2 4 3 3 6 : 37128 6:45 System check Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9 System wide check are all ok except: ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 11 3 5 0 6 1 2 3 2 7 : 38264 aida02 : 14 9 9 2 3 3 3 2 3 3 6 : 36496 aida03 : 13 7 3 5 2 3 2 3 3 3 6 : 36668 aida04 : 30 15 13 4 3 4 1 3 4 3 6 : 37888 aida05 : 17 9 3 3 1 2 2 4 3 3 6 : 36956 aida06 : 40 16 5 6 2 3 2 3 2 4 6 : 37936 aida07 : 3 2 3 2 3 4 2 2 2 2 7 : 37196 aida08 : 17 10 6 2 1 3 3 3 3 3 6 : 36852 aida09 : 29 11 7 6 3 3 2 4 3 3 6 : 37436 aida10 : 1 10 5 1 1 3 3 2 2 4 6 : 37252 aida11 : 30 9 0 5 4 4 2 3 3 3 6 : 36960 aida12 : 19 10 5 3 2 3 2 2 4 3 6 : 37196 aida13 : 22 9 6 3 2 3 3 3 3 3 6 : 36960 aida14 : 19 6 9 2 2 3 3 2 2 4 6 : 37452 aida15 : 6 8 11 2 1 3 2 3 3 3 6 : 36616 aida16 : 19 14 5 3 1 3 1 4 3 3 6 : 36908 07:33 system check Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13 System wide check all ok except ADC all failed White rabbit 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 9 4 6 0 5 1 2 2 2 7 : 37152 aida02 : 19 6 13 1 3 3 4 2 3 3 6 : 36780 aida03 : 23 5 6 4 2 3 1 3 3 3 6 : 36452 aida04 : 24 11 14 3 3 4 2 3 3 3 6 : 37048 aida05 : 17 3 6 3 1 2 2 3 3 3 6 : 36444 aida06 : 36 15 4 9 2 4 2 4 3 3 6 : 37608 aida07 : 20 8 5 3 3 4 2 2 2 2 7 : 37376 aida08 : 21 11 10 2 1 4 2 2 2 4 6 : 37324 aida09 : 27 14 7 6 3 3 2 4 3 3 6 : 37452 aida10 : 25 9 5 2 2 3 3 2 2 4 6 : 37436 aida11 : 30 14 2 4 5 3 2 3 3 3 6 : 36936 aida12 : 22 3 5 3 2 3 2 3 3 3 6 : 36640 aida13 : 10 7 1 4 2 3 3 2 3 3 6 : 36336 aida14 : 13 9 12 2 3 3 2 3 2 4 6 : 37820 aida15 : 20 8 11 3 0 3 2 3 3 3 6 : 36640 aida16 : 23 6 6 2 0 2 1 3 4 3 6 : 37164
	334	Wed May 19 20:02:27 2021	NH	AIDA Dead Time Analysis
Some plots of AIDA dead time (From PAUSE/RESUME data items) against time. The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead. Beam spills are indicated by the red bars (1 = on spill) Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates . The much quieter p+n strips show a similar pattern but much less frequently. Deadtime does not always sync up between FEE64s it seems. IN all case the pause/resume interval seems to only last for less than 100 ms.
	333	Wed May 19 18:50:01 2021	LS	Wednesday 19th May 20:00 to 24:00
20.00 Stats okay (Fig 1) Analysed R14_762 deadtimes are okay (Fig 2) 20.30 Stats okay (Fig 3) Analysed R14_771 deadtimes are okay (Fig 4) 21.00 Stats okay (Fig 5) FEE temps okay (Fig 6) Bias (Fig 7) Rate spectra (Fig 8) System wide checks okay except white rabbit 1 fail: Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~2.8M items/s Tape server ~10MB/s Analysed R14_781 deadtimes are okay (Fig 9) 21.30 Stats okay (Fig 10) Analysed R14_789 deadtimes are okay (Fig 11) 22.00 Stats okay (Fig 12) Analysed R14_799 deadtimes are okay (Fig 13) 22.30 Stats okay (Fig 14) Analysed R14_807 deadtimes are okay (Fig 15) 23.00 Stats okay (Fig 16) FEE temps okay (Fig 17) Bias (Fig 18) System wide checks okay except white rabbit 1 fail: Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~3M items/s Tape server ~10MB/s Analysed R14_819 deadtimes are okay (Fig 19) 23.30 Stats okay (Fig 20) Analysed R14_827 deadtimes are okay (Fig 21)
	332	Wed May 19 14:54:36 2021	CB	Wednesday 19 May 16:00 - 20:00
15:55 Took over from OH 16:00 Stats OK - attach 1 Temps OK - attach 2 Bias OK - attach 3 System wide checks All OK, except ADC all fail (as before) and WR Base Current Difference aida05 fault 0x500 : 0x567 : 103 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Analysed R14_691. Attach 4. Dead time <5% 16:34 MBS changes file. 17:00 Stats OK - attach 5 Temps OK - as before System wide checks as before Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 17:55 Stats OK - attach 6 Temps OK - as before Leakage current approaching 10 uA - attach 7 System-wide checks - as before except Base Current Difference aida05 fault 0x500 : 0x568 : 104 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:01 MBS switches to new file 18:52 Stats OK - attach 8 Temps OK - as before System-wide checks as before Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9 19:36 Stats OK - attach 10 Bias - attach 11 19:48 Shift handed over to LS
	331	Wed May 19 11:24:55 2021	RDP	Monday 19 April 12 noon shift
System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed. FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida07 failed FEE64 module aida09 failed Calibration test result: Passed 10, Failed 2 Base Current Difference aida07 fault 0xd052 : 0xd056 : 4 White Rabbit error counter test result: Passed 11, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x16 : 22 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 6 2 0 5 5 1 2 3 3 6 : 36120 aida02 : 25 10 0 4 3 5 2 3 3 3 6 : 36980 aida03 : 22 12 5 2 1 3 2 3 3 3 6 : 36616 aida04 : 16 3 3 3 4 4 4 2 3 3 6 : 36840 aida05 : 14 5 6 2 3 4 4 2 3 3 6 : 36800 aida06 : 30 10 6 3 3 4 2 3 3 3 6 : 36936 aida07 : 27 11 1 0 0 3 2 3 3 3 6 : 36436 aida08 : 38 7 7 3 3 2 2 3 3 3 6 : 36704 aida09 : 4 6 4 2 4 4 1 4 3 3 6 : 37056 aida10 : 21 10 3 3 1 5 1 3 3 3 6 : 36596 aida11 : 24 8 2 1 4 3 2 4 2 3 6 : 36192 aida12 : 31 12 4 3 5 5 2 2 3 3 6 : 36668 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1025 Last changed Mon Apr 19 10:32:22 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021 There should be a separate elog note about changing one of the bias voltages. All histograms zeroed at 16:13. 16.20 DAQ continues file NULL/R33_156 aida09 HEC fast comparator 0x2->0x1 DSSSD#3 bias -100->-120V
	330	Wed May 19 09:12:52 2021	Muneerah, Betool, LS	17th May 00:00-12:00
Auther: Muneerah and Betool 02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok Statistics ok - 210517_0202_Stats Temperatures ok - 210517_0202_Temp Bias and leakage ok - 210517_0200_Bias 02:05 System wide checks: Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida06 failed FEE64 module aida10 failed Calibration test result: Passed 12, Failed 4 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x1552 : 0x155d : 11 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 5 8 3 3 3 5 2 2 2 3 7 : 39460 aida02 : 22 4 5 0 3 2 2 3 2 3 7 : 39560 aida03 : 21 10 2 1 1 3 2 4 2 3 7 : 40100 aida04 : 18 7 2 0 3 3 3 2 2 3 7 : 39392 aida05 : 16 6 5 1 2 4 3 2 2 3 7 : 39520 aida06 : 18 6 6 2 2 2 2 3 2 3 7 : 39576 aida07 : 12 8 6 1 2 3 1 3 2 3 7 : 39408 aida08 : 13 11 4 0 1 3 3 2 2 3 7 : 39308 aida09 : 10 8 4 2 3 4 2 3 2 3 7 : 39848 aida10 : 16 8 2 1 2 4 2 3 1 4 7 : 40768 aida11 : 26 10 2 2 1 2 2 3 2 3 7 : 39512 aida12 : 16 9 2 1 2 4 2 2 2 3 7 : 39240 aida13 : 15 9 3 2 2 4 2 2 2 3 7 : 39284 aida14 : 25 10 2 3 3 3 2 3 2 3 7 : 39796 aida15 : 4 2 2 1 1 3 2 2 2 3 7 : 38944 aida16 : 16 7 3 3 1 3 3 2 2 3 7 : 39368 04:00 Statistics ok - 210517_0403_Stats Temperatures ok - 210517_0403_Temp Bias and leakage ok - 210517_0400_Bias 04:05 System wide checks: Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida06 failed FEE64 module aida10 failed Calibration test result: Passed 12, Failed 4 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x1552 : 0x1560 : 14 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 7 5 2 3 5 2 2 2 3 7 : 39492 aida02 : 23 6 7 2 1 3 2 3 2 3 7 : 39676 aida03 : 17 9 3 2 0 3 2 3 2 3 7 : 39548 aida04 : 16 4 1 2 1 3 3 3 2 3 7 : 39792 aida05 : 22 7 3 3 1 4 3 3 2 3 7 : 40032 aida06 : 18 6 7 3 1 3 2 3 2 3 7 : 39688 aida07 : 18 7 4 1 2 4 1 4 2 3 7 : 40032 aida08 : 19 5 4 1 1 3 3 3 2 3 7 : 39828 aida09 : 10 6 7 1 3 4 2 2 2 3 7 : 39336 aida10 : 12 10 2 3 1 4 2 3 1 4 7 : 40768 aida11 : 10 9 4 3 2 1 2 3 2 3 7 : 39440 aida12 : 8 7 6 1 2 4 2 3 2 3 7 : 39768 aida13 : 11 2 6 1 2 3 2 3 2 3 7 : 39612 aida14 : 27 7 4 2 3 3 2 3 2 3 7 : 39780 aida15 : 4 2 3 1 1 3 2 2 2 3 7 : 38960 aida16 : 18 8 4 2 1 4 2 2 2 3 7 : 39240 06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA 06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again. 07:01 it is working fine. 07:40 Screen shots after the restart Statistics ok - 210517_0742_Stats Temperatures ok - 210517_0743_Temp Bias and leakage ok - 210517_0740_Bias System wide checks FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x61da : 0x61dd : 3 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 6 3 1 1 3 1 3 3 3 7 : 40336 aida02 : 11 6 0 1 1 5 2 4 2 3 7 : 40252 aida03 : 0 5 2 3 3 3 3 4 2 3 7 : 40424 aida04 : 2 4 2 1 2 4 3 3 2 3 7 : 39912 aida05 : 11 7 5 1 2 2 4 2 3 3 7 : 40532 aida06 : 20 6 0 2 3 2 1 3 3 3 7 : 40320 aida07 : 15 5 1 2 2 3 2 2 3 3 7 : 40116 aida08 : 2 3 4 1 3 3 2 2 3 3 7 : 40128 aida09 : 18 5 1 3 2 4 2 2 3 3 7 : 40288 aida10 : 11 6 3 4 2 3 3 2 2 4 7 : 41484 aida11 : 4 4 5 2 2 2 3 3 2 3 7 : 39744 aida12 : 18 7 2 3 2 2 4 3 2 3 7 : 40064 aida13 : 2 2 1 3 0 3 2 4 2 3 7 : 39944 aida14 : 16 4 5 3 3 3 3 2 2 4 7 : 41552 aida15 : 9 3 1 1 2 3 2 3 3 3 7 : 40556 aida16 : 16 5 1 2 1 2 3 4 2 3 7 : 40184 Date:17/05/2021 Elog for shift: 08.00 to 12.00 Author: Lewis Sexton Screenshots saved to: /tmp/Elog_210517_0800_1200 *** 08.30 * Checked R12_34 through analyser, dead time seems okay: * Timestamp elapsed time: 232.154 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.021 0.000 1 2.781 0.000 2 0.849 0.000 3 5.686 0.000 4 0.000 25.816 5 0.017 0.000 6 0.177 0.000 7 1.349 0.000 8 1.791 0.000 9 0.000 91.660 10 6.194 0.000 11 0.320 0.000 12 1.425 0.000 13 0.000 72.999 14 0.052 0.000 15 1.729 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0833_Stats.png) Merger rate ~3M items/s Tape server ~12 MB/s *** 09.00 * Checked R12_48 through analyser, dead time seems okay: * Timestamp elapsed time: 170.459 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.196 0.000 1 6.294 0.000 2 0.992 0.000 3 8.944 0.000 4 0.024 0.000 5 0.001 0.000 6 0.605 0.000 7 2.096 0.000 8 1.654 0.000 9 0.000 31.395 10 6.175 0.000 11 2.524 0.000 12 2.076 0.000 13 0.000 10.337 14 0.103 0.000 15 2.203 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0902_Stats.png) *** 09.30 * AIDA crash aida05 displaying 0 in stats Power cycle performed with help of TD Writing to R13 now MBS relay restared AIDA back up at 10.20 Ran analyser for R13_16 * Timestamp elapsed time: 168.059 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.047 0.000 1 2.282 0.000 2 0.080 0.000 3 0.806 0.000 4 0.000 0.000 5 0.260 0.000 6 0.030 0.000 7 1.471 0.000 8 67.467 0.000 9 80.552 0.000 10 0.057 0.000 11 0.000 11.671 12 1.191 0.000 13 0.080 0.000 14 0.091 0.000 15 0.202 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc) Ran analyser for R13_21 for comparision * Timestamp elapsed time: 149.082 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 3.580 0.000 2 0.000 0.000 3 2.800 0.000 4 0.000 0.000 5 0.514 0.000 6 0.140 0.000 7 1.372 0.000 8 55.544 0.000 9 66.588 0.000 10 1.065 0.000 11 0.000 10.010 12 0.681 0.000 13 0.044 0.000 14 0.070 0.000 15 0.400 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png) Merger rate ~5M items/s Tape server ~17 MB/s Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision * Timestamp elapsed time: 122.147 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.027 0.000 1 19.945 0.000 2 0.601 0.000 3 21.555 0.000 4 0.042 0.000 5 3.226 0.000 6 1.211 0.000 7 9.674 0.000 8 52.817 0.000 9 54.972 0.000 10 1.848 0.000 11 0.008 0.000 12 2.034 0.000 13 0.278 0.000 14 0.018 0.000 15 0.634 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 high deatime now also seen in aida02 and aida04 Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs Back running as of 10.45 Analyser of R14_15 to double check * Timestamp elapsed time: 205.006 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 40.176 1 0.438 0.000 2 0.199 0.000 3 2.927 0.000 4 0.000 140.347 5 1.282 0.000 6 0.150 0.000 7 2.097 0.000 8 1.673 0.000 9 0.000 87.749 10 1.150 0.000 11 0.489 0.000 12 0.374 0.000 13 0.048 20.995 14 0.251 0.000 15 2.441 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs Date:17/05/2021 Elog for shift: 08.00 to 12.00 Author: Lewis Sexton Screenshots saved to: /tmp/Elog_210517_0800_1200 * 08.30 * Checked R12_34 through analyser, dead time seems okay: * Timestamp elapsed time: 232.154 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.021 0.000 1 2.781 0.000 2 0.849 0.000 3 5.686 0.000 4 0.000 25.816 5 0.017 0.000 6 0.177 0.000 7 1.349 0.000 8 1.791 0.000 9 0.000 91.660 10 6.194 0.000 11 0.320 0.000 12 1.425 0.000 13 0.000 72.999 14 0.052 0.000 15 1.729 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0833_Stats.png) Merger rate ~3M items/s Tape server ~12 MB/s *** 09.00 * Checked R12_48 through analyser, dead time seems okay: * Timestamp elapsed time: 170.459 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.196 0.000 1 6.294 0.000 2 0.992 0.000 3 8.944 0.000 4 0.024 0.000 5 0.001 0.000 6 0.605 0.000 7 2.096 0.000 8 1.654 0.000 9 0.000 31.395 10 6.175 0.000 11 2.524 0.000 12 2.076 0.000 13 0.000 10.337 14 0.103 0.000 15 2.203 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0902_Stats.png) *** 09.30 * AIDA crash aida05 displaying 0 in stats Power cycle performed with help of TD Writing to R13 now MBS relay restared AIDA back up at 10.20 Ran analyser for R13_16 * Timestamp elapsed time: 168.059 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.047 0.000 1 2.282 0.000 2 0.080 0.000 3 0.806 0.000 4 0.000 0.000 5 0.260 0.000 6 0.030 0.000 7 1.471 0.000 8 67.467 0.000 9 80.552 0.000 10 0.057 0.000 11 0.000 11.671 12 1.191 0.000 13 0.080 0.000 14 0.091 0.000 15 0.202 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc) Ran analyser for R13_21 for comparision * Timestamp elapsed time: 149.082 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 3.580 0.000 2 0.000 0.000 3 2.800 0.000 4 0.000 0.000 5 0.514 0.000 6 0.140 0.000 7 1.372 0.000 8 55.544 0.000 9 66.588 0.000 10 1.065 0.000 11 0.000 10.010 12 0.681 0.000 13 0.044 0.000 14 0.070 0.000 15 0.400 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png) Merger rate ~5M items/s Tape server ~17 MB/s Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision * Timestamp elapsed time: 122.147 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.027 0.000 1 19.945 0.000 2 0.601 0.000 3 21.555 0.000 4 0.042 0.000 5 3.226 0.000 6 1.211 0.000 7 9.674 0.000 8 52.817 0.000 9 54.972 0.000 10 1.848 0.000 11 0.008 0.000 12 2.034 0.000 13 0.278 0.000 14 0.018 0.000 15 0.634 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 high deatime now also seen in aida02 and aida04 Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs Back running as of 10.45 Analyser of R14_15 to double check * Timestamp elapsed time: 205.006 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 40.176 1 0.438 0.000 2 0.199 0.000 3 2.927 0.000 4 0.000 140.347 5 1.282 0.000 6 0.150 0.000 7 2.097 0.000 8 1.673 0.000 9 0.000 87.749 10 1.150 0.000 11 0.489 0.000 12 0.374 0.000 13 0.048 20.995 14 0.251 0.000 15 2.441 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs 12:52 Check FPGA Timestamp errors gives following error message: Got the error Server Internal Error while trying to obtain /AIDA/Check/Check.tml. can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]" Tcl Call Trace 6: DocSubstSystemFile sock8 error 500 can't\ read\ &quot\;TS_Base_Collected&quot\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n&quot\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}&quot\;\n\ \ \ \ (procedure\ &quot\;do_TS_errors&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;do_TS_errors\ &quot\;\n\ \ \ \ (&quot\;TSERRORS&quot\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...&quot\;\n\ \ \ \ (procedure\ &quot\;do_click&quot\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n&quot\;do_click&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ &quot\;&quot\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ &quot\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...&quot\;\n\ \ \ \ (file\ &quot\;Check.tcl&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;source\ Check.tcl&quot\;\n\ \ \ \ (in\ namespace\ eval\ &quot\;::Check&quot\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n&quot\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n&lt\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n&gt\;...&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ #0\ [list\ subst\ \$script\]&quot\;\n\ \ \ \ (procedure\ &quot\;SubstFile&quot\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n&quot\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]&quot\;\n\ \ \ \ (procedure\ &quot\;Subst_File&quot\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n&quot\;Subst_File\ \$template\ \$interp&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_application/x-tcl-template&quot\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n&quot\;\$cmd\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_Handle&quot\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n&quot\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;DocDomain&quot\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n&quot\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml&quot\;\n\ \ \ \ (&quot\;eval&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]&quot\; 5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" 4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE 3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {} 2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml} 1: HttpdReadPostGlobal sock8 Httpds Dead time check: [npg@aidas-gsi S496]$ ~/analyser/analyser R14_24 * TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 252995858 ( 451517.9 Hz) Other data format: 8924143 ( 15926.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 85 ( 0.2 Hz) RESUME: 85 ( 0.2 Hz) SYNC100: 34087 ( 60.8 Hz) WR48-63: 34087 ( 60.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 8855799 ( 15804.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 560.323 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 349.960 1 0.002 0.000 2 0.000 5.974 3 1.105 0.000 4 0.000 447.454 5 0.000 2.809 6 0.000 16.727 7 0.137 0.000 8 0.090 0.000 9 0.000 429.897 10 0.461 0.000 11 0.018 0.000 12 0.000 8.985 13 0.000 466.311 14 0.000 15.095 15 0.346 0.000 13:24 Beam is off at the moment... 13:43 Changing spill structure from 1.5s/3.5s to 1s. 13:50 [npg@aidas-gsi S496]$ ~/analyser/analyser R14_29 * TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 251763810 ( 930342.1 Hz) Other data format: 10156190 ( 37530.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 205 ( 0.8 Hz) RESUME: 205 ( 0.8 Hz) SYNC100: 33009 ( 122.0 Hz) WR48-63: 33009 ( 122.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 10089762 ( 37284.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 527077 ( 1947.7 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) *** Timestamp elapsed time: 270.614 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.034 6.803 1 8.461 0.000 2 0.569 0.000 3 11.664 0.000 4 0.024 26.416 5 0.854 0.000 6 0.069 0.000 7 0.741 0.000 8 1.336 0.000 9 0.000 75.926 10 0.353 0.000 11 0.105 0.000 12 0.070 0.000 13 0.000 93.585 14 0.017 0.000 15 0.121 0.000 15:06 Spill structure back to 1.5 on 2s off.
	329	Wed May 19 09:09:27 2021	MS, JS, OH	May 16 08:00-24:00
16th May 08:00 - 12:00 shift Author: MS 08:00 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x7163 : 23 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0xa : 10 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 27 7 2 3 2 3 2 4 2 3 7 : 40228 aida02 : 3 8 3 2 2 2 2 4 2 3 7 : 39996 aida03 : 23 9 6 1 0 3 2 4 2 3 7 : 40100 aida04 : 34 27 17 7 2 4 3 3 2 2 7 : 38608 aida05 : 19 9 5 2 2 2 3 3 2 3 7 : 39844 aida06 : 5 5 2 1 0 4 2 4 2 3 7 : 40060 aida07 : 28 2 10 3 2 4 1 4 2 3 7 : 40192 aida08 : 19 5 5 1 2 5 1 3 2 3 7 : 39652 aida09 : 20 8 3 3 1 3 2 3 2 3 7 : 39648 aida10 : 27 10 0 2 2 2 2 3 1 4 7 : 40572 aida11 : 3 3 2 1 2 4 2 3 2 3 7 : 39652 aida12 : 16 7 11 2 3 4 2 2 2 3 7 : 39464 aida13 : 14 10 4 3 1 5 2 2 2 3 7 : 39400 aida14 : 21 9 10 1 1 2 2 3 1 4 7 : 40604 aida15 : 19 8 2 3 0 4 3 2 2 3 7 : 39436 aida16 : 24 5 8 3 3 4 2 2 2 3 7 : 39464 * Timestamp elapsed time: 225.065 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.038 0.000 1 9.479 0.000 2 0.195 0.000 3 5.921 0.000 4 0.000 11.742 5 0.036 0.000 6 0.013 0.000 7 0.498 0.000 8 0.436 0.000 9 0.000 107.300 10 2.787 0.000 11 0.905 0.000 12 0.831 0.000 13 0.000 55.939 14 0.080 0.000 15 0.267 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 10:00 FEE64 module aida06 global clocks failed, 6 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 14, Failed 2 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x7166 : 26 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0xa : 10 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 7 6 2 2 3 2 3 2 3 7 : 39716 aida02 : 8 4 3 2 2 3 2 3 2 3 7 : 39600 aida03 : 23 11 5 1 0 4 2 3 1 4 7 : 40740 aida04 : 42 25 16 7 2 4 4 3 2 2 7 : 38864 aida05 : 24 5 5 1 2 2 3 3 1 4 7 : 40824 aida06 : 11 4 4 1 1 4 2 4 2 3 7 : 40172 aida07 : 21 8 5 1 2 3 2 4 2 3 7 : 40196 aida08 : 15 5 6 1 1 4 2 4 2 3 7 : 40228 aida09 : 15 10 4 1 2 3 2 3 2 3 7 : 39660 aida10 : 23 8 2 2 2 2 2 3 1 4 7 : 40572 aida11 : 6 4 4 2 2 4 2 3 2 3 7 : 39736 aida12 : 18 8 8 1 4 4 3 2 2 3 7 : 39720 aida13 : 23 6 2 3 2 5 2 2 2 3 7 : 39436 aida14 : 29 3 11 1 1 2 2 3 1 4 7 : 40604 aida15 : 29 7 2 2 0 4 3 3 2 3 7 : 39948 aida16 : 2 3 6 2 3 4 2 2 2 3 7 : 39296 * Timestamp elapsed time: 225.065 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.038 0.000 1 9.479 0.000 2 0.195 0.000 3 5.921 0.000 4 0.000 11.742 5 0.036 0.000 6 0.013 0.000 7 0.498 0.000 8 0.436 0.000 9 0.000 107.300 10 2.787 0.000 11 0.905 0.000 12 0.831 0.000 13 0.000 55.939 14 0.080 0.000 15 0.267 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 12:00-16:00 16th May 12:00 - 16:00 shift Author: JS 11:57 Taking over from Magda. Running full checks. usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75% Current ok 06.410 uA 006.835 uA Stats good 1Statistics aidas-gsi(6).png Temps ok 1Temperature and status scan aidas-gsi(6).png Analysis ok R7_385. Dead time FEE1 a little hight 6% PAUSE: 166 RESUME: 166 *** Timestamp elapsed time: 196.305 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.044 0.000 1 12.405 0.000 2 0.807 0.000 3 7.260 0.000 4 0.014 0.000 5 0.458 0.000 6 0.027 0.000 7 0.497 0.000 8 0.723 0.000 9 0.000 88.857 10 6.189 0.000 11 1.443 0.000 12 0.474 0.000 13 0.000 35.565 14 0.000 0.000 15 0.147 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x716e : 34 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Base Current Difference aida05 fault 0x0 : 0xa : 10 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 7 4 2 2 3 2 3 1 4 7 : 40712 aida02 : 25 10 7 1 1 4 1 3 2 3 7 : 39556 aida03 : 22 10 5 1 0 4 2 4 2 3 7 : 40216 aida04 : 40 24 17 7 2 5 3 3 2 2 7 : 38736 aida05 : 4 8 3 0 1 3 3 3 1 4 7 : 40768 aida06 : 21 6 6 2 2 3 2 4 2 3 7 : 40228 aida07 : 25 4 6 3 2 3 2 4 2 3 7 : 40260 aida08 : 19 9 4 1 1 4 2 4 2 3 7 : 40244 aida09 : 16 9 4 2 2 3 2 3 2 3 7 : 39688 aida10 : 19 10 5 1 2 2 2 4 1 4 7 : 41100 aida11 : 21 10 2 1 2 3 3 3 2 3 7 : 39908 aida12 : 25 7 7 2 2 3 2 3 2 3 7 : 39756 aida13 : 13 7 3 4 2 5 2 3 2 3 7 : 39964 aida14 : 21 7 9 2 1 2 2 4 1 4 7 : 41116 aida15 : 23 6 2 3 0 4 3 3 2 3 7 : 39948 aida16 : 9 11 10 2 3 4 2 2 2 3 7 : 39452 Tom says Aida09 clock fail is ok as its status bit is "6". The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken. 12:35 - usbec ok. Current ok Stats good Temps ok Analysis ok R7_395. Dead time FEE1 still high 6.6% 13:33 - usbec ok. Max implants ~ 1.8kHz Current ok 006.850 uA 007.250 uA Stats - Aida11 runing low < 5k was ~20k overnight Temps ok Analysis R7_415. Dead time FEE1 & FEE10 high 10% 14:00 usbec ok - ucesb1.png Current ok - bias1.png Stast - Aida11 low - 14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n. 15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool. 18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00. During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise. The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs. We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy. During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips. 18:08 System wide checks all ok - bar some ADC but waveforms disabled Statistics ok - 210516_1809_Stats Temperatures ok - 210516_1809_Temp Bias and leakage ok - 210516_1810_Bias 18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time 18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side. 19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves. Now have around 4.2TB left which will provide around 80 hours of writing 20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not. 20:57 System wide checks. Clock ok Base Current Difference aida05 fault 0x1552 : 0x1556 : 4 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Statistics ok - 210516_2056_Stats Temp ok - 210516_2058_Temp Bias and leakage current ok - 210516_2058_Bias 23:16 System wide checks: Clock still ok Base Current Difference aida05 fault 0x1552 : 0x155a : 8 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Statistics - 210516_2315_Stats Temperature - 210516_2316_Temp Bias and leakage current ok - 210516_231
	328	Wed May 19 09:06:58 2021	CA	May 16th 00:00 - 08:00
16th May 2021 - 00:00 - 08:00 Shift Author: CA 00:15 System wide checks; FEE64 module aida06 global clocks failed, 6 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 14, Failed 2 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x714c : 0x7154 : 8 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 10 3 1 2 3 2 3 2 3 7 : 39660 aida02 : 29 5 3 1 1 3 2 3 2 3 7 : 39596 aida03 : 15 7 1 1 2 3 2 3 2 3 7 : 39588 aida04 : 44 26 19 5 2 4 3 3 2 2 7 : 38608 aida05 : 20 10 3 0 1 4 2 2 3 3 7 : 40208 aida06 : 27 6 2 1 2 4 2 4 2 3 7 : 40284 aida07 : 25 5 3 1 2 3 2 3 2 3 7 : 39644 aida08 : 23 8 2 3 1 4 2 3 2 3 7 : 39772 aida09 : 23 10 3 2 3 4 1 3 2 3 7 : 39644 aida10 : 18 10 3 2 3 2 2 2 2 4 7 : 41160 aida11 : 16 7 3 2 1 2 2 3 2 3 7 : 39464 aida12 : 21 4 7 1 3 3 3 3 2 3 7 : 40004 aida13 : 25 6 3 2 1 3 1 4 2 3 7 : 39876 aida14 : 20 8 10 1 1 2 2 2 2 4 7 : 41104 aida15 : 26 6 3 3 0 4 3 2 2 3 7 : 39464 aida16 : 9 8 2 4 1 3 1 4 2 3 7 : 39876 00:30 Stats ok - 210516_0030_stats.png Temps ok - 210516_0030_temps.png Bias ok - 210516_0030_bias.png 00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt 02:21 system wide checks as above, except; Base Current Difference aida05 fault 0x714c : 0x7159 : 13 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x7 : 7 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 02:28 Stats ok - 210516_0230_stats.png Temps ok - 210516_0230_temps.png Bias ok - 210516_0230_bias.png 02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt 04:36 system wide checks same as above, except; Base Current Difference aida05 fault 0x714c : 0x715c : 16 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x9 : 9 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 04:37 Stats ok - 210516_0430_stats.png Temps ok - 210516_0430_temps.png Bias ok - 210516_0430_bias.png 04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt 06:38 system wide checks same as before, except; Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x7161 : 21 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 06:40 Stats ok - 210516_0640_stats.png Temps ok - 210516_0640_temps.png Bias ok - 210516_0640_bias.png 06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt
	327	Wed May 19 06:59:33 2021	OH	Wednesday 19th May 08:00 - 16:00
08:00 OH Takes over for CA Thresholds 0xc for p+n and 0x19 for n+n ASICs 1-3 and 0x64 for n+nASIC4 PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 Max average dead time in R14_586 3.2% which is 9.6% on spill Stable from yesterday 10:02 System wide checks. All ok expect WR Base Current Difference aida05 fault 0x500 : 0x554 : 84 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics - Attachment 1 Temp - attachment 2 Bias and leakage currents ok - Attachment 3 11:17 A new run was started as the microspill has been changed. I believe to optimise FRS deadtime. 11:45 They are still working on it 12:05 Beam off for a bit while they optimise spill 12:15 System wide checks ok expect WR Base Current Difference aida05 fault 0x500 : 0x567 : 103 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics - Attachment 4 Temp - Attachment 5 Bias - Attachment 6 13:11 Magda has said that the changes to the spill structure only gained them a few percent in dead time Further changes could be done, but would take at least two hours 13:19 Dead time following the current spill changes R14_648 R14_649 R14_650 R14_651 Sum DT (S) Avergage DT (%) On Spill DT (%) FEE 244.823 252.168 247.155 240.421 984.567 0 0.063 0.025 0.105 0.096 0.289 0.029353005 0.088059015 1 3.981 4.907 4.17 2.484 15.542 1.578561947 4.73568584 2 0.01 0 0.121 0.779 0.91 0.092426417 0.277279251 3 3.625 2.972 2.766 3.335 12.698 1.289704002 3.869112006 4 0.088 0.1 0.12 0.136 0.444 0.045095966 0.135287898 5 9.392 5.798 9.835 8.01 33.035 3.355282068 10.0658462 6 0.153 0.075 0.01 0.517 0.755 0.076683456 0.230050367 7 7.4 7.565 7.703 6.415 29.083 2.953887343 8.86166203 8 0.221 0.606 1.321 0.539 2.687 0.272911849 0.818735546 9 0 0 0 0 0 0 0 10 1.381 0.368 1.196 1.091 4.036 0.409926394 1.229779182 11 0.066 0.025 0.039 0.017 0.147 0.014930421 0.044791264 12 0.166 0.041 0.062 0.024 0.293 0.029759275 0.089277825 13 0 0 0 0 0 0 0 14 0.71 0 0.005 0.074 0.789 0.08013675 0.240410251 15 0.239 0.17 0.208 0.286 0.903 0.091715444 0.275146333 16 0 0 0 0 0 0 0 14:03 System wide checks ok - WR same difference as before Statistics - attachment 7 Temperatures - attachment 8 Bias and leakage currents - attachment 9
	326	Tue May 18 23:11:40 2021	CA	Wednesday May 19th 00:00 - 08:00
00:00 CA takes over LS has performed usual checks, see previous Elog entry - all ok 00:30 stats ok R14_470 deadtime at 4.4% 02:10 system wide checks ok except: all ADC fail calibration (expected, no waveforms) WR decoder status Base Current Difference aida05 fault 0x500 : 0x52f : 47 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 02:12 FEE64 Temps ok - attachment 1 Stats ok - attachment 2 detector bias / leakage currents ok - attachment 3 dead-times ok - R14_497 max 4% - attachment 4 03:27 DAQ continues ok - writing to file R14_517 04:07 system wide checks ok except: all ADC fail calibration (expected, no waveforms) WR decoder status Base Current Difference aida05 fault 0x500 : 0x537 : 55 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FEE64 Temps ok - attachment 6 Stats ok - attachment 5 detector bias / leakage currents ok - attachment 7 dead-times ok - R14_526 max 4% - attachment 8 05:27 bad timestamp error in merger terminal - attachment 9 06:05 all system wide checks ok except; all ADC fail calibration (expected, no waveforms) WR decoder status Base Current Difference aida05 fault 0x500 : 0x53d : 61 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 06:07 FEE64 Temps ok - attachment 10 Stats ok - attachment 11 detector bias / leakage currents ok - attachment 12 dead-times ok - R14_558 max 3.6% - attachment 13 07:01 another bad timestamp error in merger terminal - attachment 14
	325	Tue May 18 18:47:25 2021	LS	Tuesday 18th May 20:00 - 24:00
19.50 Stats okay (Fig 1) FEE temps okay (Fig 2) Bias okay (Fig 3) System wide checks okay except: Base Current Difference aida05 fault 0x500 : 0x520 : 32 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~1.8M items/s Tape server ~7MB/s Analysed R14_399, deadtimes are okay (Fig 4) 20.30 Stats okay (Fig 5) Analysed R14_407, deadtimes are okay (Fig 6) 21.00 Stats okay (Fig 7) Analysed R14_415, deadtimes are okay (Fig 8) 21.30 Stats okay (Fig 9) Analysed R14_422, deadtimes are okay (Fig 10) 22.00 Stats okay (Fig 11) FEE temps okay (Fig 12) Bias okay (Fig 13) System wide checks okay except: Base Current Difference aida05 fault 0x500 : 0x524 : 36 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~2.3M items/s Tape server ~8MB/s Analysed R14_430, deadtimes are okay (Fig 14) 22.30 Stats okay (Fig 15) Analysed R14_438, deadtimes are okay (Fig 16) FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5% 23.00 Stats okay (Fig 17) Analysed R14_445, deadtimes are okay (Fig 18) 23.30 Stats okay (Fig 19) Analysed R14_454, deadtimes are okay (Fig 20) 24.00 Stats okay (Fig 21) FEE temps okay (Fig 22) Bias okay (Fig 23) System wide checks okay except: Base Current Difference aida05 fault 0x500 : 0x529 : 41 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR The difference has increased over time (not sure if an issue) Merger rate ~3M items/s Tape server ~8MB/s Analysed R14_462, deadtimes are okay (Fig 24)
	324	Tue May 18 14:57:43 2021	CB	Tuesday 18th May 16:00 - 20:00
15:58 Took shift over from OH and PP Stats OK - attach 1 Temps OK - attach 2 Bias OK - attach 3 System-wide checks Clock - all pass ADC - all fail (as before) Base Current Difference aida05 fault 0x500 : 0x510 : 16 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA - all pass Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 8 7 4 1 4 2 2 2 2 7 : 37320 aida02 : 12 9 4 2 4 2 3 3 3 3 6 : 36856 aida03 : 19 6 7 2 2 2 2 4 3 3 6 : 37036 aida04 : 18 9 10 6 1 4 1 4 3 3 6 : 37168 aida05 : 17 12 2 1 2 3 3 3 3 3 6 : 36836 aida06 : 19 8 7 7 2 4 2 4 3 3 6 : 37468 aida07 : 28 5 4 1 2 4 2 3 2 2 7 : 37752 aida08 : 17 5 4 2 1 2 2 3 3 3 6 : 36396 aida09 : 21 10 4 5 4 4 3 3 3 3 6 : 37252 aida10 : 15 13 4 2 2 2 3 2 2 4 6 : 37284 aida11 : 22 13 3 3 4 3 1 4 3 3 6 : 37072 aida12 : 2 8 1 3 1 3 2 3 3 3 6 : 36472 aida13 : 23 5 4 4 1 4 3 2 3 3 6 : 36484 aida14 : 19 7 4 3 1 3 2 4 2 4 6 : 38116 aida15 : 20 10 2 2 3 2 3 2 3 3 6 : 36288 aida16 : 29 6 1 2 0 3 1 4 3 3 6 : 36724 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Analysed R14_342. Attach 4. Dead time <5% 17:00 Stats OK - attach 5 Temps OK - as before Bias increasing quite quickly - attach 6 System-wide checks System-wide checks Clock - all pass ADC - all fail (as before) Base Current Difference aida05 fault 0x500 : 0x512 : 18 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA - all pass Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 28 7 8 4 1 4 2 2 2 2 7 : 37352 aida02 : 24 11 6 4 3 2 3 2 3 3 6 : 36440 aida03 : 17 11 5 2 1 2 2 3 3 3 6 : 36460 aida04 : 25 13 10 6 2 4 2 3 3 3 6 : 37036 aida05 : 23 7 3 2 2 3 3 3 3 3 6 : 36868 aida06 : 1 12 3 8 2 4 3 3 3 3 6 : 37140 aida07 : 22 12 1 1 2 4 2 2 2 2 7 : 37224 aida08 : 21 8 7 2 1 3 3 2 3 3 6 : 36356 aida09 : 23 7 4 6 4 3 2 4 3 3 6 : 37396 aida10 : 27 7 2 2 2 2 3 3 2 4 6 : 37764 aida11 : 12 5 3 5 4 3 1 4 3 3 6 : 37032 aida12 : 23 9 4 1 2 3 2 3 3 3 6 : 36612 aida13 : 10 12 6 3 1 4 3 2 3 3 6 : 36488 aida14 : 23 7 3 3 1 3 2 3 2 4 6 : 37604 aida15 : 22 6 6 4 2 2 3 2 3 3 6 : 36328 aida16 : 27 7 3 1 0 3 1 4 3 3 6 : 36724 18:00 Stats OK - attach 7 Temps OK - as before Bias - attach 8 System-wide checks - as before except Base Current Difference aida05 fault 0x500 : 0x517 : 23 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Analysed R14_371. Attach 9. Dead time below 5%. 18:35 MBS DAQ crashed 18:40 Stats OK - attach 9 Temps OK - as before Bias - attach 10 System-wide checks - as before except Base Current Difference aida05 fault 0x500 : 0x51a : 26 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:42 MBS DAQ restarted - new file being opened. 19:36 Stats OK - attach 11 Temps OK - attach 12 Bias - attach 13 System-wide checks Clock OK ADC all fail Base Current Difference aida05 fault 0x500 : 0x51e : 30 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA OK Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 24 11 5 4 1 4 2 3 2 2 7 : 37832 aida02 : 0 5 6 3 4 2 3 2 3 3 6 : 36328 aida03 : 15 8 5 3 2 2 1 3 4 3 6 : 37292 aida04 : 14 8 7 5 3 4 2 3 3 3 6 : 36936 aida05 : 31 6 1 1 2 3 3 3 2 4 6 : 37852 aida06 : 9 8 3 8 2 4 3 3 3 3 6 : 37140 aida07 : 21 7 2 1 2 4 1 3 2 2 7 : 37452 aida08 : 8 4 5 3 1 3 2 3 3 3 6 : 36528 aida09 : 24 7 3 5 4 3 2 4 3 3 6 : 37352 aida10 : 19 9 7 1 2 2 3 2 2 4 6 : 37284 aida11 : 27 5 0 5 4 3 1 4 3 3 6 : 37044 aida12 : 31 9 2 1 2 3 2 3 3 3 6 : 36612 aida13 : 21 10 4 3 1 4 3 2 3 3 6 : 36484 aida14 : 22 6 4 2 1 3 2 4 2 4 6 : 38088 aida15 : 21 9 3 4 2 2 3 2 3 3 6 : 36300 aida16 : 23 11 4 2 1 4 2 3 3 3 6 : 36724 19:43 Shift handed over to LS
	323	Tue May 18 06:50:53 2021	OH, PP	Tuesday 18th May 08:00 - 16:00
08:00 OH Takes over Statistics still look good from yesterday. Rate is still down. Dead time in FEEs 2 and 4 is 2.04% System wide checks clock ok Base Current Difference aida05 fault 0x4da : 0x500 : 38 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Baseline reset FPGA check ok Stats - attachment 1 Temp - attachment 2 Bias - attachment 3 08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips. I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4 The results of the test appear to indicate that it is the network link that we are saturating causing dead time *** Before After FEE DT (s) Dead time % DT (s) Dead time % 0 0.013 0.004184451 0.026 0.010025681 1 6.352 2.04458693 4.097 1.57981599 2 0.295 0.09495484 0.292 0.112596112 3 6.349 2.043621288 8.255 3.183153771 4 0.021 0.006759497 0.03 0.011568094 5 0.285 0.091736032 3.388 1.306423377 6 0.02 0.006437616 0.149 0.057454865 7 1.219 0.392372712 2.133 0.822491459 8 1.137 0.365978485 0.457 0.176220627 9 0 0 0 0 10 1.633 0.525631369 1.545 0.595756823 11 0.061 0.01963473 0.166 0.064010118 12 0.059 0.018990968 0.085 0.032776265 13 0 0 0 0 14 0.016 0.005150093 0.023 0.008868872 15 0.282 0.09077039 0.168 0.064781324 Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased 09:00 They are stopping the run to decrease the bias on the bPlast detector 09:25 No noticable change in AIDA They will now begin to change the spill structure. 09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change 10:21 Beam is back on AIDA 10:39 System wide checks. - ADC calibration check failed (this is anticipated because waveforms are off). - 1 Failed test in White Rabbit: Base Current Difference aida05 fault 0x500 : 0x506 : 6 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR - All other systems passed 11:00 Stats - attachment 4 Temp - attachment 5 Bias - attachment 6 12:46 System wide checks - ADC calibration check failed (this is anticipated because waveforms are off). - 1 Failed test in White Rabbit: Base Current Difference aida05 fault 0x500 : 0x50b : 11 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR - All other systems passed 14:36 System wide checks - ADC calibration check failed (this is anticipated because waveforms are off). - 1 Failed test in White Rabbit: Base Current Difference aida05 fault 0x500 : 0x50f : 15 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR - All other systems passed
	322	Mon May 17 23:17:16 2021	DSJ	Tuesday 18 May 0.00-8.00 shift
0.00 - Taking over from Lewis 00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2) 01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4) 01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6) 02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8) Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11) System Wide Checks: Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 0, Failed 16 Base Current Difference aida05 fault 0x4da : 0x4ee : 20 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Timestamp error counter test result: Passed 16, Failed 0 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 43 12 6 4 1 4 2 2 1 3 7 : 38444 aida02 : 21 12 4 4 2 3 3 3 3 3 6 : 36980 aida03 : 23 8 4 2 2 4 2 3 3 3 6 : 36764 aida04 : 34 12 10 5 1 4 1 4 3 3 6 : 37224 aida05 : 18 5 5 1 2 3 2 2 1 3 7 : 38112 aida06 : 40 12 5 7 2 4 2 4 3 3 6 : 37552 aida07 : 40 10 6 2 2 4 2 2 2 2 7 : 37392 aida08 : 19 11 3 3 1 4 3 3 3 3 6 : 36980 aida09 : 43 6 2 4 3 4 3 3 3 3 6 : 37180 aida10 : 32 14 10 3 4 1 2 4 1 3 7 : 39280 aida11 : 40 7 2 5 4 3 1 4 3 3 6 : 37144 aida12 : 37 14 13 2 2 3 2 4 3 3 6 : 37396 aida13 : 18 9 3 1 1 5 3 3 3 3 6 : 37024 aida14 : 36 13 6 4 2 3 2 3 1 3 7 : 38872 aida15 : 32 11 7 4 3 4 2 3 2 4 6 : 38024 aida16 : 26 9 4 1 0 3 1 4 3 3 6 : 36752 02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13) 03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15) 03.05 - No beam, problem with SIS, don't know how long will be off 03.14 - Beam is back 03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17) 04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19) Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22) System Wide Checks: Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 0, Failed 16 Base Current Difference aida05 fault 0x4da : 0x4f2 : 24 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Timestamp error counter test result: Passed 16, Failed 0 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 26 11 5 6 1 4 2 2 1 3 7 : 38416 aida02 : 16 9 5 4 2 4 2 2 3 3 6 : 36312 aida03 : 9 7 4 4 2 4 2 3 3 3 6 : 36764 aida04 : 36 11 8 4 3 4 1 3 4 3 6 : 37800 aida05 : 20 4 3 2 2 3 2 2 1 3 7 : 38112 aida06 : 22 13 10 7 2 4 3 3 4 3 6 : 38336 aida07 : 28 7 7 2 2 4 2 2 2 2 7 : 37336 aida08 : 2 3 4 4 1 3 2 3 3 3 6 : 36512 aida09 : 40 7 1 4 4 3 3 3 3 3 6 : 37096 aida10 : 26 9 12 2 3 2 2 4 1 3 7 : 39280 aida11 : 30 7 1 5 4 3 1 3 2 4 6 : 37600 aida12 : 15 10 6 4 2 3 2 4 3 3 6 : 37228 aida13 : 1 3 2 1 2 4 3 2 3 3 6 : 36316 aida14 : 34 12 7 4 2 3 2 3 1 3 7 : 38872 aida15 : 18 11 5 3 4 4 2 3 2 4 6 : 37968 aida16 : 22 7 4 2 0 3 1 4 3 3 6 : 36752 04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24) 05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26) 05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28) 06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30) Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33) System Wide Checks: Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 0, Failed 16 Base Current Difference aida05 fault 0x4da : 0x4f9 : 31 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Timestamp error counter test result: Passed 16, Failed 0 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 16 9 3 7 1 5 1 3 1 3 7 : 38744 aida02 : 18 3 5 2 3 3 3 2 3 3 6 : 36400 aida03 : 21 9 4 2 2 4 2 4 3 3 6 : 37276 aida04 : 32 12 7 3 3 4 1 3 4 3 6 : 37744 aida05 : 12 8 3 2 2 4 2 2 1 3 7 : 38240 aida06 : 28 14 4 8 2 4 3 3 3 3 6 : 37280 aida07 : 40 12 5 2 2 4 1 3 2 2 7 : 37648 aida08 : 13 8 4 4 1 3 2 3 3 3 6 : 36596 aida09 : 35 5 2 4 3 4 2 4 3 3 6 : 37396 aida10 : 6 10 4 1 1 3 2 2 1 3 7 : 38024 aida11 : 41 9 1 4 4 3 2 4 3 3 6 : 37372 aida12 : 29 11 9 4 2 3 2 4 3 3 6 : 37340 aida13 : 24 10 5 1 2 4 3 2 3 3 6 : 36512 aida14 : 34 16 5 4 2 3 2 3 1 3 7 : 38872 aida15 : 42 8 6 2 4 4 2 3 2 4 6 : 38024 aida16 : 8 9 6 1 0 3 1 4 3 3 6 : 36712 06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35) 07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37) 07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39)
	321	Mon May 17 19:05:23 2021	LS	Monday 17th May 20.00-24.00
20.30 Checked stats, seem okay (Fig 1) Ran R14_99 through analyser, dead times all good (Fig 2) Merger rate 1.8M items/s Tape server 6.4MB/s 21.00 Checked stats seem okay (Fig 3) Ran R14_106 through analyser, dead times all good (Fig 4) 21.30 Checked stats seem okay (Fig 5) Ran R14_112 through analyser, dead times all good (Fig 6) 22.00 Stats okay (Fig 7) FEE temps okay (Fig 8) Rate spectra (Fig 9) Leakage currents (Fig 10) System wide checks: Clock status all passed White rabbit 1 fail: Base Current Difference aida05 fault 0x4da : 0x4e7 : 13 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp all passed Merger rate 2.4M items/s Tape server 6.3MB/s Ran R14_119 through analyser, dead times all good (Fig 11) 22.30 Checked stats seem okay (Fig 12) Ran R14_123 through analyser, dead times all good (Fig 13) 23.00 Checked stats seem okay (Fig 14) Ran R14_129 through analyser, dead times all good (Fig 15) 23.30 Checked stats seem okay (Fig 16) Ran R14_135 through analyser, dead times all good (Fig 17) 24.00 Stats okay (Fig 18) FEE temps okay (Fig 19) Leakage currents (Fig 20) System wide checks: Clock status all passed White rabbit 1 fail: Base Current Difference aida05 fault 0x4da : 0x4eb : 17 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp all passed Merger rate 2.5M items/s Tape server 6.5MB/s Ran R14_142 through analyser, dead times all good (Fig 21)
	320	Mon May 17 15:14:35 2021	LJW	Monday 17th May 2021 16:00-20:00
16:25 Informed at shift handover that there has been no beam. Statistics - See attachment 1 Temperatures - See attachment 2 Bias & Leakage currents - See attachment 3 System Wide checks - All passed, EXCEPT: ADC Calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit decoder status: Base Current Difference aida05 fault 0x4da : 0x4dd : 3 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:00 Usecb scalars now showing DSSD implant rates. Statistics - See attachment 4 Temperatures - See attachment 5 Bias & Leakage currents - See attachment 6 System Wide checks - All passed, EXCEPT: ADC Calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit decoder status Base Current Difference aida05 fault 0x4da : 0x4e1 : 7 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx 19:45 Usecb scalars have continued to show DSSD implant rates since the last recorded elog check. Statistics - See attachment 7 Temperatures - See attachment 8 Bias & Leakage currents - See attachment 9 System Wide checks - All passed, EXCEPT: ADC Calibration: FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit decoder status: Base Current Difference aida05 fault 0x4da : 0x4e1 : 7 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes 19:55 - Beam now back again - DSSD implants showing
	319	Fri May 14 19:01:09 2021	JS	Friday 14th May 20:00-00:00
20:00 Taking over from CB 20:33 Doing full checks. Temps ok elog:319/8 Bias ok elog:319/1 Clock check - ok ADC Calc check - ok White Rabbit - Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc5 : 13 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 FPGA Timestamp errors - Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 Memory Info- Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 34 30 8 1 4 1 2 2 3 6 : 35772 aida02 : 37 30 28 5 3 1 2 4 2 3 6 : 36644 aida03 : 41 29 30 16 4 3 1 5 1 3 6 : 36588 aida04 : 13 6 23 12 4 4 4 4 2 2 6 : 35412 aida05 : 20 32 23 10 2 2 2 2 1 4 6 : 36736 aida06 : 14 5 10 5 1 3 2 3 2 3 6 : 35680 aida07 : 44 23 22 9 4 3 1 3 2 3 6 : 36200 aida08 : 39 30 28 9 0 3 2 3 3 2 6 : 35308 aida09 : 41 31 28 10 2 5 1 3 1 3 6 : 35484 aida10 : 43 23 27 7 3 1 1 3 3 3 6 : 36916 aida11 : 39 35 17 10 3 3 2 2 2 3 6 : 35908 aida12 : 37 36 15 10 2 2 2 4 3 2 6 : 35684 aida13 : 47 32 17 7 4 2 3 4 3 2 6 : 36012 aida14 : 23 33 14 10 3 4 3 4 3 2 6 : 36164 aida15 : 44 28 24 12 3 2 4 3 3 2 6 : 35920 aida16 : 22 17 24 17 3 4 2 3 1 3 6 : 35648 Stats Aida Correlation Info #8 - elog:319/2 same as elog:317/27 Resume info #3 - elog:319/3 same as elog:317/26 Pause info #2 - elog:319/4 same as elog:317/32 AIDA disk info #6 - elog:319/5 same as elog:317/31 AIDA ADC data items - elog:319/6 same as elog:317/30 Good Events - elog:319/7 same as elog:317/29 [I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see clearly once uploaded the screenshots] Analysis R5_108 elog:319/9 Pause 149 Resume 150 Highest deadtime FEE10 3% 21:01 ucesb ok - Max 1700 Hz Implant - 1MHz Decay 21:35 Temps ok Bias ok Clock check - ok ADC Calc check - ok White Rabbit - Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc5 : 13 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 39 34 29 8 1 4 1 2 2 3 6 : 35772 aida02 : 46 29 28 5 3 2 2 3 2 3 6 : 36288 aida03 : 39 25 29 16 4 2 2 5 1 3 6 : 36660 aida04 : 14 6 19 12 4 4 3 4 2 2 6 : 35096 aida05 : 20 30 24 10 2 2 2 2 1 4 6 : 36736 aida06 : 26 4 11 5 2 4 3 2 2 3 6 : 35672 aida07 : 46 22 22 9 4 3 1 3 2 3 6 : 36200 aida08 : 28 32 28 9 0 2 2 4 3 2 6 : 35664 aida09 : 32 30 29 10 3 5 2 3 1 3 6 : 35776 aida10 : 39 23 25 7 3 1 1 3 3 3 6 : 36868 aida11 : 40 31 21 10 3 3 1 4 1 3 6 : 35688 aida12 : 17 31 12 10 2 2 3 4 3 2 6 : 35772 aida13 : 35 28 22 7 4 2 3 4 3 2 6 : 36012 aida14 : 30 33 14 10 3 4 3 4 3 2 6 : 36192 aida15 : 34 26 24 12 3 2 4 3 3 2 6 : 35864 aida16 : 38 24 27 17 3 4 2 3 1 3 6 : 35816 Stats Good Events - elog:319/10 AIDA ADC data items - elog:319/11 AIDA disk info #6 - elog:319/12 Pause info #2 - elog:319/13 Resume info #3 - elog:319/14 Aida Correlation Info #8 - elog:319/15 - rates showed zero, checked again and normal elog:319/17 Analysis R5_128 elog:319/16 Pause 167 Resume 166 Highest deadtime FEE10 5% 22:00 Temps ok Stats ok ucesb ok 22:35 Temps ok Stats ok ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up again elog:319/17 22:54 They are closing the file, we are on R5_155 23:00
	318	Fri May 14 17:30:56 2021	TD	Analysis of S496 files R5_18, R5_32 & R5_45
S496/R5_18 - attachment 1 LEC fast discriminators - all channels disabled Slow comparator thresholds - p+n FEE64s 0xa except aida09 0xf n+n FEE64s 0x20 Sampling ADCs OFF S496/R5_32 - attachment 2 LEC fast discriminators - all channels disabled Slow comparator thresholds - p+n FEE64s 0xa except aida09 & aida15 0xf n+n FEE64s 0x20 Sampling ADCs ON, trigger threshold 15000 S496/R5_45 - attachment 3 LEC fast discriminators - all channels disabled Slow comparator thresholds - p+n FEE64s 0xc n+n FEE64s 0x20 Sampling ADCs ON, trigger threshold 15000 * N.B. Ignore 'elapsed idle time(s)' metric which does not work with WR timestamps
	317	Fri May 14 14:57:32 2021	CB + OH, TD, NH	Friday 14th May 16:00-20:00
16:00 Shift taken over from CA 16:00 Temperatures OK. Attach 1. Upon reloading aida11 first returns "No respose" for all values. Upon refreshing, aida11 responds but aida 9 returns "No response". Upon refreshing again, attach 1. Bias OK. Attach 2. Statistics OK. Attach 3-7 System-wide checks Clock OK ADC - all fail Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 32 28 25 7 1 4 1 3 2 3 6 : 36112 aida02 : 43 28 29 5 4 2 3 4 1 3 6 : 36092 aida03 : 39 26 28 5 4 3 2 5 1 3 6 : 36428 aida04 : 9 5 21 8 5 4 4 4 2 2 6 : 35292 aida05 : 46 32 26 7 2 2 2 2 2 3 6 : 35768 aida06 : 4 5 16 1 3 4 2 4 1 3 6 : 35352 aida07 : 47 21 25 5 4 3 1 4 1 3 6 : 35604 aida08 : 62 33 32 6 0 2 3 3 3 2 6 : 35520 aida09 : 33 25 28 7 3 5 2 4 1 3 6 : 36140 aida10 : 47 29 22 6 3 2 2 2 2 3 6 : 35716 aida11 : 39 30 29 6 2 3 1 3 2 3 6 : 36124 aida12 : 38 36 20 6 3 2 3 3 3 2 6 : 35448 aida13 : 42 29 24 3 4 2 3 3 3 2 6 : 35440 aida14 : 31 28 18 7 3 3 4 4 2 2 6 : 35228 aida15 : 48 25 30 5 3 2 4 4 2 2 6 : 35272 aida16 : 37 16 28 6 3 3 2 2 2 3 6 : 35796 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 16:15 Control handed over to OH temporarily. 16:25 AIDA stopped. OH attempts to update Merger. Discriminators off. 16:29 AIDA restart begins. 16:33 AIDA restarted. 16:40 Dead time still very high ~50% in most FEEs. 16:50 Checking effect of threshold on dead time. 16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This reduces deadtime to <10%. Sampling ADCs were disabled (switched off) late Wednesday evening. 17:01 DAQ can be restarted. 17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back. 17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20 17:40 OH hands back control. Temperature OK - attach 10 Bias OK - attach 11 Stats - attach 12-17 System-wide checks Clock - all pass ADC - all pass WR Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 56 35 27 8 1 4 1 2 2 3 6 : 35816 aida02 : 43 31 27 5 3 1 3 3 2 3 6 : 36404 aida03 : 45 28 28 16 4 2 2 4 1 3 6 : 36180 aida04 : 2 2 18 12 4 3 3 4 2 2 6 : 34872 aida05 : 28 28 24 10 2 2 2 2 1 4 6 : 36752 aida06 : 28 11 9 5 2 3 3 3 1 3 6 : 35064 aida07 : 52 24 23 9 4 3 1 2 2 3 6 : 35744 aida08 : 21 26 27 8 0 2 3 3 3 2 6 : 35284 aida09 : 40 32 26 10 3 5 1 3 1 3 6 : 35520 aida10 : 46 23 27 7 3 1 1 3 3 3 6 : 36928 aida11 : 49 31 20 10 2 2 1 2 2 3 6 : 35516 aida12 : 39 35 17 9 2 2 3 4 3 2 6 : 35940 aida13 : 40 32 20 7 4 2 3 4 3 2 6 : 36032 aida14 : 33 33 16 10 3 4 3 4 3 2 6 : 36236 aida15 : 53 27 25 12 3 2 4 4 2 2 6 : 35452 aida16 : 35 22 32 15 3 4 3 3 1 3 6 : 36060 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 18:30 Temperatures - OK (attach 18) Bias OK (attach 19) Stats - attach 20-24 System-wide checks Clock - all pass ADC - all pass Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 41 31 31 8 1 4 1 2 2 3 6 : 35788 aida02 : 42 31 29 5 3 2 4 4 1 3 6 : 36304 aida03 : 29 30 29 16 4 3 2 4 1 3 6 : 36276 aida04 : 28 4 20 12 4 3 4 4 2 2 6 : 35280 aida05 : 25 28 23 10 2 2 2 2 1 4 6 : 36724 aida06 : 5 3 10 5 2 4 2 2 2 3 6 : 35308 aida07 : 38 26 22 9 4 3 1 2 2 3 6 : 35688 aida08 : 52 30 28 8 0 2 2 3 3 2 6 : 35200 aida09 : 22 26 21 10 3 4 1 4 1 3 6 : 35704 aida10 : 37 28 25 7 3 1 1 3 3 3 6 : 36900 aida11 : 53 30 20 10 2 3 2 2 2 3 6 : 35908 aida12 : 37 34 17 10 2 2 3 4 3 2 6 : 35956 aida13 : 47 34 16 7 4 2 3 4 3 2 6 : 36012 aida14 : 37 29 17 10 3 4 3 4 3 2 6 : 36236 aida15 : 44 28 25 12 3 2 4 4 2 2 6 : 35424 aida16 : 40 21 33 15 3 4 1 3 1 3 6 : 35576 18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5% 19:40 Temperatures OK Bias OK - attach 26 Stats - attach 27-32 System-wide checks Clock, ADC - all pass WR Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc4 : 12 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 41 33 30 8 1 4 1 2 2 3 6 : 35788 aida02 : 51 30 30 5 3 2 3 4 1 3 6 : 36092 aida03 : 52 31 28 16 4 2 2 5 1 3 6 : 36744 aida04 : 6 8 20 12 4 4 3 4 2 2 6 : 35096 aida05 : 30 31 22 10 2 2 2 2 1 4 6 : 36752 aida06 : 12 6 10 5 2 3 3 2 2 3 6 : 35488 aida07 : 53 24 21 9 4 3 1 3 2 3 6 : 36228 aida08 : 36 28 28 9 0 2 3 3 3 2 6 : 35408 aida09 : 27 28 25 10 3 4 2 3 1 3 6 : 35548 aida10 : 50 23 26 7 3 1 1 3 3 3 6 : 36928 aida11 : 42 28 22 10 3 3 2 2 2 3 6 : 35944 aida12 : 39 35 15 10 2 2 2 4 3 2 6 : 35684 aida13 : 43 34 17 7 4 2 3 4 3 2 6 : 36012 aida14 : 42 32 16 10 3 4 3 4 3 2 6 : 36264 aida15 : 36 30 25 12 3 2 4 3 3 2 6 : 35920 aida16 : 40 21 25 15 3 4 2 3 1 3 6 : 35704 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19:40 Analysed R5_86. Attached. Dead time as before. 19:57 Shift handed over to James S

Goto page Previous 1, 2, 3 ... 18, 19, 20 ... 33, 34, 35 Next

ELOG V3.1.4-unknown