ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 22 of 37

New entries since:

Thu Jan 1 01:00:00 1970

Full | Summary | Threaded | Hide attachments

728 Entries

Goto page Previous 1, 2, 3 ... 21, 22, 23 ... 35, 36, 37 Next

Selected entries:

	ID	Date	Author	Subject
	317	Fri May 14 14:57:32 2021	CB + OH, TD, NH	Friday 14th May 16:00-20:00
16:00 Shift taken over from CA 16:00 Temperatures OK. Attach 1. Upon reloading aida11 first returns "No respose" for all values. Upon refreshing, aida11 responds but aida 9 returns "No response". Upon refreshing again, attach 1. Bias OK. Attach 2. Statistics OK. Attach 3-7 System-wide checks Clock OK ADC - all fail Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 32 28 25 7 1 4 1 3 2 3 6 : 36112 aida02 : 43 28 29 5 4 2 3 4 1 3 6 : 36092 aida03 : 39 26 28 5 4 3 2 5 1 3 6 : 36428 aida04 : 9 5 21 8 5 4 4 4 2 2 6 : 35292 aida05 : 46 32 26 7 2 2 2 2 2 3 6 : 35768 aida06 : 4 5 16 1 3 4 2 4 1 3 6 : 35352 aida07 : 47 21 25 5 4 3 1 4 1 3 6 : 35604 aida08 : 62 33 32 6 0 2 3 3 3 2 6 : 35520 aida09 : 33 25 28 7 3 5 2 4 1 3 6 : 36140 aida10 : 47 29 22 6 3 2 2 2 2 3 6 : 35716 aida11 : 39 30 29 6 2 3 1 3 2 3 6 : 36124 aida12 : 38 36 20 6 3 2 3 3 3 2 6 : 35448 aida13 : 42 29 24 3 4 2 3 3 3 2 6 : 35440 aida14 : 31 28 18 7 3 3 4 4 2 2 6 : 35228 aida15 : 48 25 30 5 3 2 4 4 2 2 6 : 35272 aida16 : 37 16 28 6 3 3 2 2 2 3 6 : 35796 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 16:15 Control handed over to OH temporarily. 16:25 AIDA stopped. OH attempts to update Merger. Discriminators off. 16:29 AIDA restart begins. 16:33 AIDA restarted. 16:40 Dead time still very high ~50% in most FEEs. 16:50 Checking effect of threshold on dead time. 16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This reduces deadtime to <10%. Sampling ADCs were disabled (switched off) late Wednesday evening. 17:01 DAQ can be restarted. 17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back. 17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20 17:40 OH hands back control. Temperature OK - attach 10 Bias OK - attach 11 Stats - attach 12-17 System-wide checks Clock - all pass ADC - all pass WR Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 56 35 27 8 1 4 1 2 2 3 6 : 35816 aida02 : 43 31 27 5 3 1 3 3 2 3 6 : 36404 aida03 : 45 28 28 16 4 2 2 4 1 3 6 : 36180 aida04 : 2 2 18 12 4 3 3 4 2 2 6 : 34872 aida05 : 28 28 24 10 2 2 2 2 1 4 6 : 36752 aida06 : 28 11 9 5 2 3 3 3 1 3 6 : 35064 aida07 : 52 24 23 9 4 3 1 2 2 3 6 : 35744 aida08 : 21 26 27 8 0 2 3 3 3 2 6 : 35284 aida09 : 40 32 26 10 3 5 1 3 1 3 6 : 35520 aida10 : 46 23 27 7 3 1 1 3 3 3 6 : 36928 aida11 : 49 31 20 10 2 2 1 2 2 3 6 : 35516 aida12 : 39 35 17 9 2 2 3 4 3 2 6 : 35940 aida13 : 40 32 20 7 4 2 3 4 3 2 6 : 36032 aida14 : 33 33 16 10 3 4 3 4 3 2 6 : 36236 aida15 : 53 27 25 12 3 2 4 4 2 2 6 : 35452 aida16 : 35 22 32 15 3 4 3 3 1 3 6 : 36060 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 18:30 Temperatures - OK (attach 18) Bias OK (attach 19) Stats - attach 20-24 System-wide checks Clock - all pass ADC - all pass Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 41 31 31 8 1 4 1 2 2 3 6 : 35788 aida02 : 42 31 29 5 3 2 4 4 1 3 6 : 36304 aida03 : 29 30 29 16 4 3 2 4 1 3 6 : 36276 aida04 : 28 4 20 12 4 3 4 4 2 2 6 : 35280 aida05 : 25 28 23 10 2 2 2 2 1 4 6 : 36724 aida06 : 5 3 10 5 2 4 2 2 2 3 6 : 35308 aida07 : 38 26 22 9 4 3 1 2 2 3 6 : 35688 aida08 : 52 30 28 8 0 2 2 3 3 2 6 : 35200 aida09 : 22 26 21 10 3 4 1 4 1 3 6 : 35704 aida10 : 37 28 25 7 3 1 1 3 3 3 6 : 36900 aida11 : 53 30 20 10 2 3 2 2 2 3 6 : 35908 aida12 : 37 34 17 10 2 2 3 4 3 2 6 : 35956 aida13 : 47 34 16 7 4 2 3 4 3 2 6 : 36012 aida14 : 37 29 17 10 3 4 3 4 3 2 6 : 36236 aida15 : 44 28 25 12 3 2 4 4 2 2 6 : 35424 aida16 : 40 21 33 15 3 4 1 3 1 3 6 : 35576 18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5% 19:40 Temperatures OK Bias OK - attach 26 Stats - attach 27-32 System-wide checks Clock, ADC - all pass WR Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc4 : 12 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 41 33 30 8 1 4 1 2 2 3 6 : 35788 aida02 : 51 30 30 5 3 2 3 4 1 3 6 : 36092 aida03 : 52 31 28 16 4 2 2 5 1 3 6 : 36744 aida04 : 6 8 20 12 4 4 3 4 2 2 6 : 35096 aida05 : 30 31 22 10 2 2 2 2 1 4 6 : 36752 aida06 : 12 6 10 5 2 3 3 2 2 3 6 : 35488 aida07 : 53 24 21 9 4 3 1 3 2 3 6 : 36228 aida08 : 36 28 28 9 0 2 3 3 3 2 6 : 35408 aida09 : 27 28 25 10 3 4 2 3 1 3 6 : 35548 aida10 : 50 23 26 7 3 1 1 3 3 3 6 : 36928 aida11 : 42 28 22 10 3 3 2 2 2 3 6 : 35944 aida12 : 39 35 15 10 2 2 2 4 3 2 6 : 35684 aida13 : 43 34 17 7 4 2 3 4 3 2 6 : 36012 aida14 : 42 32 16 10 3 4 3 4 3 2 6 : 36264 aida15 : 36 30 25 12 3 2 4 3 3 2 6 : 35920 aida16 : 40 21 25 15 3 4 2 3 1 3 6 : 35704 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19:40 Analysed R5_86. Attached. Dead time as before. 19:57 Shift handed over to James S
Attachment 1: Temp1.png

Attachment 2: Bias1.png

Attachment 3: Stats1-1.png

Attachment 4: Stats1-2.png

Attachment 5: Stats1-3.png

Attachment 6: Stats1-4.png

Attachment 7: Stats1-5.png

Attachment 8: Stats1-6.png

Attachment 9: R5_45_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 257173968 ( 1605341.2 Hz) Other data format: 4746032 ( 29625.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 149 ( 0.9 Hz) RESUME: 149 ( 0.9 Hz) SYNC100: 32597 ( 203.5 Hz) WR48-63: 32597 ( 203.5 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 4680540 ( 29217.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 198582 ( 1239.6 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 160.199 s FEE elapsed dead time(s) elapsed idle time(s) 1 1.826 60.560 2 1.857 72.569 3 3.101 18.747 4 0.000 149.092 5 0.725 60.979 6 0.276 85.413 7 1.668 27.289 8 2.719 31.235 9 0.000 157.899 10 6.867 10.073 11 2.634 42.733 12 0.375 69.093 13 0.000 148.656 14 0.255 79.878 15 3.499 32.266 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8318308 3120 0 0 3 3 1077 1077 0 960 0 11152 1 17482671 4536 0 0 10 10 2258 2258 0 0 0 22118 2 12591028 1472109 0 0 7 7 1790 1790 0 1468515 0 17237 3 26467090 1465014 0 0 17 17 3574 3574 0 1457832 0 25917 4 2005748 477188 0 0 0 0 331 331 0 476526 0 4838 5 16244903 298514 0 0 5 5 2012 2012 0 294480 0 14514 6 11816116 567974 0 0 7 7 1554 1554 0 564852 0 9683 7 24437155 423701 0 0 8 8 3155 3155 0 417375 0 17498 8 22741388 5724 0 0 10 10 2852 2852 0 0 0 21729 9 1027419 274 0 0 0 0 137 137 0 0 0 1781 10 39497864 9636 0 0 47 47 4771 4771 0 0 0 12114 11 19835466 4892 0 0 8 8 2438 2438 0 0 0 4107 12 15299334 3708 0 0 8 8 1846 1846 0 0 0 19448 13 3576125 840 0 0 0 0 420 420 0 0 0 1437 14 13120071 3182 0 0 7 7 1584 1584 0 0 0 12115 15 22713282 5620 0 0 12 12 2798 2798 0 0 0 2894 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:63108.355s ( 0.507 blocks/s, 0.032 Mb/s)
Attachment 10: Temps2.png

Attachment 11: Bias2.png

Attachment 12: Stats2-1.png

Attachment 13: Stats2-2.png

Attachment 14: Stats2-3.png

Attachment 15: Stats2-4.png

Attachment 16: Stats2-5.png

Attachment 17: Stats2-6.png

Attachment 18: Temp3.png

Attachment 19: Bias3.png

Attachment 20: Stats3-1.png

Attachment 21: Stats3-2.png

Attachment 22: Stats3-3.png

Attachment 23: Stats3-4.png

Attachment 24: Stats3-5.png

Attachment 25: R5_67_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 255938817 ( 1305497.2 Hz) Other data format: 5981183 ( 30508.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 114 ( 0.6 Hz) RESUME: 113 ( 0.6 Hz) SYNC100: 32730 ( 166.9 Hz) WR48-63: 32730 ( 166.9 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 5915496 ( 30173.9 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 257806 ( 1315.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 196.047 s FEE elapsed dead time(s) elapsed idle time(s) 1 2.111 102.399 2 0.513 105.076 3 3.587 39.451 4 0.000 182.105 5 0.792 103.780 6 0.167 115.991 7 1.244 72.680 8 0.805 56.828 9 0.000 190.127 10 2.629 6.967 11 0.295 76.751 12 0.222 114.996 13 0.000 188.225 14 0.031 124.690 15 2.491 63.947 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8223770 3364 0 0 2 2 1092 1092 0 1176 0 13825 1 16628095 4338 0 0 12 12 2157 2157 0 0 0 28636 2 13173045 1831289 0 0 3 3 1864 1864 0 1827555 0 22667 3 27251797 1807380 0 0 13 13 3668 3668 0 1800018 0 33190 4 2564061 622927 0 0 0 0 404 404 0 622119 0 6108 5 14810679 385418 0 0 6 6 1903 1903 0 381600 0 19053 6 12067625 742516 0 0 6 6 1616 1616 0 739272 0 12600 7 21266971 549224 0 0 11 11 2723 2723 0 543756 0 22659 8 23290358 5786 0 0 5 5 2888 2888 0 0 0 28199 9 1157906 324 0 0 0 0 162 162 0 0 0 2443 10 45608185 11489 0 0 27 26 5718 5718 0 0 0 16139 11 20146427 4876 0 0 6 6 2432 2432 0 0 0 5579 12 13411417 3300 0 0 7 7 1643 1643 0 0 0 24922 13 2681820 654 0 0 0 0 327 327 0 0 0 1925 14 11884997 2904 0 0 1 1 1451 1451 0 0 0 15892 15 21771664 5394 0 0 15 15 2682 2682 0 0 0 3969 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:67311.180s ( 0.475 blocks/s, 0.030 Mb/s)
Attachment 26: Stats_RESUME.png

Attachment 27: Stats-CORRELATION.png

Attachment 28: Bias4.png

Attachment 29: Stats-good.png

Attachment 30: Stats-AIDAADC.png

Attachment 31: Stats_DISC.png

Attachment 32: Stats_PAUSE.png

Attachment 33: R5_86_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 256871525 ( 1549964.4 Hz) Other data format: 5048475 ( 30462.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 133 ( 0.8 Hz) RESUME: 132 ( 0.8 Hz) SYNC100: 32617 ( 196.8 Hz) WR48-63: 32617 ( 196.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 4982976 ( 30067.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 217412 ( 1311.9 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 165.727 s FEE elapsed dead time(s) elapsed idle time(s) 1 1.819 66.425 2 0.365 78.785 3 3.792 29.131 4 0.000 149.995 5 0.868 74.751 6 0.080 85.583 7 2.138 33.988 8 2.899 39.410 9 0.000 164.566 10 6.499 11.006 11 1.139 43.564 12 0.944 80.601 13 0.000 152.925 14 0.102 87.597 15 2.546 36.680 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8622288 3316 0 0 4 3 1158 1158 0 993 0 11812 1 17129008 4510 0 0 12 12 2243 2243 0 0 0 24429 2 13474006 1553973 0 0 6 6 1869 1869 0 1550223 0 19028 3 26129234 1524704 0 0 20 20 3374 3374 0 1517916 0 27835 4 2168475 521045 0 0 0 0 352 352 0 520341 0 5247 5 15823578 326080 0 0 4 4 1963 1963 0 322146 0 16074 6 12515224 622754 0 0 2 2 1625 1625 0 619500 0 10516 7 23990039 458079 0 0 12 12 3099 3099 0 451857 0 18924 8 22108276 5660 0 0 13 13 2817 2817 0 0 0 24206 9 1062355 236 0 0 0 0 118 118 0 0 0 1949 10 41223894 10244 0 0 40 40 5082 5082 0 0 0 13136 11 20572413 4974 0 0 7 7 2480 2480 0 0 0 4577 12 14036482 3446 0 0 3 3 1720 1720 0 0 0 21370 13 3191210 818 0 0 0 0 409 409 0 0 0 1573 14 12949121 3198 0 0 3 3 1596 1596 0 0 0 13450 15 21875922 5438 0 0 7 7 2712 2712 0 0 0 3286 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:70812.570s ( 0.452 blocks/s, 0.028 Mb/s)
	316	Fri May 14 07:05:08 2021	CA	Friday 14th May 08:00 - 16:00 shift
08:00 CA takes over 08:28 Problem at UNILAC - no beam 09:30 all system wide checks ok except all FEE64 fail ADC calibration (no waveforms) WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc2 : 10 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp errors: Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 09:36 FEE64 Temperatures ok - attachment 1 Detector bias / leakage currents ok - attachment 2 Statistics - attachments 3-8 09:50 beam is back - writing to file R4_265 09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA 10:10 FRS increase degrader thickness 10.28 all histograms, stats and merger stats zero'd 10.50 all system wide checks ok except Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc2 : 10 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 3 2 2 2 2 2 2 4 3 3 6 : 36860 aida02 : 7 3 5 1 2 5 2 4 3 3 6 : 37284 aida03 : 7 5 3 1 4 4 2 3 3 3 6 : 36756 aida04 : 3 1 20 4 2 3 3 5 1 3 6 : 36052 aida05 : 19 25 26 6 2 2 3 2 2 3 6 : 35828 aida06 : 1 3 16 1 3 4 1 5 1 3 6 : 35580 aida07 : 6 4 1 2 3 3 2 3 3 3 6 : 36552 aida08 : 8 5 2 3 2 3 2 2 2 4 6 : 37064 aida09 : 9 7 1 2 2 4 2 3 3 3 6 : 36652 aida10 : 2 5 3 1 1 4 2 3 3 3 6 : 36544 aida11 : 16 4 4 2 1 2 2 3 3 3 6 : 36384 aida12 : 2 1 1 3 1 2 2 3 3 3 6 : 36288 aida13 : 0 1 1 2 0 4 3 2 3 3 6 : 36184 aida14 : 6 3 2 0 2 3 2 3 3 3 6 : 36432 aida15 : 25 8 2 1 2 3 1 2 2 4 6 : 36836 aida16 : 3 5 1 2 2 2 3 2 3 3 6 : 36100 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 10.50 DAQ continues S496/R4_304 11:18 FEE64 Temperatures ok - attachment 9 Detector bias & leakage currents ok - attachment 10 statistics - attachments 11-16 15:00 FEE64 avg. CPU usage (%) 1 52 2 55 3 55 4 60 5 93 6 50 7 53 8 55 9 55 10 51 11 50 12 67 13 56 14 55 15 55 16 58 15:08 all system wide checks ok except all FEE64 fail ADC calibration (no waveform) WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA errors: Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 15:10 Temperatures ok - attachment 17 Detector bias / leakage current ok - attachment 18 Statistics - attachments 18 -24
Attachment 1: Screenshot_from_2021-05-14_08-32-16.png

Attachment 2: Screenshot_from_2021-05-14_08-32-45.png

Attachment 3: Screenshot_from_2021-05-14_08-33-11.png

Attachment 4: Screenshot_from_2021-05-14_08-33-33.png

Attachment 5: Screenshot_from_2021-05-14_08-34-12.png

Attachment 6: Screenshot_from_2021-05-14_08-34-44.png

Attachment 7: Screenshot_from_2021-05-14_08-35-17.png

Attachment 8: Screenshot_from_2021-05-14_08-35-55.png

Attachment 9: Screenshot_from_2021-05-14_10-17-25.png

Attachment 10: Screenshot_from_2021-05-14_10-17-43.png

Attachment 11: Screenshot_from_2021-05-14_10-22-39.png

Attachment 12: Screenshot_from_2021-05-14_10-23-13.png

Attachment 13: Screenshot_from_2021-05-14_10-23-41.png

Attachment 14: Screenshot_from_2021-05-14_10-24-12.png

Attachment 15: Screenshot_from_2021-05-14_10-24-37.png

Attachment 16: Screenshot_from_2021-05-14_10-25-20.png

Attachment 17: Screenshot_from_2021-05-14_14-07-48.png

Attachment 18: Screenshot_from_2021-05-14_14-11-23.png

Attachment 19: Screenshot_from_2021-05-14_14-13-21.png

Attachment 20: Screenshot_from_2021-05-14_14-13-38.png

Attachment 21: Screenshot_from_2021-05-14_14-14-12.png

Attachment 22: Screenshot_from_2021-05-14_14-14-39.png

Attachment 23: Screenshot_from_2021-05-14_14-15-06.png

Attachment 24: Screenshot_from_2021-05-14_14-15-26.png

	315	Thu May 13 23:58:05 2021	OH	Friday 14th May 00:00 - 08:00
00:00 OH Takes over. PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 OK 0x210 p+n Thresholds 0xa (100keV) n+n Threshold 0x20 (320keV) 01:00 System wide checks Clock status ok ADC calibration fail as expected (Waveforms off) WR Errors Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dbf : 7 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Errors Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Statistics ok - attachment 1 Temp ok - attachment 2 Bias and leakage current ok - attachment 3 01:42 Comparing the CS of AIDA3 and 4 Stats have been running for a while (Believe since 5am) Looking at the counter ratio - attachment 4 We still have a deadtime of around 14% in FEE4 (Assuming FEE3 has no dead time) 01:56 Noticed /media/ThirdDrive was at 97% full Let DESPEC locals know I would like to use the next run stop to change the directory. They said they could do it now so they stopped their run. Changed the symbolic link to /media/1e2.... Now writing to R4_ 03:26 MBS DAQ Stopped as beam stopped 03:33 Beam back now and DAQ running again 04:44 System wide checks as before Statistics ok - attachment 5 Temperatures ok - attachment - 7 Bias and leakage currents ok - attachment 6 05:53 MBS stops run as beam drops 06:00 MBS started a new run as beam is back 07:37 System wide checks clock ok and FPGA as before WR additional difference on FEE5 Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc0 : 8 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics ok - attachment 8 Temp ok - attachment 9 Bias and leakage ok - attachment 10
Attachment 1: 210514_0100_Stats.png

Attachment 2: 210514_0101_Temp.png

Attachment 3: 210514_0102_Bias.png

Attachment 4: 210514_0141_CS_Stats.png

Attachment 5: 210514_0442_Stats.png

Attachment 6: 210514_0443_Bias.png

Attachment 7: 210514_0443_Temp.png

Attachment 8: 210514_0737_Stats.png

Attachment 9: 210514_0738_Temp.png

Attachment 10: 210514_0739_Bias.png

Attachment 11: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png

Attachment 12: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png

Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi.png

Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(1).png

Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(2).png

Attachment 16: Screenshot_2021-05-14_Statistics_aidas-gsi(3).png

Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(4).png

Attachment 18: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png

Attachment 19: Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png

Attachment 20: 600.png

Attachment 21: 601.png

	314	Thu May 13 21:21:25 2021	Philippos and Marc	system wide checks
Wide checks at Time 22:22 CET Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dbc : 4 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 11 3 5 5 4 1 3 2 4 6 : 37876 aida02 : 20 5 4 3 2 5 2 4 3 3 6 : 37400 aida03 : 20 3 4 3 4 4 1 3 2 4 6 : 37640 aida04 : 25 6 2 3 4 4 3 3 3 3 6 : 37140 aida05 : 30 10 5 7 2 6 2 4 2 4 6 : 38776 aida06 : 21 11 4 4 3 5 2 4 3 3 6 : 37548 aida07 : 23 8 4 4 2 6 1 3 2 4 6 : 37852 aida08 : 27 7 2 3 4 3 2 4 3 3 6 : 37284 aida09 : 31 6 5 6 3 3 2 4 3 3 6 : 37372 aida10 : 22 5 3 4 3 6 2 2 2 4 6 : 37616 aida11 : 23 5 4 2 5 2 1 3 2 4 6 : 37444 aida12 : 20 7 4 7 3 5 2 3 3 3 6 : 37096 aida13 : 28 5 3 3 1 5 2 3 3 3 6 : 36840 aida14 : 38 12 2 4 2 4 1 4 3 3 6 : 37144 aida15 : 27 9 5 4 2 5 3 2 3 3 6 : 36740 aida16 : 21 6 4 4 1 4 2 3 3 3 6 : 36740
Attachment 1: 2021-05-13_ucesb_scalers_22h22mn.png

Attachment 2: 2021-05-13_Currents_22h22mn.png

Attachment 3: 2021-05-13_temperatures_22h22.png

Attachment 4: 2021-05-13-22h22mn_Stat-good-events.png

Attachment 5: 2021-05-13-22h22mn_ADC-data-items.png

Attachment 6: 2021-05-13_22h22mn-Correlation-info.png

Attachment 7: 2021-05-13_22h22mn-Pause-info.png

Attachment 8: 2021-05-13_22h22mn-Resume-Info.png

Attachment 9: 2021-05-13_merger_22h22.png

Attachment 10: 2021-05-13_22h22mn-LAY-ID1.png

Attachment 11: 2021-05-13_22h22mn-LAY-ID2.png

Attachment 12: 2021-05-13_22h22mn-LAY-ID3.png

Attachment 13: 2021-05-13_22h22mn-LAY-ID4.png

	313	Thu May 13 19:52:36 2021	Philippos and Marc	Connection problem
At 20:15, We lost connection with AnyDesk. Liliana restarted Anydesk on the AIDA DAQ PC. All seemed fine on AIDA DAQ but there was a more generic issue on the main DAQ, possibly the timesorter. Helena restarted all after ~30mn and it's all working again now.
Attachment 1: 2021-05-13_ucesb_scalers.png

	312	Thu May 13 17:05:42 2021	Muneerah, MArc	Thu 13 May 04:00-00:00
19:08 System Check Rates from ucesb below 1 kHz Atachment 1 Stats Atachment 2 Temperatures Atachment 3 Voltages System wide checks are all ok except ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dba : 2 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA all passed Memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 6 2 2 1 2 4 3 4 2 3 7 : 40424 aida02 : 3 4 2 2 1 2 1 3 3 3 7 : 40140 aida03 : 10 10 0 3 2 4 2 4 2 3 7 : 40280 aida04 : 3 2 0 2 2 3 2 3 1 4 7 : 40540 aida05 : 4 9 2 3 2 4 3 3 1 4 7 : 41048 aida06 : 8 6 1 1 2 3 2 2 3 3 7 : 40064 aida07 : 1 4 0 1 1 3 1 3 3 3 7 : 40196 aida08 : 1 4 2 2 2 4 3 4 2 3 7 : 40452 aida09 : 8 5 2 2 1 2 1 3 3 3 7 : 40168 aida10 : 22 5 4 2 1 3 1 4 2 3 7 : 39872 aida11 : 2 1 2 1 2 2 3 3 2 3 7 : 39632 aida12 : 8 5 2 2 1 3 1 4 2 3 7 : 39784 aida13 : 18 8 2 1 0 2 3 3 2 3 7 : 39624 aida14 : 23 12 2 2 1 2 2 4 2 3 7 : 40028 aida15 : 3 5 3 2 2 3 1 4 2 3 7 : 39844 aida16 : 0 2 1 2 1 4 2 4 2 3 7 : 40096
Attachment 1: 130521_1802_Stats.png

Attachment 2: 130521_1802_Temp.png

Attachment 3: 130521_1804_Voltages.png

	311	Thu May 13 07:10:17 2021	CA, TD	May 13th 08:00 - 16:00 shift
08:10 all system wide checks ok except all FEE64 modules fail ADC calibration WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9db9 : 1 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 08:16 attempt to recalibrate ADCs using FADC align and control - unsuccessful on all 08:18 FEE64 temps ok - attachment 1 bias/leakage currents ok - attachment 2 statistics - attachments 3-8 10:13 all system wide checks ok except all FEE64 modules fail ADC calibration WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9db9 : 1 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 10:16 FEE64 temps ok - attachment 9 bias/leakage currents ok - attachment 10 statistics - attachments 11-16 12:16 all system wide checks ok except all FEE64 modules fail ADC calibration WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dba : 2 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 12:16 FEE64 temps ok - attachment 17 bias/leakage currents ok - attachment 18 statistics - attachments 19-25 14:08 all system wide checks ok except all FEE64 modules fail ADC calibration WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dba : 2 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FEE64 temps ok - attachment 16 bias/leakage currents ok - attachment 27 statistics - attachments 28-35
Attachment 1: Screenshot_from_2021-05-13_07-17-47.png

Attachment 2: Screenshot_from_2021-05-13_07-18-57.png

Attachment 3: Screenshot_from_2021-05-13_07-19-42.png

Attachment 4: Screenshot_from_2021-05-13_07-20-45.png

Attachment 5: Screenshot_from_2021-05-13_07-23-01.png

Attachment 6: Screenshot_from_2021-05-13_07-23-35.png

Attachment 7: Screenshot_from_2021-05-13_07-24-02.png

Attachment 8: Screenshot_from_2021-05-13_07-25-15.png

Attachment 9: Screenshot_from_2021-05-13_09-17-57.png

Attachment 10: Screenshot_from_2021-05-13_09-18-21.png

Attachment 11: Screenshot_from_2021-05-13_09-18-58.png

Attachment 12: Screenshot_from_2021-05-13_09-19-32.png

Attachment 13: Screenshot_from_2021-05-13_09-20-23.png

Attachment 14: Screenshot_from_2021-05-13_09-21-05.png

Attachment 15: Screenshot_from_2021-05-13_09-21-53.png

Attachment 16: Screenshot_from_2021-05-13_09-22-18.png

Attachment 17: Screenshot_from_2021-05-13_11-19-16.png

Attachment 18: Screenshot_from_2021-05-13_11-19-30.png

Attachment 19: Screenshot_from_2021-05-13_11-20-09.png

Attachment 20: Screenshot_from_2021-05-13_11-20-33.png

Attachment 21: Screenshot_from_2021-05-13_11-20-59.png

Attachment 22: Screenshot_from_2021-05-13_11-21-29.png

Attachment 23: Screenshot_from_2021-05-13_11-21-53.png

Attachment 24: Screenshot_from_2021-05-13_11-22-30.png

Attachment 25: Screenshot_from_2021-05-13_11-24-36.png

Attachment 26: Screenshot_from_2021-05-13_13-09-43.png

Attachment 27: Screenshot_from_2021-05-13_13-10-03.png

Attachment 28: Screenshot_from_2021-05-13_13-10-28.png

Attachment 29: Screenshot_from_2021-05-13_13-10-52.png

Attachment 30: Screenshot_from_2021-05-13_13-11-22.png

Attachment 31: Screenshot_from_2021-05-13_13-11-51.png

Attachment 32: Screenshot_from_2021-05-13_13-12-24.png

Attachment 33: Screenshot_from_2021-05-13_13-13-34.png

Attachment 34: Screenshot_from_2021-05-13_13-14-16.png

	310	Thu May 13 00:51:40 2021	Muneerah and OH	Thu 13 May 00:00-08:00
01:50 Beam on The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing. Tempetratures OK Rates OK Attachment 1 Voltages Atachment 2 Rates Atachment 3 Temperatures Attachment 4 ucesb rates less than 1 kHz System wide check are ok except: Base Current Difference aida05 fault 0x36ca : 0x36ce : 4 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows: Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida13 fault 0xa : 0xe9 : 223 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760 aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644 aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188 aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404 aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104 aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128 aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500 aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672 aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588 aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460 aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348 aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000 aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840 aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996 aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280 aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716 <strong>05:00 OH Took over at 04:00</strong> System wide checks ok except Base Current Difference aida05 fault 0x36ca : 0x36d0 : 6 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x1 : 1 aida13 fault 0xa : 0xe9 : 223 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 05:41 AIDA 05 has dropped out. Will restart the system 05:58 DAQ recovered from powercycle p+n strips at 0xa n+n strips at 0x20 ADC control register at 0xFF All waveforms turned off in LED Statistics - attachment 5 Correlation scaler in FEE 3 and 4 are mostly tracking 1:1 06:34 Local users have been unable to recover MBS And have been unable to contact local experts 07:22 Bias and leakage currents ok - attachment 6 Stats attachment -7
Attachment 1: 130521_0155_Voltages.png

Attachment 2: 130521_0157_Stats.png

Attachment 3: 130521_0158_Temp.png

Attachment 4: 130521_0204_ucesb.png

Attachment 5: 210513_0600_Stats.png

Attachment 6: 210513_0722_bias.png

Attachment 7: 210513_0722_Stats.png

	309	Wed May 12 18:53:38 2021	JS	Wed 12 20:00-00:00
19:51 Shift change Stats ok ucesb ok, Max implants ~1500 Hz 20:30 Stats ok ucesb ok, Max implants ~1000 Hz current ok (maybe leveling out) 20:51 The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340. 20:59 Check Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 16, Failed 0 WR: Base Current Difference aida05 fault 0x36ca : 0x36ce : 4 White Rabbit error counter test result: Passed 15, Failed 1 FPGA time: Base Current Difference aida13 fault 0xa : 0xb6 : 172 FPGA Timestamp error counter test result: Passed 15, Failed 1 --- both errors above same module as earlier shift Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 3 2 0 1 3 3 2 3 3 6 : 36092 aida02 : 6 4 4 2 2 3 1 3 3 3 6 : 36280 aida03 : 6 3 2 0 0 3 3 2 3 3 6 : 36048 aida04 : 10 3 1 1 3 3 1 3 3 3 6 : 36272 aida05 : 19 8 2 0 2 3 1 3 3 3 6 : 36268 aida06 : 2 2 0 2 3 4 2 2 3 3 6 : 36120 aida07 : 6 7 3 0 3 4 3 2 3 3 6 : 36416 aida08 : 7 5 4 0 3 2 2 3 3 3 6 : 36420 aida09 : 21 6 3 2 2 4 2 3 2 3 6 : 35700 aida10 : 13 3 1 1 3 3 1 4 2 3 6 : 35772 aida11 : 25 6 4 2 1 4 2 4 2 3 6 : 36180 aida12 : 0 4 0 3 2 3 3 3 2 3 6 : 35712 aida13 : 18 5 2 3 2 3 2 4 2 3 6 : 36080 aida14 : 2 1 1 1 2 3 2 4 2 3 6 : 35904 aida15 : 0 1 1 3 0 4 2 2 3 3 6 : 35960 aida16 : 4 5 3 2 3 4 1 3 2 3 6 : 35432 ucesb ok elog:309/1 current ok elog:309/2 Stats ok elog:309/3 temps ok elog:309/4 21:36 ucesb ok Stats ok 22:01 ucesb ok 1500 Hz peak implant Stats ok temps ok bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values. 22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate). Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time. Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume. Oscar is going to raise the threshold on one of the y FEEs as a test Increased aida4 slow comparator threshold changed There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate? Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler 23:40 FRS magnet issue, possibly power, no beam 00:03 bias ok elog:309/5 has come down because no beam handing over to muneerah
Attachment 1: ucesb.png

Attachment 2: bias1.png

Attachment 3: Screenshot_2021-05-12_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-05-12_Temperature_and_status_scan_aidas-gsi.png

Attachment 5: bias2.png

	308	Wed May 12 15:03:10 2021	CB, OH	Wed 12 16:00-20:00
16:00 Shift change Statistics OK (attach 1) Temps OK (attach 2) Bias OK (attach 3) System wide checks Clock, ADC - OK WR Base Current Difference aida05 fault 0x36ca : 0x36cb : 1 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Base Current Difference aida13 fault 0xa : 0x4b : 65 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 5 6 1 1 2 2 3 3 3 6 : 36388 aida02 : 21 10 2 2 2 3 2 2 3 3 6 : 36100 aida03 : 25 5 3 1 0 3 3 2 3 3 6 : 36188 aida04 : 5 2 0 3 2 3 2 3 3 3 6 : 36484 aida05 : 16 6 3 1 2 4 1 3 2 4 6 : 37440 aida06 : 2 2 2 2 2 3 1 3 3 3 6 : 36216 aida07 : 16 9 3 0 3 4 3 2 3 3 6 : 36472 aida08 : 0 7 1 1 2 2 3 3 2 3 6 : 35560 aida09 : 15 7 4 2 2 4 2 3 2 3 6 : 35700 aida10 : 3 7 3 4 1 3 1 4 1 4 6 : 36788 aida11 : 5 1 3 1 1 4 3 3 2 3 6 : 35756 aida12 : 5 5 0 3 2 3 2 4 2 3 6 : 35996 aida13 : 2 5 1 2 2 4 3 3 2 3 6 : 35840 aida14 : 19 5 3 1 1 2 1 3 3 3 6 : 36100 aida15 : 20 6 6 2 1 3 2 2 3 3 6 : 36064 aida16 : 7 9 4 1 3 4 2 3 2 3 6 : 35716 File size change Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 10 16:47:51 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 16:20 Beam is back. Currently on File R1_200 16:43 Beam is being optimised. 17:00 UCESB OK Stats OK (attach 4) Temps OK Bias OK (attach 5) System-wide checks as before 17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK 17:52 Started compressing R1_1* 18:00 UCESB ok (attach 5) Stats OK (attach 6) Temps OK Bias OK (attach 7) System wide checks as before 19:00 UCESB ok (as before) Stats OK (attach 8) Temps OK Bias OK (attach 9) Merger OK (attach 10) 19:32 UCESB ok Stats OK (attach 11) Temps OK (attach 12) Bias OK (attach 13) System wide checks as before 19:51 Shift handed over to James S
Attachment 1: Stat1.png

Attachment 2: Temp1.png

Attachment 3: Bias1.png

Attachment 4: Stats2.png

Attachment 5: Bias2.png

Attachment 6: Stats3.png

Attachment 7: Bias3.png

Attachment 8: ucesb1.png

Attachment 9: Stats4.png

Attachment 10: Bias4.png

Attachment 11: Merger.png

Attachment 12: ucesb2.png

Attachment 13: Stats5.png

Attachment 14: Temps2.png

Attachment 15: Bias5.png

	307	Wed May 12 06:59:55 2021	OH, RDP	Wed 12 May 08:00-16:00
08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now. They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID 08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2 09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage. Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data Data rate 16.3MB/s ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then. 09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage 09:47 Beam back turned off no storage. On file R1_10 11:04 Now that we have implants on the detector we get a better look at the strips From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2 This adaptor board corresponds to FEE11 12:17 Leakage currents screenshot 512.png 13:30 Beam off. Expected back at 15:00... 14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11. They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there. Rates are as before. No better, no worse. 15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...
Attachment 1: 1dImplants.png

Attachment 2: 1dImplants2.png

Attachment 3: 512.png

Attachment 4: 513.png

	306	Wed May 12 02:07:59 2021	MA, BA	Wed 12 May 00:00-08:00
03:12 System Check attachment 1 : Current attachment 2 : Temperature attachment 3 : Rates Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Calibration test result: Passed 16, Failed 0 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x36ca : 0x36cb : 1 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida13 fault 0xa : 0xf : 5 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 5 7 2 1 2 2 3 3 3 6 : 36464 aida02 : 9 8 3 3 1 4 1 2 4 3 6 : 36916 aida03 : 5 2 6 2 0 3 2 3 3 3 6 : 36420 aida04 : 6 5 2 3 4 4 2 3 3 3 6 : 36800 aida05 : 17 6 6 2 2 4 1 3 2 4 6 : 37524 aida06 : 7 12 3 4 3 3 1 3 3 3 6 : 36460 aida07 : 17 11 5 1 3 3 3 2 3 3 6 : 36428 aida08 : 3 5 0 1 4 2 2 4 2 3 6 : 35924 aida09 : 27 6 4 2 0 2 2 2 3 3 6 : 35868 aida10 : 16 11 8 0 2 2 1 3 2 4 6 : 37272 aida11 : 15 2 2 3 1 4 3 4 2 3 6 : 36364 aida12 : 1 6 4 3 1 3 2 4 2 3 6 : 35988 aida13 : 22 14 10 2 4 2 2 4 2 3 6 : 36264 aida14 : 26 10 4 3 2 1 1 3 3 3 6 : 36184 aida15 : 14 2 3 2 2 4 1 2 3 3 6 : 35896 aida16 : 7 5 4 0 2 4 2 3 2 3 6 : 35588 06:10 DSSD1 rate high! Attached 4 Called OH and he woke up to fix it :) It was a problem with one of the ASIC according to what he says: One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35 07:03 System check attachment 5 Spectrum rate attachment 6 Voltages attachment 7 Rates attachment 8 Temperature System wide clock are all ok except White rabbit Base Current Difference aida05 fault 0x36ca : 0x36cb : 1 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Base Current Difference aida13 fault 0xa : 0x14 : 10 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last
Attachment 1: 1205_0310.png

Attachment 2: 1205_0311.png

Attachment 3: 1205_0312.png

Attachment 4: 120521_0630ucesb.png

Attachment 5: 120521_0639_SpecRates.png

Attachment 6: 120521_0655_Voltages.png

Attachment 7: 120521_0657_Rates.png

Attachment 8: 120521_0685_Temp.png

Attachment 9: 120521_0635_ASICs.png

	305	Tue May 11 07:21:24 2021	OH	Tuesday 11 May
08:20 System wide checks all ok except Base Current Difference aida01 fault 0xc10c : 0xc10e : 2 aida02 fault 0xbe2a : 0xbe2c : 2 aida03 fault 0xf665 : 0xf667 : 2 aida04 fault 0xeabd : 0xeabf : 2 aida05 fault 0x36c8 : 0x36ca : 2 aida06 fault 0x2ea2 : 0x2ea4 : 2 aida07 fault 0x9186 : 0x9188 : 2 aida08 fault 0x5443 : 0x5445 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR and Base Current Difference aida01 fault 0xc10c : 0xc10e : 2 aida02 fault 0xbe2a : 0xbe2c : 2 aida03 fault 0xf665 : 0xf667 : 2 aida04 fault 0xeabd : 0xeabf : 2 aida05 fault 0x36c8 : 0x36ca : 2 aida06 fault 0x2ea2 : 0x2ea4 : 2 aida07 fault 0x9186 : 0x9188 : 2 aida08 fault 0x5443 : 0x5445 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics still high - attachment 1 Temperatures ok - attachment 2 Bias and leakage currents ok - attachment 3
Attachment 1: 210511_0823_Stats.png

Attachment 2: 210511_0824_Temp.png

Attachment 3: 210511_0825_Bias.png

	304	Mon May 10 11:45:15 2021	NH	Confusing behaviour of rates
Testing the behaviour of powering down the fast ADCs in aida01 (the hot FEE) Set local control 2 to 0xff in aida01 Wait... see temperature drpos by ~3 degrees (PJCS reports much larger drop in T9) So far so normal but aida05 rates go very high when this waveform is disabled!! Set local control to 0x00 again They drop back... Recalibrated FADCs and back to normal
	303	Mon May 10 08:49:37 2021	TD	Monday 10 May
09.51 All system wide checks OK except Base Current Difference aida05 fault 0x8441 : 0x8442 : 1 aida06 fault 0xc5c5 : 0xc5c6 : 1 aida07 fault 0x6f07 : 0x6f08 : 1 aida08 fault 0x4814 : 0x4815 : 1 White Rabbit error counter test result: Passed 12, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 1 4 6 3 5 5 2 3 2 3 6 : 36004 aida02 : 4 1 3 6 3 4 3 3 2 3 6 : 36040 aida03 : 12 8 5 2 3 3 3 4 2 3 6 : 36416 aida04 : 1 2 5 4 1 4 3 4 2 3 6 : 36388 aida05 : 22 8 4 2 4 5 1 4 1 4 6 : 37272 aida06 : 5 4 0 6 3 6 3 3 2 3 6 : 36276 aida07 : 38 16 7 7 1 6 1 3 3 3 6 : 37032 aida08 : 16 4 2 3 1 4 1 3 3 3 6 : 36384 aida09 : 2 3 7 6 5 5 2 3 2 3 6 : 36112 aida10 : 42 15 4 6 2 6 2 4 2 3 6 : 36768 aida11 : 1 5 13 7 3 3 3 3 2 3 6 : 36124 aida12 : 23 9 3 4 7 3 2 3 2 3 6 : 35988 aida13 : 2 6 7 3 4 4 3 2 2 3 6 : 35592 aida14 : 17 11 3 4 3 5 2 3 2 3 6 : 35980 aida15 : 4 7 5 2 1 6 2 3 2 3 6 : 35864 aida16 : 4 1 3 3 4 2 3 4 2 3 6 : 36264 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Sun May 09 15:10:52 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 09.52 All histograms, stats and merger stats zero'd FEE64 temps OK - attachment 1 ADC data, disc/WR 28-27/pause/correlation stats OK - attachments 2-6 Rate spectra - attachment 7 ASIC check load Rate spectra - attachment 8 1.8.W spectra 20us FSR - attachments 9 & 10 Grafana - most recent 7 days DSSSD bias and leakage current - attachment 11 Merger stats/tape server/merger - attachment 12 DSSSD bias and leakage current OK - attachment 13 10.08 1.8.W spectra 20us, 200us, 2ms & 20ms FSR - attachments 14-21 we are not missing extrinsic noise features at longer time ranges cf. 20us FSR 12.18 Cannot connect to port 8015, MIDAS terminal says "Killed" Restart MIDAS 8015 Collect FPGA baseline to avoid TCL error FEEs unaffected and running as before 13.25 FEEs power cycled to see if any change in weird noise/rates situation No change ASICs synchronised and back to normal 14.34 Start a pulser walkthrough in May21/R10 Attenuation 5x, Rep Rate 25 Hz Amplitudes 10..9...1 14.45 Back to 2 Hz @ 2V Pulser, walkthrough complete. No Storage again. PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 (NB. actually it is 12 MeV) Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 OK
Attachment 1: Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-05-10_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-10_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-10_Statistics_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-10_Statistics_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-10_Statistics_aidas-gsi(4).png

Attachment 7: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png

Attachment 8: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png

Attachment 9: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png

Attachment 10: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png

Attachment 11: Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png

Attachment 12: 510.png

Attachment 13: 511.png

Attachment 14: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png

Attachment 15: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png

Attachment 16: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png

Attachment 17: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png

Attachment 18: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png

Attachment 19: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png

Attachment 20: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png

Attachment 21: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png

	302	Sun May 9 19:29:23 2021	TD	Sunday 9 May

Attachment 1: Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-05-09_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-09_Statistics_aidas-gsi(1).png

Attachment 4: Screenshot_2021-05-09_Statistics_aidas-gsi(2).png

Attachment 5: Screenshot_2021-05-09_Statistics_aidas-gsi(3).png

Attachment 6: Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png

Attachment 7: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png

Attachment 8: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png

Attachment 9: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png

Attachment 10: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png

Attachment 11: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png

Attachment 12: Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png

Attachment 13: 503.png

Attachment 14: 504.png

	301	Sat May 8 09:05:04 2021	TD	Saturday 8 May
09.54 FEE64 temperatures OK - attachment 1 ADC data item stats OK - attachment 2 p+n FEE64s 11x <100k, 1x <200k Rate spectra - attachments 3 & 4 before and after ASIC check load 1.8.L spectra - attachments 5 & 6 1.8.W spectra 20us FSR - attachments 7 & 8 Disc, WR-28-47, pause, resume info data items - attachments 9-12 note large number pause/resume for aida13 - legacy numbers since last restart/zero? attachment 16 for pause stats since stats zero at 10.02 - looks OK Grafana DSSSD bias & leakage currents for most recent 48h OK - atatchment 13 DSSSD bias & leakage currents - attachment 14 Merger stats/merger/tape server messages - attachment 15 10.02 All histograms and stats zero'd 10.06 All system wide checks OK except Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 06 15:29:51 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 10.25 Pulser peak width aida01 97 ch FWHM, aida02 350 ch FWHM peak widths are increasing cf. earlier this week - see https://elog.ph.ed.ac.uk/DESPEC/291 , https://elog.ph.ed.ac.uk/DESPEC/298
Attachment 1: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-05-08_Statistics_aidas-gsi.png

Attachment 3: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi.png

Attachment 4: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(1).png

Attachment 5: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(2).png

Attachment 6: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(3).png

Attachment 7: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(4).png

Attachment 8: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(5).png

Attachment 9: Screenshot_2021-05-08_Statistics_aidas-gsi(1).png

Attachment 10: Screenshot_2021-05-08_Statistics_aidas-gsi(2).png

Attachment 11: Screenshot_2021-05-08_Statistics_aidas-gsi(3).png

Attachment 12: Screenshot_2021-05-08_Statistics_aidas-gsi(4).png

Attachment 13: Screenshot_2021-05-08_AIDA_Alerting_-_Grafana.png

Attachment 14: 500.png

Attachment 15: 501.png

Attachment 16: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png

Attachment 17: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png

Attachment 18: Screenshot_2021-05-08_Statistics_aidas-gsi(6).png

Attachment 19: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 20: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(6).png

Attachment 21: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(7).png

Attachment 22: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(8).png

Attachment 23: Screenshot_2021-05-08_Statistics_aidas-gsi(7).png

Attachment 24: Screenshot_2021-05-08_Statistics_aidas-gsi(8).png

Attachment 25: Screenshot_2021-05-08_Statistics_aidas-gsi(9).png

Attachment 26: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(9).png

Attachment 27: Screenshot_2021-05-08_Statistics_aidas-gsi(10).png

	300	Fri May 7 19:47:54 2021	PJCS	HowTo mitigate excessive temperature in an FEE64
After tests in the Daresbury T9 system. Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees. To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected. Alternatively STOP acquisition, set the ADC Control register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.
	299	Fri May 7 13:11:22 2021	NH	Friday 7th May
14:11 - Alpha has been running most of morning Just saw rates in tape spike to 6 MB/s... Stop output to tape and look: ASIC check & load... all rates except aida04 back to 0 Start output to tape Back to ~300 KB/s R6_49 will be affected by this. Others seem OK System wide check failures: Base Current Difference aida01 fault 0xb405 : 0xb406 : 1 aida02 fault 0xefc7 : 0xefc8 : 1 aida03 fault 0xdaab : 0xdaac : 1 aida04 fault 0x8f7c : 0x8f7d : 1 aida05 fault 0xb5bd : 0xb5be : 1 aida06 fault 0xeff1 : 0xeff2 : 1 aida07 fault 0x8f57 : 0x8f58 : 1 aida08 fault 0xbef7 : 0xbef8 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x200 : 512 aida13 fault 0x0 : 0x61 : 97 FPGA Timestamp error counter test result: Passed 14, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page) Temps & Rates OK Will now stop Alpha run and check noise situation Run stopped at R6_49 Switching to nostorage (NOTAPE/R2) Set ASICs to threshold 0xa, Rates OK Rate to Tape/MBS ca 9 MB/s No change to situation since yesterday... good sign During MBS test aida13 rebooted... crash log from ttyUSB5 attached It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again... cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :) A power cycle of all FEEs was performed after aida13's reboot to make sure things are good. All systems check OK
Attachment 1: elog_locks.png

Attachment 2: elog_temps.png

Attachment 3: elog_rates.png

Attachment 4: elog_rates1.png

Attachment 5: aida13.txt
06/19:25:39\|Data Acquisition Statistics counters now cleared^M 06/19:25:55\|Clear Statistics (1)^M 06/19:25:55\|------------[ cut here ]------------^M 07/15:23:58\|kernel BUG at mm/slab.c:2974!^M 07/15:23:58\|Oops: Exception in kernel mode, sig: 5 [#1]^M 07/15:23:58\|PREEMPT Xilinx Virtex440^M 07/15:23:59\|Modules linked in: aidamem xdriver xh_spidev_register^M 07/15:23:59\|NIP: c009211c LR: c00920b0 CTR: 00000007^M 07/15:23:59\|REGS: c0391bf0 TRAP: 0700 Not tainted (2.6.31)^M 07/15:23:59\|MSR: 00021000 <ME,CE> CR: 24022048 XER: 00000000^M 07/15:23:59\|TASK = c036e318[0] 'swapper' THREAD: c0390000^M 07/15:23:59\|GPR00: 00000001 c0391ca0 c036e318 c680daf0 c694003c 0000000a c6940020 00000009 ^M 07/15:23:59\|GPR08: 0000001b c680dae0 00000cf0 c680dae0 24008042 00005aa8 c03a0000 00000020 ^M 07/15:23:59\|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 ^M 07/15:23:59\|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000006 c680e400 ^M 07/15:23:59\|NIP [c009211c] cache_alloc_refill+0x130/0x608^M 07/15:23:59\|LR [c00920b0] cache_alloc_refill+0xc4/0x608^M 07/15:23:59\|Call Trace:^M 07/15:23:59\|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)^M 07/15:23:59\|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M 07/15:23:59\|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M 07/15:23:59\|[c0391d40] [c0042938] __send_signal+0x78/0x260^M 07/15:23:59\|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M 07/15:24:00\|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M 07/15:24:00\|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M 07/15:24:00\|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M 07/15:24:00\|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M 07/15:24:00\|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M 07/15:24:00\|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M 07/15:24:00\|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M 07/15:24:00\|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M 07/15:24:00\|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M 07/15:24:00\|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M 07/15:24:00\|[c0391fb0] [c000172c] rest_init+0x70/0x84^M 07/15:24:00\|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M 07/15:24:00\|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M 07/15:24:00\|Instruction dump:^M 07/15:24:00\|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 ^M 07/15:24:00\|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 ^M 07/15:24:00\|Kernel panic - not syncing: Fatal exception in interrupt^M 07/15:24:00\|Call Trace:^M 07/15:24:00\|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)^M 07/15:24:00\|[c0391a80] [c00345bc] panic+0x94/0x168^M 07/15:24:01\|[c0391ad0] [c000bd44] die+0x178/0x18c^M 07/15:24:01\|[c0391af0] [c000c000] _exception+0x164/0x1b4^M 07/15:24:01\|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c^M 07/15:24:01\|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608^M 07/15:24:01\|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M 07/15:24:01\|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M 07/15:24:01\|[c0391d40] [c0042938] __send_signal+0x78/0x260^M 07/15:24:01\|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M 07/15:24:01\|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M 07/15:24:01\|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M 07/15:24:01\|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M 07/15:24:01\|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M 07/15:24:01\|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M 07/15:24:01\|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M 07/15:24:01\|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M 07/15:24:01\|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M 07/15:24:01\|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M 07/15:24:01\|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M 07/15:24:01\|[c0391fb0] [c000172c] rest_init+0x70/0x84^M 07/15:24:01\|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M 07/15:24:02\|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M 07/15:24:02\|Rebooting in 180 seconds.. 07/15:27:02\|^MISOL Version 1.00 Date 9th January 2017
	298	Thu May 6 16:59:20 2021	TD, OH	Wednesday 6 May
18.09 NH has installed jumpers LK2-4 for all adaptor PCBs per FEE64 rate spectra - attachments 1-4 attachment 4 with BNC PB-5 pulser 10Hz ADC data items stats - attachment 5 1.8.W spectra - attachments 6 & 7 Observe significant reduction in DSSSD/ribbon cable boundary channel rates - status quo ante Tuesday 4 May https://elog.ph.ed.ac.uk/DESPEC/291 ? 1.8.L spectra - attachments 8 & 9 pulser peak width aida01 85 ch FWHM, aida2 295 ch FWHM 19:21 Alpha run started May21/R6 Statistics dominated by aida04 with a rate of 6k System wide checks all ok except: Base Current Difference aida13 fault 0x0 : 0x17 : 23 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Writing to file at ~300kB/s Rate spectra for aida04 shows dominated by first asic - attachment 10 1*L spectra for aida04 asic 1 - attachment 11 20.25 per FEE64 rate spectra - attachment 12 handful of hot channels aida06, 09 & 10 ADC data item stats - attachment 13
Attachment 1: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(4).png

Attachment 2: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(5).png

Attachment 3: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(6).png

Attachment 4: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(7).png

Attachment 5: Screenshot_2021-05-06_Statistics_aidas-gsi(1).png

Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(8).png

Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(9).png

Attachment 8: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png

Attachment 9: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png

Attachment 10: 210506_1924_aida04Rate.png

Attachment 11: 210506_1927_1*L.png

Attachment 12: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(2).png

Attachment 13: Screenshot_2021-05-06_Statistics_aidas-gsi.png

Goto page Previous 1, 2, 3 ... 21, 22, 23 ... 35, 36, 37 Next

ELOG V3.1.3-7933898