ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 19 of 35

Full | Summary | Threaded | Hide attachments

686 Entries

Goto page Previous 1, 2, 3 ... 18, 19, 20 ... 33, 34, 35 Next

Selected entries:

	ID	Date	Author	Subject
	335	Thu May 20 01:00:30 2021	Muneerah	Thu May 20 00:00-08:00
02:00 System check The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA! The rates on ucesb got back to work by around 01:12 Deat Time check ok R14_870, attachment 1 Statistics are ok attachment 2 Temperatures are ok attacment 3 Voltages are ok attachment 4 system wide check All ok except ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbite 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 8 8 5 0 6 1 2 2 2 7 : 37296 aida02 : 19 8 11 2 3 4 2 2 3 3 6 : 36412 aida03 : 14 6 3 4 1 3 1 3 3 3 6 : 36312 aida04 : 25 12 12 4 3 4 1 4 3 3 6 : 37316 aida05 : 4 8 3 4 1 2 2 4 3 3 6 : 36928 aida06 : 42 12 8 7 2 3 3 4 3 3 6 : 37736 aida07 : 24 6 7 3 3 3 1 3 2 2 7 : 37536 aida08 : 10 15 7 2 1 4 2 3 3 3 6 : 36752 aida09 : 29 13 6 6 4 3 3 3 3 3 6 : 37244 aida10 : 18 12 7 2 2 3 3 2 2 4 6 : 37464 aida11 : 20 8 0 5 5 4 2 3 3 3 6 : 36976 aida12 : 7 2 4 3 2 3 2 2 4 3 6 : 37068 aida13 : 2 5 5 4 2 3 2 3 3 3 6 : 36608 aida14 : 9 13 8 2 2 2 3 3 2 4 6 : 37836 aida15 : 24 12 12 1 0 3 2 3 3 3 6 : 36640 aida16 : 12 9 9 2 1 3 2 4 3 3 6 : 37128 6:45 System check Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9 System wide check are all ok except: ADC calibration all failed FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 11 3 5 0 6 1 2 3 2 7 : 38264 aida02 : 14 9 9 2 3 3 3 2 3 3 6 : 36496 aida03 : 13 7 3 5 2 3 2 3 3 3 6 : 36668 aida04 : 30 15 13 4 3 4 1 3 4 3 6 : 37888 aida05 : 17 9 3 3 1 2 2 4 3 3 6 : 36956 aida06 : 40 16 5 6 2 3 2 3 2 4 6 : 37936 aida07 : 3 2 3 2 3 4 2 2 2 2 7 : 37196 aida08 : 17 10 6 2 1 3 3 3 3 3 6 : 36852 aida09 : 29 11 7 6 3 3 2 4 3 3 6 : 37436 aida10 : 1 10 5 1 1 3 3 2 2 4 6 : 37252 aida11 : 30 9 0 5 4 4 2 3 3 3 6 : 36960 aida12 : 19 10 5 3 2 3 2 2 4 3 6 : 37196 aida13 : 22 9 6 3 2 3 3 3 3 3 6 : 36960 aida14 : 19 6 9 2 2 3 3 2 2 4 6 : 37452 aida15 : 6 8 11 2 1 3 2 3 3 3 6 : 36616 aida16 : 19 14 5 3 1 3 1 4 3 3 6 : 36908 07:33 system check Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13 System wide check all ok except ADC all failed White rabbit 1 failed Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR memory information Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 9 4 6 0 5 1 2 2 2 7 : 37152 aida02 : 19 6 13 1 3 3 4 2 3 3 6 : 36780 aida03 : 23 5 6 4 2 3 1 3 3 3 6 : 36452 aida04 : 24 11 14 3 3 4 2 3 3 3 6 : 37048 aida05 : 17 3 6 3 1 2 2 3 3 3 6 : 36444 aida06 : 36 15 4 9 2 4 2 4 3 3 6 : 37608 aida07 : 20 8 5 3 3 4 2 2 2 2 7 : 37376 aida08 : 21 11 10 2 1 4 2 2 2 4 6 : 37324 aida09 : 27 14 7 6 3 3 2 4 3 3 6 : 37452 aida10 : 25 9 5 2 2 3 3 2 2 4 6 : 37436 aida11 : 30 14 2 4 5 3 2 3 3 3 6 : 36936 aida12 : 22 3 5 3 2 3 2 3 3 3 6 : 36640 aida13 : 10 7 1 4 2 3 3 2 3 3 6 : 36336 aida14 : 13 9 12 2 3 3 2 3 2 4 6 : 37820 aida15 : 20 8 11 3 0 3 2 3 3 3 6 : 36640 aida16 : 23 6 6 2 0 2 1 3 4 3 6 : 37164
Attachment 1: 210520_0156_DT.png

Attachment 2: 210520_0157_Stats.png

Attachment 3: 210520_0157_Temps.png

Attachment 4: 210520_0157_Voltages.png

Attachment 5: 210520_0537_Temps.png

Attachment 6: 210520_0539_VOltages.png

Attachment 7: 210520_0540_Histo.png

Attachment 8: 210520_0541_Rates.png

Attachment 9: 210520_0542_DT.png

Attachment 10: 210520_0730_DT.png

Attachment 11: 210520_0730_Voltages.png

Attachment 12: 210520_0731_Stats.png

Attachment 13: 210520_0731_Tems.png

	334	Wed May 19 20:02:27 2021	NH	AIDA Dead Time Analysis
Some plots of AIDA dead time (From PAUSE/RESUME data items) against time. The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead. Beam spills are indicated by the red bars (1 = on spill) Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates . The much quieter p+n strips show a similar pattern but much less frequently. Deadtime does not always sync up between FEE64s it seems. IN all case the pause/resume interval seems to only last for less than 100 ms.
Attachment 1: dt_fee2.png

Attachment 2: dt_fee4.png

Attachment 3: dt_fee6.png

Attachment 4: dt_fee8.png

Attachment 5: dt_fee9.png

Attachment 6: dt_fee9_lr.png

	333	Wed May 19 18:50:01 2021	LS	Wednesday 19th May 20:00 to 24:00
20.00 Stats okay (Fig 1) Analysed R14_762 deadtimes are okay (Fig 2) 20.30 Stats okay (Fig 3) Analysed R14_771 deadtimes are okay (Fig 4) 21.00 Stats okay (Fig 5) FEE temps okay (Fig 6) Bias (Fig 7) Rate spectra (Fig 8) System wide checks okay except white rabbit 1 fail: Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~2.8M items/s Tape server ~10MB/s Analysed R14_781 deadtimes are okay (Fig 9) 21.30 Stats okay (Fig 10) Analysed R14_789 deadtimes are okay (Fig 11) 22.00 Stats okay (Fig 12) Analysed R14_799 deadtimes are okay (Fig 13) 22.30 Stats okay (Fig 14) Analysed R14_807 deadtimes are okay (Fig 15) 23.00 Stats okay (Fig 16) FEE temps okay (Fig 17) Bias (Fig 18) System wide checks okay except white rabbit 1 fail: Base Current Difference aida05 fault 0x500 : 0x569 : 105 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~3M items/s Tape server ~10MB/s Analysed R14_819 deadtimes are okay (Fig 19) 23.30 Stats okay (Fig 20) Analysed R14_827 deadtimes are okay (Fig 21)
Attachment 1: 210519_Stats_2000.png

Attachment 2: Deadtimes_R14_762.png

Attachment 3: 210519_Stats_2030.png

Attachment 4: Deadtimes_R14_771.png

Attachment 5: 210519_Stats_2100.png

Attachment 6: 210519_FEEtemps_2100.png

Attachment 7: 210519_BIAS_2100.png

Attachment 8: 210519_Spectra_2100.png

Attachment 9: deadtimes_R14_781.png

Attachment 10: 210519_Stats_2130.png

Attachment 11: deadtimes_R14_789.png

Attachment 12: 210519_Stats_2200.png

Attachment 13: deadtimes_R14_799.png

Attachment 14: 210519_Stats_2230.png

Attachment 15: deatimes_R14_807.png

Attachment 16: 210519_Stats_2300.png

Attachment 17: 210519_FEEtemps_2300.png

Attachment 18: 210519_bias_2300.png

Attachment 19: deatimes_R14_819.png

Attachment 20: 210519_Stats_2330.png

Attachment 21: deadtimes_R14_827.png

	332	Wed May 19 14:54:36 2021	CB	Wednesday 19 May 16:00 - 20:00
15:55 Took over from OH 16:00 Stats OK - attach 1 Temps OK - attach 2 Bias OK - attach 3 System wide checks All OK, except ADC all fail (as before) and WR Base Current Difference aida05 fault 0x500 : 0x567 : 103 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Analysed R14_691. Attach 4. Dead time <5% 16:34 MBS changes file. 17:00 Stats OK - attach 5 Temps OK - as before System wide checks as before Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 17:55 Stats OK - attach 6 Temps OK - as before Leakage current approaching 10 uA - attach 7 System-wide checks - as before except Base Current Difference aida05 fault 0x500 : 0x568 : 104 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:01 MBS switches to new file 18:52 Stats OK - attach 8 Temps OK - as before System-wide checks as before Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9 19:36 Stats OK - attach 10 Bias - attach 11 19:48 Shift handed over to LS
Attachment 1: Stats1.png

Attachment 2: Temps1.png

Attachment 3: Bias1.png

Attachment 4: Analysis_R14_691
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 251761346 ( 995725.1 Hz) Other data format: 10158654 ( 40177.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 199 ( 0.8 Hz) RESUME: 199 ( 0.8 Hz) SYNC100: 32942 ( 130.3 Hz) WR48-63: 32942 ( 130.3 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 10092372 ( 39915.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 503210 ( 1990.2 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 252.842 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.053 17.526 1 1.803 0.000 2 0.604 2.918 3 3.773 0.000 4 0.070 17.512 5 9.601 0.000 6 0.278 0.000 7 9.149 0.000 8 0.114 0.000 9 0.000 64.964 10 0.536 0.000 11 0.070 0.000 12 0.024 6.415 13 0.000 66.391 14 0.036 7.395 15 0.097 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 6023981 3019 0 0 3 3 749 749 0 1515 0 26768 1 22620294 5924 0 0 14 14 2948 2948 0 0 0 55394 2 11082857 2607083 0 0 5 5 1799 1799 0 2603475 0 43681 3 30564809 2580290 0 0 26 26 4295 4295 0 2571648 0 65104 4 3958862 1393424 0 0 7 7 678 678 0 1392054 0 11940 5 37388332 780379 0 0 62 62 4752 4752 0 770751 0 33878 6 10191568 1631763 0 0 8 8 1480 1480 0 1628787 0 23955 7 39960208 1134610 0 0 55 55 5179 5179 0 1124142 0 42303 8 17144158 4230 0 0 3 3 2112 2112 0 0 0 56568 9 2458065 662 0 0 0 0 331 331 0 0 0 4321 10 20331742 4810 0 0 6 6 2399 2399 0 0 0 34970 11 11809904 2968 0 0 4 4 1480 1480 0 0 0 10406 12 9134013 2250 0 0 1 1 1124 1124 0 0 0 49240 13 2508032 620 0 0 0 0 310 310 0 0 0 3587 14 8541585 2182 0 0 2 2 1089 1089 0 0 0 33394 15 18042936 4440 0 0 3 3 2217 2217 0 0 0 7701 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 33.672s ( 950.348 blocks/s, 59.397 Mb/s)
Attachment 5: Stats2.png

Attachment 6: Stats3.png

Attachment 7: Bias3.png

Attachment 8: Analysis_R14_741
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 253670855 ( 1280798.3 Hz) Other data format: 8249145 ( 41650.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 236 ( 1.2 Hz) RESUME: 236 ( 1.2 Hz) SYNC100: 32737 ( 165.3 Hz) WR48-63: 32737 ( 165.3 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 8183199 ( 41317.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 413031 ( 2085.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 198.057 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.011 2.771 1 4.649 0.000 2 0.028 0.000 3 4.808 0.000 4 0.111 7.488 5 7.093 0.000 6 0.128 0.000 7 10.340 0.000 8 0.971 0.000 9 0.000 24.260 10 0.713 0.000 11 0.245 0.000 12 0.122 0.000 13 0.000 21.524 14 0.232 0.000 15 1.605 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5709345 2725 0 0 1 1 769 769 0 1185 0 22625 1 23799538 6034 0 0 27 27 2990 2990 0 0 0 44162 2 11722022 2092139 0 0 1 1 1767 1767 0 2088603 0 36240 3 29654318 2044332 0 0 38 38 3940 3940 0 2036376 0 53125 4 3438629 1158241 0 0 8 8 579 579 0 1157067 0 9655 5 33876189 636041 0 0 60 60 4396 4396 0 627129 0 27488 6 10663472 1364583 0 0 5 5 1426 1426 0 1361721 0 20341 7 38937063 921272 0 0 54 54 5023 5023 0 911118 0 34159 8 16044621 3966 0 0 9 9 1974 1974 0 0 0 46740 9 2236927 610 0 0 0 0 305 305 0 0 0 3583 10 23087850 5792 0 0 6 6 2890 2890 0 0 0 28946 11 13996775 3362 0 0 8 8 1673 1673 0 0 0 8557 12 9568717 2316 0 0 4 4 1154 1154 0 0 0 40620 13 2488330 626 0 0 0 0 313 313 0 0 0 2902 14 9604847 2392 0 0 6 6 1190 1190 0 0 0 27522 15 18842212 4714 0 0 9 9 2348 2348 0 0 0 6366 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 34.094s ( 938.588 blocks/s, 58.662 Mb/s)
Attachment 9: Stats4.png

Attachment 10: Stats6.png

Attachment 11: Bias6.png

	331	Wed May 19 11:24:55 2021	RDP	Monday 19 April 12 noon shift
System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed. FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida07 failed FEE64 module aida09 failed Calibration test result: Passed 10, Failed 2 Base Current Difference aida07 fault 0xd052 : 0xd056 : 4 White Rabbit error counter test result: Passed 11, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida12 fault 0x0 : 0x16 : 22 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 2 6 2 0 5 5 1 2 3 3 6 : 36120 aida02 : 25 10 0 4 3 5 2 3 3 3 6 : 36980 aida03 : 22 12 5 2 1 3 2 3 3 3 6 : 36616 aida04 : 16 3 3 3 4 4 4 2 3 3 6 : 36840 aida05 : 14 5 6 2 3 4 4 2 3 3 6 : 36800 aida06 : 30 10 6 3 3 4 2 3 3 3 6 : 36936 aida07 : 27 11 1 0 0 3 2 3 3 3 6 : 36436 aida08 : 38 7 7 3 3 2 2 3 3 3 6 : 36704 aida09 : 4 6 4 2 4 4 1 4 3 3 6 : 37056 aida10 : 21 10 3 3 1 5 1 3 3 3 6 : 36596 aida11 : 24 8 2 1 4 3 2 4 2 3 6 : 36192 aida12 : 31 12 4 3 5 5 2 2 3 3 6 : 36668 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1025 Last changed Mon Apr 19 10:32:22 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021 FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021 There should be a separate elog note about changing one of the bias voltages. All histograms zeroed at 16:13. 16.20 DAQ continues file NULL/R33_156 aida09 HEC fast comparator 0x2->0x1 DSSSD#3 bias -100->-120V
Attachment 1: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png

Attachment 2: Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png

Attachment 3: 62.png

	330	Wed May 19 09:12:52 2021	Muneerah, Betool, LS	17th May 00:00-12:00
Auther: Muneerah and Betool 02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok Statistics ok - 210517_0202_Stats Temperatures ok - 210517_0202_Temp Bias and leakage ok - 210517_0200_Bias 02:05 System wide checks: Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida06 failed FEE64 module aida10 failed Calibration test result: Passed 12, Failed 4 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x1552 : 0x155d : 11 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 5 8 3 3 3 5 2 2 2 3 7 : 39460 aida02 : 22 4 5 0 3 2 2 3 2 3 7 : 39560 aida03 : 21 10 2 1 1 3 2 4 2 3 7 : 40100 aida04 : 18 7 2 0 3 3 3 2 2 3 7 : 39392 aida05 : 16 6 5 1 2 4 3 2 2 3 7 : 39520 aida06 : 18 6 6 2 2 2 2 3 2 3 7 : 39576 aida07 : 12 8 6 1 2 3 1 3 2 3 7 : 39408 aida08 : 13 11 4 0 1 3 3 2 2 3 7 : 39308 aida09 : 10 8 4 2 3 4 2 3 2 3 7 : 39848 aida10 : 16 8 2 1 2 4 2 3 1 4 7 : 40768 aida11 : 26 10 2 2 1 2 2 3 2 3 7 : 39512 aida12 : 16 9 2 1 2 4 2 2 2 3 7 : 39240 aida13 : 15 9 3 2 2 4 2 2 2 3 7 : 39284 aida14 : 25 10 2 3 3 3 2 3 2 3 7 : 39796 aida15 : 4 2 2 1 1 3 2 2 2 3 7 : 38944 aida16 : 16 7 3 3 1 3 3 2 2 3 7 : 39368 04:00 Statistics ok - 210517_0403_Stats Temperatures ok - 210517_0403_Temp Bias and leakage ok - 210517_0400_Bias 04:05 System wide checks: Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida06 failed FEE64 module aida10 failed Calibration test result: Passed 12, Failed 4 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x1552 : 0x1560 : 14 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 15 7 5 2 3 5 2 2 2 3 7 : 39492 aida02 : 23 6 7 2 1 3 2 3 2 3 7 : 39676 aida03 : 17 9 3 2 0 3 2 3 2 3 7 : 39548 aida04 : 16 4 1 2 1 3 3 3 2 3 7 : 39792 aida05 : 22 7 3 3 1 4 3 3 2 3 7 : 40032 aida06 : 18 6 7 3 1 3 2 3 2 3 7 : 39688 aida07 : 18 7 4 1 2 4 1 4 2 3 7 : 40032 aida08 : 19 5 4 1 1 3 3 3 2 3 7 : 39828 aida09 : 10 6 7 1 3 4 2 2 2 3 7 : 39336 aida10 : 12 10 2 3 1 4 2 3 1 4 7 : 40768 aida11 : 10 9 4 3 2 1 2 3 2 3 7 : 39440 aida12 : 8 7 6 1 2 4 2 3 2 3 7 : 39768 aida13 : 11 2 6 1 2 3 2 3 2 3 7 : 39612 aida14 : 27 7 4 2 3 3 2 3 2 3 7 : 39780 aida15 : 4 2 3 1 1 3 2 2 2 3 7 : 38960 aida16 : 18 8 4 2 1 4 2 2 2 3 7 : 39240 06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA 06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again. 07:01 it is working fine. 07:40 Screen shots after the restart Statistics ok - 210517_0742_Stats Temperatures ok - 210517_0743_Temp Bias and leakage ok - 210517_0740_Bias System wide checks FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x61da : 0x61dd : 3 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 6 3 1 1 3 1 3 3 3 7 : 40336 aida02 : 11 6 0 1 1 5 2 4 2 3 7 : 40252 aida03 : 0 5 2 3 3 3 3 4 2 3 7 : 40424 aida04 : 2 4 2 1 2 4 3 3 2 3 7 : 39912 aida05 : 11 7 5 1 2 2 4 2 3 3 7 : 40532 aida06 : 20 6 0 2 3 2 1 3 3 3 7 : 40320 aida07 : 15 5 1 2 2 3 2 2 3 3 7 : 40116 aida08 : 2 3 4 1 3 3 2 2 3 3 7 : 40128 aida09 : 18 5 1 3 2 4 2 2 3 3 7 : 40288 aida10 : 11 6 3 4 2 3 3 2 2 4 7 : 41484 aida11 : 4 4 5 2 2 2 3 3 2 3 7 : 39744 aida12 : 18 7 2 3 2 2 4 3 2 3 7 : 40064 aida13 : 2 2 1 3 0 3 2 4 2 3 7 : 39944 aida14 : 16 4 5 3 3 3 3 2 2 4 7 : 41552 aida15 : 9 3 1 1 2 3 2 3 3 3 7 : 40556 aida16 : 16 5 1 2 1 2 3 4 2 3 7 : 40184 Date:17/05/2021 Elog for shift: 08.00 to 12.00 Author: Lewis Sexton Screenshots saved to: /tmp/Elog_210517_0800_1200 *** 08.30 * Checked R12_34 through analyser, dead time seems okay: * Timestamp elapsed time: 232.154 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.021 0.000 1 2.781 0.000 2 0.849 0.000 3 5.686 0.000 4 0.000 25.816 5 0.017 0.000 6 0.177 0.000 7 1.349 0.000 8 1.791 0.000 9 0.000 91.660 10 6.194 0.000 11 0.320 0.000 12 1.425 0.000 13 0.000 72.999 14 0.052 0.000 15 1.729 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0833_Stats.png) Merger rate ~3M items/s Tape server ~12 MB/s *** 09.00 * Checked R12_48 through analyser, dead time seems okay: * Timestamp elapsed time: 170.459 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.196 0.000 1 6.294 0.000 2 0.992 0.000 3 8.944 0.000 4 0.024 0.000 5 0.001 0.000 6 0.605 0.000 7 2.096 0.000 8 1.654 0.000 9 0.000 31.395 10 6.175 0.000 11 2.524 0.000 12 2.076 0.000 13 0.000 10.337 14 0.103 0.000 15 2.203 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0902_Stats.png) *** 09.30 * AIDA crash aida05 displaying 0 in stats Power cycle performed with help of TD Writing to R13 now MBS relay restared AIDA back up at 10.20 Ran analyser for R13_16 * Timestamp elapsed time: 168.059 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.047 0.000 1 2.282 0.000 2 0.080 0.000 3 0.806 0.000 4 0.000 0.000 5 0.260 0.000 6 0.030 0.000 7 1.471 0.000 8 67.467 0.000 9 80.552 0.000 10 0.057 0.000 11 0.000 11.671 12 1.191 0.000 13 0.080 0.000 14 0.091 0.000 15 0.202 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc) Ran analyser for R13_21 for comparision * Timestamp elapsed time: 149.082 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 3.580 0.000 2 0.000 0.000 3 2.800 0.000 4 0.000 0.000 5 0.514 0.000 6 0.140 0.000 7 1.372 0.000 8 55.544 0.000 9 66.588 0.000 10 1.065 0.000 11 0.000 10.010 12 0.681 0.000 13 0.044 0.000 14 0.070 0.000 15 0.400 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png) Merger rate ~5M items/s Tape server ~17 MB/s Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision * Timestamp elapsed time: 122.147 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.027 0.000 1 19.945 0.000 2 0.601 0.000 3 21.555 0.000 4 0.042 0.000 5 3.226 0.000 6 1.211 0.000 7 9.674 0.000 8 52.817 0.000 9 54.972 0.000 10 1.848 0.000 11 0.008 0.000 12 2.034 0.000 13 0.278 0.000 14 0.018 0.000 15 0.634 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 high deatime now also seen in aida02 and aida04 Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs Back running as of 10.45 Analyser of R14_15 to double check * Timestamp elapsed time: 205.006 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 40.176 1 0.438 0.000 2 0.199 0.000 3 2.927 0.000 4 0.000 140.347 5 1.282 0.000 6 0.150 0.000 7 2.097 0.000 8 1.673 0.000 9 0.000 87.749 10 1.150 0.000 11 0.489 0.000 12 0.374 0.000 13 0.048 20.995 14 0.251 0.000 15 2.441 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs Date:17/05/2021 Elog for shift: 08.00 to 12.00 Author: Lewis Sexton Screenshots saved to: /tmp/Elog_210517_0800_1200 * 08.30 * Checked R12_34 through analyser, dead time seems okay: * Timestamp elapsed time: 232.154 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.021 0.000 1 2.781 0.000 2 0.849 0.000 3 5.686 0.000 4 0.000 25.816 5 0.017 0.000 6 0.177 0.000 7 1.349 0.000 8 1.791 0.000 9 0.000 91.660 10 6.194 0.000 11 0.320 0.000 12 1.425 0.000 13 0.000 72.999 14 0.052 0.000 15 1.729 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0833_Stats.png) Merger rate ~3M items/s Tape server ~12 MB/s *** 09.00 * Checked R12_48 through analyser, dead time seems okay: * Timestamp elapsed time: 170.459 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.196 0.000 1 6.294 0.000 2 0.992 0.000 3 8.944 0.000 4 0.024 0.000 5 0.001 0.000 6 0.605 0.000 7 2.096 0.000 8 1.654 0.000 9 0.000 31.395 10 6.175 0.000 11 2.524 0.000 12 2.076 0.000 13 0.000 10.337 14 0.103 0.000 15 2.203 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 Stats checked seem okay (0902_Stats.png) *** 09.30 * AIDA crash aida05 displaying 0 in stats Power cycle performed with help of TD Writing to R13 now MBS relay restared AIDA back up at 10.20 Ran analyser for R13_16 * Timestamp elapsed time: 168.059 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.047 0.000 1 2.282 0.000 2 0.080 0.000 3 0.806 0.000 4 0.000 0.000 5 0.260 0.000 6 0.030 0.000 7 1.471 0.000 8 67.467 0.000 9 80.552 0.000 10 0.057 0.000 11 0.000 11.671 12 1.191 0.000 13 0.080 0.000 14 0.091 0.000 15 0.202 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc) Ran analyser for R13_21 for comparision * Timestamp elapsed time: 149.082 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 3.580 0.000 2 0.000 0.000 3 2.800 0.000 4 0.000 0.000 5 0.514 0.000 6 0.140 0.000 7 1.372 0.000 8 55.544 0.000 9 66.588 0.000 10 1.065 0.000 11 0.000 10.010 12 0.681 0.000 13 0.044 0.000 14 0.070 0.000 15 0.400 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png) Merger rate ~5M items/s Tape server ~17 MB/s Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision * Timestamp elapsed time: 122.147 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.027 0.000 1 19.945 0.000 2 0.601 0.000 3 21.555 0.000 4 0.042 0.000 5 3.226 0.000 6 1.211 0.000 7 9.674 0.000 8 52.817 0.000 9 54.972 0.000 10 1.848 0.000 11 0.008 0.000 12 2.034 0.000 13 0.278 0.000 14 0.018 0.000 15 0.634 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 high deatime now also seen in aida02 and aida04 Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs Back running as of 10.45 Analyser of R14_15 to double check * Timestamp elapsed time: 205.006 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 40.176 1 0.438 0.000 2 0.199 0.000 3 2.927 0.000 4 0.000 140.347 5 1.282 0.000 6 0.150 0.000 7 2.097 0.000 8 1.673 0.000 9 0.000 87.749 10 1.150 0.000 11 0.489 0.000 12 0.374 0.000 13 0.048 20.995 14 0.251 0.000 15 2.441 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs 12:52 Check FPGA Timestamp errors gives following error message: Got the error Server Internal Error while trying to obtain /AIDA/Check/Check.tml. can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]" Tcl Call Trace 6: DocSubstSystemFile sock8 error 500 can't\ read\ &quot\;TS_Base_Collected&quot\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n&quot\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}&quot\;\n\ \ \ \ (procedure\ &quot\;do_TS_errors&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;do_TS_errors\ &quot\;\n\ \ \ \ (&quot\;TSERRORS&quot\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...&quot\;\n\ \ \ \ (procedure\ &quot\;do_click&quot\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n&quot\;do_click&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ &quot\;&quot\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ &quot\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...&quot\;\n\ \ \ \ (file\ &quot\;Check.tcl&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;source\ Check.tcl&quot\;\n\ \ \ \ (in\ namespace\ eval\ &quot\;::Check&quot\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n&quot\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n&lt\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n&gt\;...&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ #0\ [list\ subst\ \$script\]&quot\;\n\ \ \ \ (procedure\ &quot\;SubstFile&quot\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n&quot\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]&quot\;\n\ \ \ \ (procedure\ &quot\;Subst_File&quot\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n&quot\;Subst_File\ \$template\ \$interp&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_application/x-tcl-template&quot\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n&quot\;\$cmd\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_Handle&quot\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n&quot\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;DocDomain&quot\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n&quot\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml&quot\;\n\ \ \ \ (&quot\;eval&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]&quot\; 5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" 4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE 3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {} 2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml} 1: HttpdReadPostGlobal sock8 Httpds Dead time check: [npg@aidas-gsi S496]$ ~/analyser/analyser R14_24 * TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 252995858 ( 451517.9 Hz) Other data format: 8924143 ( 15926.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 85 ( 0.2 Hz) RESUME: 85 ( 0.2 Hz) SYNC100: 34087 ( 60.8 Hz) WR48-63: 34087 ( 60.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 8855799 ( 15804.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 560.323 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 349.960 1 0.002 0.000 2 0.000 5.974 3 1.105 0.000 4 0.000 447.454 5 0.000 2.809 6 0.000 16.727 7 0.137 0.000 8 0.090 0.000 9 0.000 429.897 10 0.461 0.000 11 0.018 0.000 12 0.000 8.985 13 0.000 466.311 14 0.000 15.095 15 0.346 0.000 13:24 Beam is off at the moment... 13:43 Changing spill structure from 1.5s/3.5s to 1s. 13:50 [npg@aidas-gsi S496]$ ~/analyser/analyser R14_29 * TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 251763810 ( 930342.1 Hz) Other data format: 10156190 ( 37530.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 205 ( 0.8 Hz) RESUME: 205 ( 0.8 Hz) SYNC100: 33009 ( 122.0 Hz) WR48-63: 33009 ( 122.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 10089762 ( 37284.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 527077 ( 1947.7 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) *** Timestamp elapsed time: 270.614 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.034 6.803 1 8.461 0.000 2 0.569 0.000 3 11.664 0.000 4 0.024 26.416 5 0.854 0.000 6 0.069 0.000 7 0.741 0.000 8 1.336 0.000 9 0.000 75.926 10 0.353 0.000 11 0.105 0.000 12 0.070 0.000 13 0.000 93.585 14 0.017 0.000 15 0.121 0.000 15:06 Spill structure back to 1.5 on 2s off.
Attachment 1: 210517_0200_Bias.png

Attachment 2: 210517_0202_stats.png

Attachment 3: 210517_0202_temp.png

Attachment 4: 210517_0400_Bias.png

Attachment 5: 210517_0403_stats.png

Attachment 6: 210517_0403_temp.png

Attachment 7: 210517_0740_Bias.png

Attachment 8: 210517_0742_Stats.png

Attachment 9: 210517_0743_Temp.png

Attachment 10: 0833_Stats.png

Attachment 11: 0902_Stats.png

Attachment 12: 1241_Stats.png

Attachment 13: 1347_Stats.png

Attachment 14: FEETemps_1042.png

Attachment 15: FEETemps_1246.png

Attachment 16: FEETemps_1350.png

Attachment 17: leakage_currents_1302.png

	329	Wed May 19 09:09:27 2021	MS, JS, OH	May 16 08:00-24:00
16th May 08:00 - 12:00 shift Author: MS 08:00 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x7163 : 23 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0xa : 10 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 27 7 2 3 2 3 2 4 2 3 7 : 40228 aida02 : 3 8 3 2 2 2 2 4 2 3 7 : 39996 aida03 : 23 9 6 1 0 3 2 4 2 3 7 : 40100 aida04 : 34 27 17 7 2 4 3 3 2 2 7 : 38608 aida05 : 19 9 5 2 2 2 3 3 2 3 7 : 39844 aida06 : 5 5 2 1 0 4 2 4 2 3 7 : 40060 aida07 : 28 2 10 3 2 4 1 4 2 3 7 : 40192 aida08 : 19 5 5 1 2 5 1 3 2 3 7 : 39652 aida09 : 20 8 3 3 1 3 2 3 2 3 7 : 39648 aida10 : 27 10 0 2 2 2 2 3 1 4 7 : 40572 aida11 : 3 3 2 1 2 4 2 3 2 3 7 : 39652 aida12 : 16 7 11 2 3 4 2 2 2 3 7 : 39464 aida13 : 14 10 4 3 1 5 2 2 2 3 7 : 39400 aida14 : 21 9 10 1 1 2 2 3 1 4 7 : 40604 aida15 : 19 8 2 3 0 4 3 2 2 3 7 : 39436 aida16 : 24 5 8 3 3 4 2 2 2 3 7 : 39464 * Timestamp elapsed time: 225.065 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.038 0.000 1 9.479 0.000 2 0.195 0.000 3 5.921 0.000 4 0.000 11.742 5 0.036 0.000 6 0.013 0.000 7 0.498 0.000 8 0.436 0.000 9 0.000 107.300 10 2.787 0.000 11 0.905 0.000 12 0.831 0.000 13 0.000 55.939 14 0.080 0.000 15 0.267 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 10:00 FEE64 module aida06 global clocks failed, 6 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 14, Failed 2 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x7166 : 26 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0xa : 10 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 7 6 2 2 3 2 3 2 3 7 : 39716 aida02 : 8 4 3 2 2 3 2 3 2 3 7 : 39600 aida03 : 23 11 5 1 0 4 2 3 1 4 7 : 40740 aida04 : 42 25 16 7 2 4 4 3 2 2 7 : 38864 aida05 : 24 5 5 1 2 2 3 3 1 4 7 : 40824 aida06 : 11 4 4 1 1 4 2 4 2 3 7 : 40172 aida07 : 21 8 5 1 2 3 2 4 2 3 7 : 40196 aida08 : 15 5 6 1 1 4 2 4 2 3 7 : 40228 aida09 : 15 10 4 1 2 3 2 3 2 3 7 : 39660 aida10 : 23 8 2 2 2 2 2 3 1 4 7 : 40572 aida11 : 6 4 4 2 2 4 2 3 2 3 7 : 39736 aida12 : 18 8 8 1 4 4 3 2 2 3 7 : 39720 aida13 : 23 6 2 3 2 5 2 2 2 3 7 : 39436 aida14 : 29 3 11 1 1 2 2 3 1 4 7 : 40604 aida15 : 29 7 2 2 0 4 3 3 2 3 7 : 39948 aida16 : 2 3 6 2 3 4 2 2 2 3 7 : 39296 * Timestamp elapsed time: 225.065 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.038 0.000 1 9.479 0.000 2 0.195 0.000 3 5.921 0.000 4 0.000 11.742 5 0.036 0.000 6 0.013 0.000 7 0.498 0.000 8 0.436 0.000 9 0.000 107.300 10 2.787 0.000 11 0.905 0.000 12 0.831 0.000 13 0.000 55.939 14 0.080 0.000 15 0.267 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 12:00-16:00 16th May 12:00 - 16:00 shift Author: JS 11:57 Taking over from Magda. Running full checks. usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75% Current ok 06.410 uA 006.835 uA Stats good 1Statistics aidas-gsi(6).png Temps ok 1Temperature and status scan aidas-gsi(6).png Analysis ok R7_385. Dead time FEE1 a little hight 6% PAUSE: 166 RESUME: 166 *** Timestamp elapsed time: 196.305 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.044 0.000 1 12.405 0.000 2 0.807 0.000 3 7.260 0.000 4 0.014 0.000 5 0.458 0.000 6 0.027 0.000 7 0.497 0.000 8 0.723 0.000 9 0.000 88.857 10 6.189 0.000 11 1.443 0.000 12 0.474 0.000 13 0.000 35.565 14 0.000 0.000 15 0.147 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 15, Failed 1 FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x716e : 34 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Base Current Difference aida05 fault 0x0 : 0xa : 10 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 7 4 2 2 3 2 3 1 4 7 : 40712 aida02 : 25 10 7 1 1 4 1 3 2 3 7 : 39556 aida03 : 22 10 5 1 0 4 2 4 2 3 7 : 40216 aida04 : 40 24 17 7 2 5 3 3 2 2 7 : 38736 aida05 : 4 8 3 0 1 3 3 3 1 4 7 : 40768 aida06 : 21 6 6 2 2 3 2 4 2 3 7 : 40228 aida07 : 25 4 6 3 2 3 2 4 2 3 7 : 40260 aida08 : 19 9 4 1 1 4 2 4 2 3 7 : 40244 aida09 : 16 9 4 2 2 3 2 3 2 3 7 : 39688 aida10 : 19 10 5 1 2 2 2 4 1 4 7 : 41100 aida11 : 21 10 2 1 2 3 3 3 2 3 7 : 39908 aida12 : 25 7 7 2 2 3 2 3 2 3 7 : 39756 aida13 : 13 7 3 4 2 5 2 3 2 3 7 : 39964 aida14 : 21 7 9 2 1 2 2 4 1 4 7 : 41116 aida15 : 23 6 2 3 0 4 3 3 2 3 7 : 39948 aida16 : 9 11 10 2 3 4 2 2 2 3 7 : 39452 Tom says Aida09 clock fail is ok as its status bit is "6". The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken. 12:35 - usbec ok. Current ok Stats good Temps ok Analysis ok R7_395. Dead time FEE1 still high 6.6% 13:33 - usbec ok. Max implants ~ 1.8kHz Current ok 006.850 uA 007.250 uA Stats - Aida11 runing low < 5k was ~20k overnight Temps ok Analysis R7_415. Dead time FEE1 & FEE10 high 10% 14:00 usbec ok - ucesb1.png Current ok - bias1.png Stast - Aida11 low - 14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n. 15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool. 18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00. During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise. The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs. We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy. During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips. 18:08 System wide checks all ok - bar some ADC but waveforms disabled Statistics ok - 210516_1809_Stats Temperatures ok - 210516_1809_Temp Bias and leakage ok - 210516_1810_Bias 18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time 18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side. 19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves. Now have around 4.2TB left which will provide around 80 hours of writing 20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not. 20:57 System wide checks. Clock ok Base Current Difference aida05 fault 0x1552 : 0x1556 : 4 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Statistics ok - 210516_2056_Stats Temp ok - 210516_2058_Temp Bias and leakage current ok - 210516_2058_Bias 23:16 System wide checks: Clock still ok Base Current Difference aida05 fault 0x1552 : 0x155a : 8 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Statistics - 210516_2315_Stats Temperature - 210516_2316_Temp Bias and leakage current ok - 210516_231
Attachment 1: Screenshot_2021-05-16_Statistics_aidas-gsi(4).png

Attachment 2: Screenshot_2021-05-16_Statistics_aidas-gsi(5).png

Attachment 3: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png

Attachment 4: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png

Attachment 5: 1Statistics_aidas-gsi(6).png

Attachment 6: 1Temperature_and_status_scan_aidas-gsi(6).png

Attachment 7: Bias1.png

Attachment 8: 210516_0857_Temp.png

Attachment 9: 210516_1554_Layout8.png

Attachment 10: 210516_1555_Layout7.png

Attachment 11: 210516_1600_Layout1.png

Attachment 12: 210516_1601_Stats.png

Attachment 13: 210516_1605_1606.png

Attachment 14: 210516_1808_Stats.png

Attachment 15: 210516_1809_Temp.png

Attachment 16: 210516_1810_Bias.png

Attachment 17: 210516_2056_Stats.png

Attachment 18: 210516_2058_Bias.png

Attachment 19: 210516_2313_Stats.png

Attachment 20: 210516_2315_Temp.png

Attachment 21: 210516_2316_Bias.png

	328	Wed May 19 09:06:58 2021	CA	May 16th 00:00 - 08:00
16th May 2021 - 00:00 - 08:00 Shift Author: CA 00:15 System wide checks; FEE64 module aida06 global clocks failed, 6 FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 14, Failed 2 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida09 failed Calibration test result: Passed 15, Failed 1 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x714c : 0x7154 : 8 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 10 3 1 2 3 2 3 2 3 7 : 39660 aida02 : 29 5 3 1 1 3 2 3 2 3 7 : 39596 aida03 : 15 7 1 1 2 3 2 3 2 3 7 : 39588 aida04 : 44 26 19 5 2 4 3 3 2 2 7 : 38608 aida05 : 20 10 3 0 1 4 2 2 3 3 7 : 40208 aida06 : 27 6 2 1 2 4 2 4 2 3 7 : 40284 aida07 : 25 5 3 1 2 3 2 3 2 3 7 : 39644 aida08 : 23 8 2 3 1 4 2 3 2 3 7 : 39772 aida09 : 23 10 3 2 3 4 1 3 2 3 7 : 39644 aida10 : 18 10 3 2 3 2 2 2 2 4 7 : 41160 aida11 : 16 7 3 2 1 2 2 3 2 3 7 : 39464 aida12 : 21 4 7 1 3 3 3 3 2 3 7 : 40004 aida13 : 25 6 3 2 1 3 1 4 2 3 7 : 39876 aida14 : 20 8 10 1 1 2 2 2 2 4 7 : 41104 aida15 : 26 6 3 3 0 4 3 2 2 3 7 : 39464 aida16 : 9 8 2 4 1 3 1 4 2 3 7 : 39876 00:30 Stats ok - 210516_0030_stats.png Temps ok - 210516_0030_temps.png Bias ok - 210516_0030_bias.png 00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt 02:21 system wide checks as above, except; Base Current Difference aida05 fault 0x714c : 0x7159 : 13 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x7 : 7 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 02:28 Stats ok - 210516_0230_stats.png Temps ok - 210516_0230_temps.png Bias ok - 210516_0230_bias.png 02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt 04:36 system wide checks same as above, except; Base Current Difference aida05 fault 0x714c : 0x715c : 16 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x9 : 9 aida12 fault 0x0 : 0x3 : 3 aida13 fault 0x0 : 0x4d : 77 FPGA Timestamp error counter test result: Passed 13, Failed 3 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 04:37 Stats ok - 210516_0430_stats.png Temps ok - 210516_0430_temps.png Bias ok - 210516_0430_bias.png 04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt 06:38 system wide checks same as before, except; Base Current Difference aida01 fault 0xf294 : 0xf296 : 2 aida02 fault 0xd8ec : 0xd8ee : 2 aida03 fault 0xf001 : 0xf003 : 2 aida04 fault 0xd992 : 0xd994 : 2 aida05 fault 0x714c : 0x7161 : 21 aida06 fault 0x5a49 : 0x5a4a : 1 aida07 fault 0x5aca : 0x5acb : 1 aida08 fault 0xb92e : 0xb92f : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 06:40 Stats ok - 210516_0640_stats.png Temps ok - 210516_0640_temps.png Bias ok - 210516_0640_bias.png 06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt
Attachment 1: 210516_0030_bias.png

Attachment 2: 210516_0030_stats.png

Attachment 3: 210516_0030_temps.png

Attachment 4: 210516_0230_bias.png

Attachment 5: 210516_0230_stats.png

Attachment 6: 210516_0230_temps.png

Attachment 7: 210516_0430_bias.png

Attachment 8: 210516_0430_stats.png

Attachment 9: 210516_0430_temps.png

Attachment 10: 210516_0640_bias.png

Attachment 11: 210516_0640_stats.png

Attachment 12: 210516_0640_temps.png.png

Attachment 13: analysis.txt
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 255660963 ( 1413413.6 Hz) Other data format: 6259037 ( 34602.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 192 ( 1.1 Hz) RESUME: 192 ( 1.1 Hz) SYNC100: 32674 ( 180.6 Hz) WR48-63: 32674 ( 180.6 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 6193305 ( 34239.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 287054 ( 1587.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 180.882 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.117 0.000 1 8.836 0.000 2 1.027 0.000 3 5.545 0.000 4 0.000 5.604 5 0.873 0.000 6 0.370 0.000 7 0.861 0.000 8 0.767 0.000 9 0.000 63.434 10 13.201 0.000 11 3.930 0.000 12 0.813 0.000 13 0.000 30.131 14 0.448 0.000 15 1.895 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 9469697 3614 0 0 3 3 1264 1264 0 1080 0 15256 1 23268933 6078 0 0 41 41 2998 2998 0 0 0 31257 2 14135373 1756135 0 0 5 5 1968 1968 0 1752189 0 25654 3 26646926 1718468 0 0 28 28 3507 3507 0 1711398 0 36818 4 2828773 749772 0 0 0 0 426 426 0 748920 0 7104 5 14183721 453202 0 0 5 5 1830 1830 0 449532 0 20813 6 11921192 886700 0 0 5 5 1583 1583 0 883524 0 13410 7 19791515 651950 0 0 5 5 2639 2639 0 646662 0 25026 8 22244739 5464 0 0 4 4 2728 2728 0 0 0 32840 9 1601175 378 0 0 0 0 189 189 0 0 0 2522 10 41869813 10332 0 0 63 63 5103 5103 0 0 0 18083 11 18332381 4602 0 0 16 16 2285 2285 0 0 0 5732 12 14937821 3786 0 0 7 7 1886 1886 0 0 0 28026 13 2220427 536 0 0 0 0 268 268 0 0 0 2121 14 11574320 2908 0 0 3 3 1451 1451 0 0 0 17899 15 20634157 5112 0 0 7 7 2549 2549 0 0 0 4493 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 47.672s ( 671.255 blocks/s, 41.953 Mb/s)
Attachment 14: analysis.txt
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 255713262 ( 1398395.0 Hz) Other data format: 6206738 ( 33942.2 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 185 ( 1.0 Hz) RESUME: 185 ( 1.0 Hz) SYNC100: 32681 ( 178.7 Hz) WR48-63: 32681 ( 178.7 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 6141006 ( 33582.7 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 279005 ( 1525.8 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 182.862 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.488 0.000 1 8.465 0.000 2 0.216 0.000 3 7.120 0.000 4 0.011 8.896 5 0.764 0.000 6 0.401 0.000 7 1.395 0.000 8 0.721 0.000 9 0.000 86.292 10 11.881 0.000 11 1.912 0.000 12 0.187 0.000 13 0.000 15.181 14 0.528 0.000 15 0.095 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 9424415 3624 0 0 2 2 1264 1264 0 1092 0 14964 1 23109905 5922 0 0 39 39 2922 2922 0 0 0 29876 2 14322958 1782809 0 0 3 3 2029 2029 0 1778745 0 25108 3 26234410 1716299 0 0 35 35 3458 3458 0 1709313 0 35100 4 2817740 734656 0 0 1 1 433 433 0 733788 0 6814 5 13978921 443934 0 0 4 4 1871 1871 0 440184 0 20242 6 11924553 862150 0 0 4 4 1621 1621 0 858900 0 13000 7 19324336 623898 0 0 11 11 2446 2446 0 618984 0 24043 8 21963572 5366 0 0 5 5 2678 2678 0 0 0 32304 9 1580877 372 0 0 0 0 186 186 0 0 0 2567 10 42874488 10618 0 0 59 59 5250 5250 0 0 0 17549 11 18504188 4566 0 0 14 14 2269 2269 0 0 0 5838 12 14580871 3684 0 0 2 2 1840 1840 0 0 0 27687 13 2240991 612 0 0 0 0 306 306 0 0 0 2138 14 11530867 2858 0 0 4 4 1425 1425 0 0 0 17152 15 21300170 5370 0 0 2 2 2683 2683 0 0 0 4623 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 45.405s ( 704.764 blocks/s, 44.048 Mb/s)
Attachment 15: analysis.txt
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 255591110 ( 1400708.0 Hz) Other data format: 6328890 ( 34684.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 179 ( 1.0 Hz) RESUME: 179 ( 1.0 Hz) SYNC100: 32680 ( 179.1 Hz) WR48-63: 32680 ( 179.1 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 6263172 ( 34323.9 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 288458 ( 1580.8 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 182.473 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.000 0.000 1 7.649 0.000 2 0.265 0.000 3 5.929 0.000 4 0.000 21.293 5 1.614 0.000 6 0.390 0.000 7 0.004 0.000 8 1.749 0.000 9 0.000 75.748 10 12.608 0.000 11 1.662 0.000 12 0.334 0.000 13 0.000 39.700 14 0.092 0.000 15 1.009 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 9481493 3580 0 0 0 0 1244 1244 0 1092 0 15504 1 23234003 5924 0 0 42 42 2920 2920 0 0 0 30950 2 14360490 1787895 0 0 2 2 2026 2026 0 1783839 0 26238 3 26580599 1734745 0 0 32 32 3537 3537 0 1727607 0 36830 4 2867864 759325 0 0 0 0 449 449 0 758427 0 7075 5 13998560 454755 0 0 6 6 1806 1806 0 451131 0 20732 6 12019427 893028 0 0 3 3 1644 1644 0 889734 0 13550 7 19745451 656494 0 0 1 1 2575 2575 0 651342 0 25286 8 21649633 5478 0 0 7 7 2732 2732 0 0 0 32719 9 1602793 374 0 0 0 0 187 187 0 0 0 2586 10 42457575 10646 0 0 62 62 5261 5261 0 0 0 18276 11 18423606 4634 0 0 16 16 2301 2301 0 0 0 6050 12 14493303 3540 0 0 2 2 1768 1768 0 0 0 27878 13 2240564 512 0 0 0 0 256 256 0 0 0 2153 14 11446056 2794 0 0 2 2 1395 1395 0 0 0 17997 15 20989693 5166 0 0 4 4 2579 2579 0 0 0 4634 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 52.041s ( 614.900 blocks/s, 38.431 Mb/s)
Attachment 16: analysis.txt
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 254484788 ( 1267953.1 Hz) Other data format: 7435212 ( 37045.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 166 ( 0.8 Hz) RESUME: 166 ( 0.8 Hz) SYNC100: 32747 ( 163.2 Hz) WR48-63: 32747 ( 163.2 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 7369386 ( 36717.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 359272 ( 1790.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 200.705 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.411 0.000 1 11.545 0.000 2 1.046 0.000 3 6.943 0.000 4 0.008 8.993 5 0.438 0.000 6 0.000 0.000 7 1.702 0.000 8 0.655 0.000 9 0.000 75.734 10 6.237 0.000 11 1.119 0.000 12 0.217 0.000 13 0.000 28.278 14 0.250 0.000 15 0.105 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8401665 3348 0 0 2 2 1072 1072 0 1200 0 19564 1 26165949 6958 0 0 57 57 3422 3422 0 0 0 37826 2 13624578 2004081 0 0 4 4 1973 1973 0 2000127 0 32349 3 29939387 1947767 0 0 35 35 4004 4004 0 1939689 0 45332 4 3232848 946545 0 0 1 1 548 548 0 945447 0 8745 5 15270389 573894 0 0 6 6 1986 1986 0 569910 0 26107 6 10372112 1114231 0 0 0 0 1457 1457 0 1111317 0 17063 7 19130864 806574 0 0 8 8 2431 2431 0 801696 0 31336 8 24018310 5808 0 0 5 5 2899 2899 0 0 0 41117 9 1787485 418 0 0 0 0 209 209 0 0 0 3357 10 42125761 10654 0 0 29 29 5298 5298 0 0 0 22828 11 16187533 3960 0 0 11 11 1969 1969 0 0 0 7397 12 14311704 3490 0 0 1 1 1744 1744 0 0 0 35224 13 2388212 620 0 0 0 0 310 310 0 0 0 2664 14 9736084 2414 0 0 1 1 1206 1206 0 0 0 22553 15 17791907 4450 0 0 6 6 2219 2219 0 0 0 5810 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 43.287s ( 739.250 blocks/s, 46.203 Mb/s)
	327	Wed May 19 06:59:33 2021	OH	Wednesday 19th May 08:00 - 16:00
08:00 OH Takes over for CA Thresholds 0xc for p+n and 0x19 for n+n ASICs 1-3 and 0x64 for n+nASIC4 PULSER SETTINGS --------------------------- Pulse is ON Positive Tail Pulse Trigger Source is Internal Clock Trigger Threshold is 3.5 Amplitude : 2.0 Volts Rep Rate : 2.0 hZ Delay : 250.0 ns Fall Time : 1 ms Attenuation : 1 Display is : Volts Equivalent keV is : 200.0 Ramp Start at 0.01 Volts Ramp Stop at 9.99 Volts Ramp Start at 1.0 keV Ramp Stop at 999.0 keV Ramp Time is 60 seconds # Ramp Cycles is 1 Max average dead time in R14_586 3.2% which is 9.6% on spill Stable from yesterday 10:02 System wide checks. All ok expect WR Base Current Difference aida05 fault 0x500 : 0x554 : 84 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics - Attachment 1 Temp - attachment 2 Bias and leakage currents ok - Attachment 3 11:17 A new run was started as the microspill has been changed. I believe to optimise FRS deadtime. 11:45 They are still working on it 12:05 Beam off for a bit while they optimise spill 12:15 System wide checks ok expect WR Base Current Difference aida05 fault 0x500 : 0x567 : 103 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Statistics - Attachment 4 Temp - Attachment 5 Bias - Attachment 6 13:11 Magda has said that the changes to the spill structure only gained them a few percent in dead time Further changes could be done, but would take at least two hours 13:19 Dead time following the current spill changes R14_648 R14_649 R14_650 R14_651 Sum DT (S) Avergage DT (%) On Spill DT (%) FEE 244.823 252.168 247.155 240.421 984.567 0 0.063 0.025 0.105 0.096 0.289 0.029353005 0.088059015 1 3.981 4.907 4.17 2.484 15.542 1.578561947 4.73568584 2 0.01 0 0.121 0.779 0.91 0.092426417 0.277279251 3 3.625 2.972 2.766 3.335 12.698 1.289704002 3.869112006 4 0.088 0.1 0.12 0.136 0.444 0.045095966 0.135287898 5 9.392 5.798 9.835 8.01 33.035 3.355282068 10.0658462 6 0.153 0.075 0.01 0.517 0.755 0.076683456 0.230050367 7 7.4 7.565 7.703 6.415 29.083 2.953887343 8.86166203 8 0.221 0.606 1.321 0.539 2.687 0.272911849 0.818735546 9 0 0 0 0 0 0 0 10 1.381 0.368 1.196 1.091 4.036 0.409926394 1.229779182 11 0.066 0.025 0.039 0.017 0.147 0.014930421 0.044791264 12 0.166 0.041 0.062 0.024 0.293 0.029759275 0.089277825 13 0 0 0 0 0 0 0 14 0.71 0 0.005 0.074 0.789 0.08013675 0.240410251 15 0.239 0.17 0.208 0.286 0.903 0.091715444 0.275146333 16 0 0 0 0 0 0 0 14:03 System wide checks ok - WR same difference as before Statistics - attachment 7 Temperatures - attachment 8 Bias and leakage currents - attachment 9
Attachment 1: 210519_1002_Stats.png

Attachment 2: 210519_1002_Temp.png

Attachment 3: 210519_1003_Bias.png

Attachment 4: 210519_1213_Stats.png

Attachment 5: 210519_1214_Temp.png

Attachment 6: 210519_1215_Bioas.png

Attachment 7: 210519_1403_Stats.png

Attachment 8: 210519_1404_Temp.png

Attachment 9: 210519_1405_bias.png

	326	Tue May 18 23:11:40 2021	CA	Wednesday May 19th 00:00 - 08:00
00:00 CA takes over LS has performed usual checks, see previous Elog entry - all ok 00:30 stats ok R14_470 deadtime at 4.4% 02:10 system wide checks ok except: all ADC fail calibration (expected, no waveforms) WR decoder status Base Current Difference aida05 fault 0x500 : 0x52f : 47 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 02:12 FEE64 Temps ok - attachment 1 Stats ok - attachment 2 detector bias / leakage currents ok - attachment 3 dead-times ok - R14_497 max 4% - attachment 4 03:27 DAQ continues ok - writing to file R14_517 04:07 system wide checks ok except: all ADC fail calibration (expected, no waveforms) WR decoder status Base Current Difference aida05 fault 0x500 : 0x537 : 55 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FEE64 Temps ok - attachment 6 Stats ok - attachment 5 detector bias / leakage currents ok - attachment 7 dead-times ok - R14_526 max 4% - attachment 8 05:27 bad timestamp error in merger terminal - attachment 9 06:05 all system wide checks ok except; all ADC fail calibration (expected, no waveforms) WR decoder status Base Current Difference aida05 fault 0x500 : 0x53d : 61 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 06:07 FEE64 Temps ok - attachment 10 Stats ok - attachment 11 detector bias / leakage currents ok - attachment 12 dead-times ok - R14_558 max 3.6% - attachment 13 07:01 another bad timestamp error in merger terminal - attachment 14
Attachment 1: Screenshot_from_2021-05-19_01-13-48.png

Attachment 2: Screenshot_from_2021-05-19_01-14-09.png

Attachment 3: Screenshot_from_2021-05-19_01-14-24.png

Attachment 4: Screenshot_from_2021-05-19_01-19-37.png

Attachment 5: Screenshot_from_2021-05-19_03-04-39.png

Attachment 6: Screenshot_from_2021-05-19_03-05-33.png

Attachment 7: Screenshot_from_2021-05-19_03-05-53.png

Attachment 8: Screenshot_from_2021-05-19_03-09-03.png

Attachment 9: Screenshot_from_2021-05-19_04-26-09.png

Attachment 10: Screenshot_from_2021-05-19_05-08-01.png

Attachment 11: Screenshot_from_2021-05-19_05-08-23.png

Attachment 12: Screenshot_from_2021-05-19_05-08-46.png

Attachment 13: Screenshot_from_2021-05-19_05-11-33.png

Attachment 14: Screenshot_from_2021-05-19_06-12-29.png

	325	Tue May 18 18:47:25 2021	LS	Tuesday 18th May 20:00 - 24:00
19.50 Stats okay (Fig 1) FEE temps okay (Fig 2) Bias okay (Fig 3) System wide checks okay except: Base Current Difference aida05 fault 0x500 : 0x520 : 32 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~1.8M items/s Tape server ~7MB/s Analysed R14_399, deadtimes are okay (Fig 4) 20.30 Stats okay (Fig 5) Analysed R14_407, deadtimes are okay (Fig 6) 21.00 Stats okay (Fig 7) Analysed R14_415, deadtimes are okay (Fig 8) 21.30 Stats okay (Fig 9) Analysed R14_422, deadtimes are okay (Fig 10) 22.00 Stats okay (Fig 11) FEE temps okay (Fig 12) Bias okay (Fig 13) System wide checks okay except: Base Current Difference aida05 fault 0x500 : 0x524 : 36 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Merger rate ~2.3M items/s Tape server ~8MB/s Analysed R14_430, deadtimes are okay (Fig 14) 22.30 Stats okay (Fig 15) Analysed R14_438, deadtimes are okay (Fig 16) FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5% 23.00 Stats okay (Fig 17) Analysed R14_445, deadtimes are okay (Fig 18) 23.30 Stats okay (Fig 19) Analysed R14_454, deadtimes are okay (Fig 20) 24.00 Stats okay (Fig 21) FEE temps okay (Fig 22) Bias okay (Fig 23) System wide checks okay except: Base Current Difference aida05 fault 0x500 : 0x529 : 41 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR The difference has increased over time (not sure if an issue) Merger rate ~3M items/s Tape server ~8MB/s Analysed R14_462, deadtimes are okay (Fig 24)
Attachment 1: 210518_Stats_1950.png

Attachment 2: 210518_FEEtemps_1950.png

Attachment 3: 210518_Bias_1950.png

Attachment 4: 210518_Deadtimes_R14_399.png

Attachment 5: 210518_Stats_2028.png

Attachment 6: 210518_Deadtimes_R14_407.png

Attachment 7: 210518_Stats_2059.png

Attachment 8: 210518_Deadtimes_R14_415.png

Attachment 9: 210518_Stats_2130.png

Attachment 10: 210518_Deadtimes_R14_422.png

Attachment 11: 210518_Stats_2158.png

Attachment 12: 210518_FEEtemps_2158.png

Attachment 13: 210518_bias_2158.png

Attachment 14: 210518_Deadtimes_R14_430.png

Attachment 15: 210518_Stats_2230.png

Attachment 16: 210518_deadtimes_R14_438.png

Attachment 17: 210518_Stats_2300.png

Attachment 18: 210518_Deadtimes_R14_445.png

Attachment 19: 210518_Stats_2330.png

Attachment 20: 210518_Deadtimes_R14_454.png

Attachment 21: 210518_Stats_2400.png

Attachment 22: 210518_FEEtemps_2400.png

Attachment 23: 210518_bias_2400.png

Attachment 24: 210518_Deadtimes_R14_462.png

	324	Tue May 18 14:57:43 2021	CB	Tuesday 18th May 16:00 - 20:00
15:58 Took shift over from OH and PP Stats OK - attach 1 Temps OK - attach 2 Bias OK - attach 3 System-wide checks Clock - all pass ADC - all fail (as before) Base Current Difference aida05 fault 0x500 : 0x510 : 16 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA - all pass Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 8 7 4 1 4 2 2 2 2 7 : 37320 aida02 : 12 9 4 2 4 2 3 3 3 3 6 : 36856 aida03 : 19 6 7 2 2 2 2 4 3 3 6 : 37036 aida04 : 18 9 10 6 1 4 1 4 3 3 6 : 37168 aida05 : 17 12 2 1 2 3 3 3 3 3 6 : 36836 aida06 : 19 8 7 7 2 4 2 4 3 3 6 : 37468 aida07 : 28 5 4 1 2 4 2 3 2 2 7 : 37752 aida08 : 17 5 4 2 1 2 2 3 3 3 6 : 36396 aida09 : 21 10 4 5 4 4 3 3 3 3 6 : 37252 aida10 : 15 13 4 2 2 2 3 2 2 4 6 : 37284 aida11 : 22 13 3 3 4 3 1 4 3 3 6 : 37072 aida12 : 2 8 1 3 1 3 2 3 3 3 6 : 36472 aida13 : 23 5 4 4 1 4 3 2 3 3 6 : 36484 aida14 : 19 7 4 3 1 3 2 4 2 4 6 : 38116 aida15 : 20 10 2 2 3 2 3 2 3 3 6 : 36288 aida16 : 29 6 1 2 0 3 1 4 3 3 6 : 36724 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Mon May 17 11:43:04 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 Analysed R14_342. Attach 4. Dead time <5% 17:00 Stats OK - attach 5 Temps OK - as before Bias increasing quite quickly - attach 6 System-wide checks System-wide checks Clock - all pass ADC - all fail (as before) Base Current Difference aida05 fault 0x500 : 0x512 : 18 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA - all pass Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 28 7 8 4 1 4 2 2 2 2 7 : 37352 aida02 : 24 11 6 4 3 2 3 2 3 3 6 : 36440 aida03 : 17 11 5 2 1 2 2 3 3 3 6 : 36460 aida04 : 25 13 10 6 2 4 2 3 3 3 6 : 37036 aida05 : 23 7 3 2 2 3 3 3 3 3 6 : 36868 aida06 : 1 12 3 8 2 4 3 3 3 3 6 : 37140 aida07 : 22 12 1 1 2 4 2 2 2 2 7 : 37224 aida08 : 21 8 7 2 1 3 3 2 3 3 6 : 36356 aida09 : 23 7 4 6 4 3 2 4 3 3 6 : 37396 aida10 : 27 7 2 2 2 2 3 3 2 4 6 : 37764 aida11 : 12 5 3 5 4 3 1 4 3 3 6 : 37032 aida12 : 23 9 4 1 2 3 2 3 3 3 6 : 36612 aida13 : 10 12 6 3 1 4 3 2 3 3 6 : 36488 aida14 : 23 7 3 3 1 3 2 3 2 4 6 : 37604 aida15 : 22 6 6 4 2 2 3 2 3 3 6 : 36328 aida16 : 27 7 3 1 0 3 1 4 3 3 6 : 36724 18:00 Stats OK - attach 7 Temps OK - as before Bias - attach 8 System-wide checks - as before except Base Current Difference aida05 fault 0x500 : 0x517 : 23 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Analysed R14_371. Attach 9. Dead time below 5%. 18:35 MBS DAQ crashed 18:40 Stats OK - attach 9 Temps OK - as before Bias - attach 10 System-wide checks - as before except Base Current Difference aida05 fault 0x500 : 0x51a : 26 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:42 MBS DAQ restarted - new file being opened. 19:36 Stats OK - attach 11 Temps OK - attach 12 Bias - attach 13 System-wide checks Clock OK ADC all fail Base Current Difference aida05 fault 0x500 : 0x51e : 30 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA OK Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 24 11 5 4 1 4 2 3 2 2 7 : 37832 aida02 : 0 5 6 3 4 2 3 2 3 3 6 : 36328 aida03 : 15 8 5 3 2 2 1 3 4 3 6 : 37292 aida04 : 14 8 7 5 3 4 2 3 3 3 6 : 36936 aida05 : 31 6 1 1 2 3 3 3 2 4 6 : 37852 aida06 : 9 8 3 8 2 4 3 3 3 3 6 : 37140 aida07 : 21 7 2 1 2 4 1 3 2 2 7 : 37452 aida08 : 8 4 5 3 1 3 2 3 3 3 6 : 36528 aida09 : 24 7 3 5 4 3 2 4 3 3 6 : 37352 aida10 : 19 9 7 1 2 2 3 2 2 4 6 : 37284 aida11 : 27 5 0 5 4 3 1 4 3 3 6 : 37044 aida12 : 31 9 2 1 2 3 2 3 3 3 6 : 36612 aida13 : 21 10 4 3 1 4 3 2 3 3 6 : 36484 aida14 : 22 6 4 2 1 3 2 4 2 4 6 : 38088 aida15 : 21 9 3 4 2 2 3 2 3 3 6 : 36300 aida16 : 23 11 4 2 1 4 2 3 3 3 6 : 36724 19:43 Shift handed over to LS
Attachment 1: Stats1.png

Attachment 2: Temps1.png

Attachment 3: Bias1.png

Attachment 4: Analysis_R14_342
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 252065796 ( 1015168.3 Hz) Other data format: 9854206 ( 39686.8 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 242 ( 1.0 Hz) RESUME: 241 ( 1.0 Hz) SYNC100: 32924 ( 132.6 Hz) WR48-63: 32924 ( 132.6 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 9787875 ( 39419.6 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 499592 ( 2012.1 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 248.300 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.028 16.674 1 4.476 0.000 2 0.095 0.000 3 4.784 0.000 4 0.046 31.798 5 8.905 0.000 6 0.093 3.062 7 8.121 0.000 8 0.937 0.000 9 0.000 59.989 10 0.965 0.000 11 0.025 0.000 12 0.042 0.000 13 0.000 63.078 14 0.061 0.000 15 0.217 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5702434 2900 0 0 3 3 703 703 0 1488 0 26833 1 25200522 6341 0 0 31 30 3140 3140 0 0 0 55809 2 10737118 2562866 0 0 3 3 1687 1687 0 2559486 0 44132 3 32448674 2519917 0 0 37 37 4558 4558 0 2510727 0 63444 4 3679860 1335111 0 0 5 5 577 577 0 1333947 0 11808 5 37244520 759044 0 0 62 62 4940 4940 0 749040 0 34314 6 9560033 1568630 0 0 3 3 1348 1348 0 1565928 0 23903 7 40864979 1077667 0 0 66 66 5138 5138 0 1067259 0 40193 8 17664728 4388 0 0 6 6 2188 2188 0 0 0 56741 9 2360528 644 0 0 0 0 322 322 0 0 0 4285 10 20213561 5020 0 0 12 12 2498 2498 0 0 0 34696 11 11474187 2786 0 0 1 1 1392 1392 0 0 0 10298 12 8797735 2188 0 0 1 1 1093 1093 0 0 0 49042 13 2441760 604 0 0 0 0 302 302 0 0 0 3463 14 7875814 1932 0 0 4 4 962 962 0 0 0 33004 15 15799343 4168 0 0 8 8 2076 2076 0 0 0 7627 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 34.352s ( 931.544 blocks/s, 58.222 Mb/s)
Attachment 5: Stats2.png

Attachment 6: Bias2.png

Attachment 7: Stats3.png

Attachment 8: Bias3.png

Attachment 9: Analysis_R14_371
* TDR format 3.3.0 analyser - TD - May 2021 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 252182561 ( 1038396.9 Hz) Other data format: 9737439 ( 40095.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 234 ( 1.0 Hz) RESUME: 234 ( 1.0 Hz) SYNC100: 32904 ( 135.5 Hz) WR48-63: 32904 ( 135.5 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 9671163 ( 39822.4 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 493534 ( 2032.2 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 242.858 s FEE elapsed dead time(s) elapsed idle time(s) 0 0.060 15.876 1 5.312 0.000 2 0.091 0.000 3 3.590 0.000 4 0.163 29.634 5 9.407 0.000 6 0.049 0.000 7 6.820 0.000 8 2.387 0.000 9 0.000 29.724 10 0.676 0.000 11 0.043 0.000 12 0.046 0.000 13 0.000 57.696 14 0.016 3.102 15 0.214 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 5708701 2832 0 0 4 4 686 686 0 1452 0 26548 1 24896226 6554 0 0 36 36 3241 3241 0 0 0 54251 2 10734245 2512200 0 0 3 3 1683 1683 0 2508828 0 43373 3 32202526 2481239 0 0 32 32 4391 4391 0 2472393 0 63125 4 3649395 1322612 0 0 11 11 611 611 0 1321368 0 11610 5 36874871 743171 0 0 64 64 4722 4722 0 733599 0 33565 6 9551266 1557362 0 0 1 1 1434 1434 0 1554492 0 23558 7 41439213 1089893 0 0 57 57 5374 5374 0 1079031 0 40478 8 17460031 4238 0 0 7 7 2112 2112 0 0 0 55845 9 2360192 582 0 0 0 0 291 291 0 0 0 4290 10 20593873 5182 0 0 6 6 2585 2585 0 0 0 34440 11 11697470 2942 0 0 2 2 1469 1469 0 0 0 10244 12 8815559 2192 0 0 1 1 1095 1095 0 0 0 48502 13 2454200 586 0 0 0 0 293 293 0 0 0 3481 14 7942567 1924 0 0 2 2 960 960 0 0 0 32639 15 15802226 3930 0 0 8 8 1957 1957 0 0 0 7585 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 33.879s ( 944.541 blocks/s, 59.034 Mb/s)
Attachment 10: Stats4.png

Attachment 11: Bias4.png

	323	Tue May 18 06:50:53 2021	OH, PP	Tuesday 18th May 08:00 - 16:00
08:00 OH Takes over Statistics still look good from yesterday. Rate is still down. Dead time in FEEs 2 and 4 is 2.04% System wide checks clock ok Base Current Difference aida05 fault 0x4da : 0x500 : 38 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Baseline reset FPGA check ok Stats - attachment 1 Temp - attachment 2 Bias - attachment 3 08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips. I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4 The results of the test appear to indicate that it is the network link that we are saturating causing dead time *** Before After FEE DT (s) Dead time % DT (s) Dead time % 0 0.013 0.004184451 0.026 0.010025681 1 6.352 2.04458693 4.097 1.57981599 2 0.295 0.09495484 0.292 0.112596112 3 6.349 2.043621288 8.255 3.183153771 4 0.021 0.006759497 0.03 0.011568094 5 0.285 0.091736032 3.388 1.306423377 6 0.02 0.006437616 0.149 0.057454865 7 1.219 0.392372712 2.133 0.822491459 8 1.137 0.365978485 0.457 0.176220627 9 0 0 0 0 10 1.633 0.525631369 1.545 0.595756823 11 0.061 0.01963473 0.166 0.064010118 12 0.059 0.018990968 0.085 0.032776265 13 0 0 0 0 14 0.016 0.005150093 0.023 0.008868872 15 0.282 0.09077039 0.168 0.064781324 Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased 09:00 They are stopping the run to decrease the bias on the bPlast detector 09:25 No noticable change in AIDA They will now begin to change the spill structure. 09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change 10:21 Beam is back on AIDA 10:39 System wide checks. - ADC calibration check failed (this is anticipated because waveforms are off). - 1 Failed test in White Rabbit: Base Current Difference aida05 fault 0x500 : 0x506 : 6 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR - All other systems passed 11:00 Stats - attachment 4 Temp - attachment 5 Bias - attachment 6 12:46 System wide checks - ADC calibration check failed (this is anticipated because waveforms are off). - 1 Failed test in White Rabbit: Base Current Difference aida05 fault 0x500 : 0x50b : 11 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR - All other systems passed 14:36 System wide checks - ADC calibration check failed (this is anticipated because waveforms are off). - 1 Failed test in White Rabbit: Base Current Difference aida05 fault 0x500 : 0x50f : 15 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR - All other systems passed
Attachment 1: 210518_0757_Stats.png

Attachment 2: 210518_0758_Temp.png

Attachment 3: 210518_0759_Bias.png

Attachment 4: 210518_1059_Stats.png

Attachment 5: 210518_1100_Temp.png

Attachment 6: 210518_1101_Bias.png

	322	Mon May 17 23:17:16 2021	DSJ	Tuesday 18 May 0.00-8.00 shift
0.00 - Taking over from Lewis 00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2) 01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4) 01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6) 02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8) Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11) System Wide Checks: Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 0, Failed 16 Base Current Difference aida05 fault 0x4da : 0x4ee : 20 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Timestamp error counter test result: Passed 16, Failed 0 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 43 12 6 4 1 4 2 2 1 3 7 : 38444 aida02 : 21 12 4 4 2 3 3 3 3 3 6 : 36980 aida03 : 23 8 4 2 2 4 2 3 3 3 6 : 36764 aida04 : 34 12 10 5 1 4 1 4 3 3 6 : 37224 aida05 : 18 5 5 1 2 3 2 2 1 3 7 : 38112 aida06 : 40 12 5 7 2 4 2 4 3 3 6 : 37552 aida07 : 40 10 6 2 2 4 2 2 2 2 7 : 37392 aida08 : 19 11 3 3 1 4 3 3 3 3 6 : 36980 aida09 : 43 6 2 4 3 4 3 3 3 3 6 : 37180 aida10 : 32 14 10 3 4 1 2 4 1 3 7 : 39280 aida11 : 40 7 2 5 4 3 1 4 3 3 6 : 37144 aida12 : 37 14 13 2 2 3 2 4 3 3 6 : 37396 aida13 : 18 9 3 1 1 5 3 3 3 3 6 : 37024 aida14 : 36 13 6 4 2 3 2 3 1 3 7 : 38872 aida15 : 32 11 7 4 3 4 2 3 2 4 6 : 38024 aida16 : 26 9 4 1 0 3 1 4 3 3 6 : 36752 02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13) 03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15) 03.05 - No beam, problem with SIS, don't know how long will be off 03.14 - Beam is back 03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17) 04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19) Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22) System Wide Checks: Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 0, Failed 16 Base Current Difference aida05 fault 0x4da : 0x4f2 : 24 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Timestamp error counter test result: Passed 16, Failed 0 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 26 11 5 6 1 4 2 2 1 3 7 : 38416 aida02 : 16 9 5 4 2 4 2 2 3 3 6 : 36312 aida03 : 9 7 4 4 2 4 2 3 3 3 6 : 36764 aida04 : 36 11 8 4 3 4 1 3 4 3 6 : 37800 aida05 : 20 4 3 2 2 3 2 2 1 3 7 : 38112 aida06 : 22 13 10 7 2 4 3 3 4 3 6 : 38336 aida07 : 28 7 7 2 2 4 2 2 2 2 7 : 37336 aida08 : 2 3 4 4 1 3 2 3 3 3 6 : 36512 aida09 : 40 7 1 4 4 3 3 3 3 3 6 : 37096 aida10 : 26 9 12 2 3 2 2 4 1 3 7 : 39280 aida11 : 30 7 1 5 4 3 1 3 2 4 6 : 37600 aida12 : 15 10 6 4 2 3 2 4 3 3 6 : 37228 aida13 : 1 3 2 1 2 4 3 2 3 3 6 : 36316 aida14 : 34 12 7 4 2 3 2 3 1 3 7 : 38872 aida15 : 18 11 5 3 4 4 2 3 2 4 6 : 37968 aida16 : 22 7 4 2 0 3 1 4 3 3 6 : 36752 04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24) 05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26) 05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28) 06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30) Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33) System Wide Checks: Clock status test result: Passed 16, Failed 0 Calibration test result: Passed 0, Failed 16 Base Current Difference aida05 fault 0x4da : 0x4f9 : 31 White Rabbit error counter test result: Passed 15, Failed 1 FPGA Timestamp error counter test result: Passed 16, Failed 0 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 16 9 3 7 1 5 1 3 1 3 7 : 38744 aida02 : 18 3 5 2 3 3 3 2 3 3 6 : 36400 aida03 : 21 9 4 2 2 4 2 4 3 3 6 : 37276 aida04 : 32 12 7 3 3 4 1 3 4 3 6 : 37744 aida05 : 12 8 3 2 2 4 2 2 1 3 7 : 38240 aida06 : 28 14 4 8 2 4 3 3 3 3 6 : 37280 aida07 : 40 12 5 2 2 4 1 3 2 2 7 : 37648 aida08 : 13 8 4 4 1 3 2 3 3 3 6 : 36596 aida09 : 35 5 2 4 3 4 2 4 3 3 6 : 37396 aida10 : 6 10 4 1 1 3 2 2 1 3 7 : 38024 aida11 : 41 9 1 4 4 3 2 4 3 3 6 : 37372 aida12 : 29 11 9 4 2 3 2 4 3 3 6 : 37340 aida13 : 24 10 5 1 2 4 3 2 3 3 6 : 36512 aida14 : 34 16 5 4 2 3 2 3 1 3 7 : 38872 aida15 : 42 8 6 2 4 4 2 3 2 4 6 : 38024 aida16 : 8 9 6 1 0 3 1 4 3 3 6 : 36712 06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35) 07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37) 07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39)
Attachment 1: Screenshot_2021-05-18_Statistics_aidas-gsi(1).png

Attachment 2: R14_147_deadtime_23.41.06.png

Attachment 3: Screenshot_2021-05-18_Statistics_aidas-gsi(2).png

Attachment 4: R14_153_deadtime_00.07.12.png

Attachment 5: Screenshot_2021-05-18_Statistics_aidas-gsi(3).png

Attachment 6: R14_158_deadtime_00.32.42.png

Attachment 7: Screenshot_2021-05-18_Statistics_aidas-gsi(4).png

Attachment 8: R14_164_deadtime_01.02.32.png

Attachment 9: TMP.png

Attachment 10: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi.png

Attachment 11: Screenshot_2021-05-18_ucesb.png

Attachment 12: Screenshot_2021-05-18_Statistics_aidas-gsi(5).png

Attachment 13: R14_170_deadtime_01.35.24.png

Attachment 14: Screenshot_2021-05-18_Statistics_aidas-gsi(6).png

Attachment 15: R14_175_deadtime_01.59.37.png

Attachment 16: Screenshot_2021-05-18_Statistics_aidas-gsi(7).png

Attachment 17: R14_180_deadtime_02.31.32.png

Attachment 18: Screenshot_2021-05-18_Statistics_aidas-gsi(8).png

Attachment 19: R14_186_deadtime_03.00.27.png

Attachment 20: TMP.png

Attachment 21: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 22: Screenshot_2021-05-18_ucesb(1).png

Attachment 23: Screenshot_2021-05-18_Statistics_aidas-gsi(9).png

Attachment 24: R14_192_deadtime_03.31.26.png

Attachment 25: Screenshot_2021-05-18_Statistics_aidas-gsi(10).png

Attachment 26: R14_198_deadtime_04.00.21.png

Attachment 27: Screenshot_2021-05-18_Statistics_aidas-gsi(11).png

Attachment 28: R14_198_deadtime_04.27.46.png

Attachment 29: Screenshot_2021-05-18_Statistics_aidas-gsi(12).png

Attachment 30: R14_203_deadtime_05.png

Attachment 31: TMP.png

Attachment 32: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 33: Screenshot_2021-05-18_ucesb(3).png

Attachment 34: Screenshot_2021-05-18_Statistics_aidas-gsi(13).png

Attachment 35: R14_215_deadtime_05.30.26.png

Attachment 36: Screenshot_2021-05-18_Statistics_aidas-gsi(14).png

Attachment 37: R14__221_DEADTIME_06.00.35.png

Attachment 38: Screenshot_2021-05-18_Statistics_aidas-gsi(15).png

Attachment 39: R14_226_deadtime_06.25.32.png

	321	Mon May 17 19:05:23 2021	LS	Monday 17th May 20.00-24.00
20.30 Checked stats, seem okay (Fig 1) Ran R14_99 through analyser, dead times all good (Fig 2) Merger rate 1.8M items/s Tape server 6.4MB/s 21.00 Checked stats seem okay (Fig 3) Ran R14_106 through analyser, dead times all good (Fig 4) 21.30 Checked stats seem okay (Fig 5) Ran R14_112 through analyser, dead times all good (Fig 6) 22.00 Stats okay (Fig 7) FEE temps okay (Fig 8) Rate spectra (Fig 9) Leakage currents (Fig 10) System wide checks: Clock status all passed White rabbit 1 fail: Base Current Difference aida05 fault 0x4da : 0x4e7 : 13 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp all passed Merger rate 2.4M items/s Tape server 6.3MB/s Ran R14_119 through analyser, dead times all good (Fig 11) 22.30 Checked stats seem okay (Fig 12) Ran R14_123 through analyser, dead times all good (Fig 13) 23.00 Checked stats seem okay (Fig 14) Ran R14_129 through analyser, dead times all good (Fig 15) 23.30 Checked stats seem okay (Fig 16) Ran R14_135 through analyser, dead times all good (Fig 17) 24.00 Stats okay (Fig 18) FEE temps okay (Fig 19) Leakage currents (Fig 20) System wide checks: Clock status all passed White rabbit 1 fail: Base Current Difference aida05 fault 0x4da : 0x4eb : 17 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp all passed Merger rate 2.5M items/s Tape server 6.5MB/s Ran R14_142 through analyser, dead times all good (Fig 21)
Attachment 1: 210517_Stats_2026.png

Attachment 2: 210517_Deadtime_R14_99.png

Attachment 3: 210517_Stats_2101.png

Attachment 4: 210517_Deadtimes_R14_106.png

Attachment 5: 210517_Stats_2128.png

Attachment 6: 210517_Deadtimes_R14_112.png

Attachment 7: 210517_Stats_2159.png

Attachment 8: 210517_FEEtemps_2159.png

Attachment 9: 210517_Spectra_2159.png

Attachment 10: 210517_Bias_2159.png

Attachment 11: 210517_Deadtimes_R14_119.png

Attachment 12: 210517_Stats_2230.png

Attachment 13: 210517_Deadtimes_R14_123.png

Attachment 14: 210517_Stats_2300.png

Attachment 15: 210517_Deadtimes_R14_129.png

Attachment 16: 210517_Stats_2328.png

Attachment 17: 210517_Deadtimes_R14_135.png

Attachment 18: 210517_Stats_2359.png

Attachment 19: 210517_FEEtemps_2359.png

Attachment 20: 210517_bias_2359.png

Attachment 21: 210517_deadtime_R14_142_1.png

	320	Mon May 17 15:14:35 2021	LJW	Monday 17th May 2021 16:00-20:00
16:25 Informed at shift handover that there has been no beam. Statistics - See attachment 1 Temperatures - See attachment 2 Bias & Leakage currents - See attachment 3 System Wide checks - All passed, EXCEPT: ADC Calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit decoder status: Base Current Difference aida05 fault 0x4da : 0x4dd : 3 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 18:00 Usecb scalars now showing DSSD implant rates. Statistics - See attachment 4 Temperatures - See attachment 5 Bias & Leakage currents - See attachment 6 System Wide checks - All passed, EXCEPT: ADC Calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit decoder status Base Current Difference aida05 fault 0x4da : 0x4e1 : 7 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx 19:45 Usecb scalars have continued to show DSSD implant rates since the last recorded elog check. Statistics - See attachment 7 Temperatures - See attachment 8 Bias & Leakage currents - See attachment 9 System Wide checks - All passed, EXCEPT: ADC Calibration: FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit decoder status: Base Current Difference aida05 fault 0x4da : 0x4e1 : 7 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR 19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes 19:55 - Beam now back again - DSSD implants showing
Attachment 1: 170521_1.png

Attachment 2: 170521_2.png

Attachment 3: 170521_3.png

Attachment 4: 170521_4.png

Attachment 5: 170521_5.png

Attachment 6: 170521_6.png

Attachment 7: 170521_7.png

Attachment 8: 170521_8.png

Attachment 9: 170521_9.png

	319	Fri May 14 19:01:09 2021	JS	Friday 14th May 20:00-00:00
20:00 Taking over from CB 20:33 Doing full checks. Temps ok elog:319/8 Bias ok elog:319/1 Clock check - ok ADC Calc check - ok White Rabbit - Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc5 : 13 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 FPGA Timestamp errors - Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 Memory Info- Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 34 30 8 1 4 1 2 2 3 6 : 35772 aida02 : 37 30 28 5 3 1 2 4 2 3 6 : 36644 aida03 : 41 29 30 16 4 3 1 5 1 3 6 : 36588 aida04 : 13 6 23 12 4 4 4 4 2 2 6 : 35412 aida05 : 20 32 23 10 2 2 2 2 1 4 6 : 36736 aida06 : 14 5 10 5 1 3 2 3 2 3 6 : 35680 aida07 : 44 23 22 9 4 3 1 3 2 3 6 : 36200 aida08 : 39 30 28 9 0 3 2 3 3 2 6 : 35308 aida09 : 41 31 28 10 2 5 1 3 1 3 6 : 35484 aida10 : 43 23 27 7 3 1 1 3 3 3 6 : 36916 aida11 : 39 35 17 10 3 3 2 2 2 3 6 : 35908 aida12 : 37 36 15 10 2 2 2 4 3 2 6 : 35684 aida13 : 47 32 17 7 4 2 3 4 3 2 6 : 36012 aida14 : 23 33 14 10 3 4 3 4 3 2 6 : 36164 aida15 : 44 28 24 12 3 2 4 3 3 2 6 : 35920 aida16 : 22 17 24 17 3 4 2 3 1 3 6 : 35648 Stats Aida Correlation Info #8 - elog:319/2 same as elog:317/27 Resume info #3 - elog:319/3 same as elog:317/26 Pause info #2 - elog:319/4 same as elog:317/32 AIDA disk info #6 - elog:319/5 same as elog:317/31 AIDA ADC data items - elog:319/6 same as elog:317/30 Good Events - elog:319/7 same as elog:317/29 [I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see clearly once uploaded the screenshots] Analysis R5_108 elog:319/9 Pause 149 Resume 150 Highest deadtime FEE10 3% 21:01 ucesb ok - Max 1700 Hz Implant - 1MHz Decay 21:35 Temps ok Bias ok Clock check - ok ADC Calc check - ok White Rabbit - Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc5 : 13 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 39 34 29 8 1 4 1 2 2 3 6 : 35772 aida02 : 46 29 28 5 3 2 2 3 2 3 6 : 36288 aida03 : 39 25 29 16 4 2 2 5 1 3 6 : 36660 aida04 : 14 6 19 12 4 4 3 4 2 2 6 : 35096 aida05 : 20 30 24 10 2 2 2 2 1 4 6 : 36736 aida06 : 26 4 11 5 2 4 3 2 2 3 6 : 35672 aida07 : 46 22 22 9 4 3 1 3 2 3 6 : 36200 aida08 : 28 32 28 9 0 2 2 4 3 2 6 : 35664 aida09 : 32 30 29 10 3 5 2 3 1 3 6 : 35776 aida10 : 39 23 25 7 3 1 1 3 3 3 6 : 36868 aida11 : 40 31 21 10 3 3 1 4 1 3 6 : 35688 aida12 : 17 31 12 10 2 2 3 4 3 2 6 : 35772 aida13 : 35 28 22 7 4 2 3 4 3 2 6 : 36012 aida14 : 30 33 14 10 3 4 3 4 3 2 6 : 36192 aida15 : 34 26 24 12 3 2 4 3 3 2 6 : 35864 aida16 : 38 24 27 17 3 4 2 3 1 3 6 : 35816 Stats Good Events - elog:319/10 AIDA ADC data items - elog:319/11 AIDA disk info #6 - elog:319/12 Pause info #2 - elog:319/13 Resume info #3 - elog:319/14 Aida Correlation Info #8 - elog:319/15 - rates showed zero, checked again and normal elog:319/17 Analysis R5_128 elog:319/16 Pause 167 Resume 166 Highest deadtime FEE10 5% 22:00 Temps ok Stats ok ucesb ok 22:35 Temps ok Stats ok ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up again elog:319/17 22:54 They are closing the file, we are on R5_155 23:00
Attachment 1: bias1.png

Attachment 2: Screenshot_2021-05-14_Statistics_aidas-gsi(10).png

Attachment 3: Screenshot_2021-05-14_Statistics_aidas-gsi(9).png

Attachment 4: Screenshot_2021-05-14_Statistics_aidas-gsi(8).png

Attachment 5: Screenshot_2021-05-14_Statistics_aidas-gsi(7).png

Attachment 6: Screenshot_2021-05-14_Statistics_aidas-gsi(6).png

Attachment 7: Screenshot_2021-05-14_Statistics_aidas-gsi(5).png

Attachment 8: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 9: R5_108_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 256861958 ( 1503007.9 Hz) Other data format: 5058042 ( 29596.7 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 149 ( 0.9 Hz) RESUME: 150 ( 0.9 Hz) SYNC100: 32636 ( 191.0 Hz) WR48-63: 32636 ( 191.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 4992471 ( 29213.1 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 207149 ( 1212.1 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 170.899 s FEE elapsed dead time(s) elapsed idle time(s) 1 4.459 72.353 2 1.217 86.117 3 3.357 28.177 4 0.000 160.970 5 0.495 76.386 6 0.216 93.617 7 1.418 40.259 8 3.253 44.571 9 0.000 168.837 10 5.250 9.058 11 1.956 49.014 12 0.182 82.483 13 0.000 160.585 14 0.143 88.740 15 3.053 35.456 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8464368 3303 0 0 2 2 1138 1138 0 1023 0 11424 1 16399642 4233 0 0 17 18 2099 2099 0 0 0 22505 2 13045073 1576129 0 0 6 6 1793 1793 0 1572531 0 18179 3 26563169 1558422 0 0 14 14 3487 3487 0 1551420 0 26353 4 2213090 510296 0 0 0 0 328 328 0 509640 0 5100 5 15498485 314714 0 0 4 4 1986 1986 0 310734 0 15610 6 12553775 609520 0 0 7 7 1573 1573 0 606360 0 10360 7 23384876 446841 0 0 10 10 3029 3029 0 440763 0 18175 8 21832531 5542 0 0 15 15 2756 2756 0 0 0 22662 9 1045474 278 0 0 0 0 139 139 0 0 0 1866 10 42357744 10552 0 0 43 43 5233 5233 0 0 0 12639 11 20606319 4998 0 0 11 11 2488 2488 0 0 0 4351 12 14311836 3576 0 0 6 6 1782 1782 0 0 0 20348 13 3016503 780 0 0 0 0 390 390 0 0 0 1508 14 13047482 3248 0 0 4 4 1620 1620 0 0 0 12826 15 22521591 5610 0 0 10 10 2795 2795 0 0 0 3243 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:74697.445s ( 0.428 blocks/s, 0.027 Mb/s)
Attachment 10: Screenshot_2021-05-14_Statistics_aidas-gsi(16).png

Attachment 11: Screenshot_2021-05-14_Statistics_aidas-gsi(15).png

Attachment 12: Screenshot_2021-05-14_Statistics_aidas-gsi(14).png

Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi(13).png

Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(12).png

Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(11).png

Attachment 16: R5_128_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 256841478 ( 1482693.9 Hz) Other data format: 5078522 ( 29317.3 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 167 ( 1.0 Hz) RESUME: 166 ( 1.0 Hz) SYNC100: 32644 ( 188.4 Hz) WR48-63: 32644 ( 188.4 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 5012901 ( 28938.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 205165 ( 1184.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 173.226 s FEE elapsed dead time(s) elapsed idle time(s) 1 2.621 76.080 2 1.556 81.876 3 3.719 27.628 4 0.000 162.954 5 0.710 80.266 6 0.170 93.769 7 2.714 44.740 8 3.483 45.565 9 0.000 167.005 10 8.522 10.948 11 2.181 50.562 12 0.202 85.035 13 0.000 161.510 14 0.146 88.959 15 1.812 38.597 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8468806 3273 0 0 7 7 1112 1112 0 1035 0 11260 1 16364560 4360 0 0 13 13 2167 2167 0 0 0 23505 2 13265609 1594590 0 0 6 6 1881 1881 0 1590816 0 18254 3 26054686 1576337 0 0 14 14 3422 3422 0 1569465 0 26152 4 2223261 507850 0 0 0 0 362 362 0 507126 0 4952 5 15152054 312213 0 0 8 7 1938 1938 0 308322 0 15336 6 12659379 606961 0 0 4 4 1612 1612 0 603729 0 10099 7 22669666 438118 0 0 14 14 2841 2841 0 432408 0 17594 8 22337729 5564 0 0 15 15 2767 2767 0 0 0 22524 9 1020352 240 0 0 0 0 120 120 0 0 0 1807 10 42596491 10676 0 0 59 59 5279 5279 0 0 0 12076 11 20672887 5060 0 0 13 13 2517 2517 0 0 0 4379 12 14335521 3608 0 0 5 5 1799 1799 0 0 0 20105 13 2812573 690 0 0 0 0 345 345 0 0 0 1405 14 13325633 3260 0 0 4 4 1626 1626 0 0 0 12554 15 22882271 5722 0 0 5 5 2856 2856 0 0 0 3163 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:78055.344s ( 0.410 blocks/s, 0.026 Mb/s)
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(17).png

Attachment 18: ucesb.png

Attachment 19: _~.png

Attachment 20: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 21: Screenshot_2021-05-14_Statistics_aidas-gsi(23).png

Attachment 22: Screenshot_2021-05-14_Statistics_aidas-gsi(22).png

Attachment 23: Screenshot_2021-05-14_Statistics_aidas-gsi(21).png

Attachment 24: Screenshot_2021-05-14_Statistics_aidas-gsi(20).png

Attachment 25: Screenshot_2021-05-14_Statistics_aidas-gsi(19).png

Attachment 26: Screenshot_2021-05-14_Statistics_aidas-gsi(18).png

Attachment 27: R5_157_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 256650793 ( 1485751.4 Hz) Other data format: 5269207 ( 30503.4 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 130 ( 0.8 Hz) RESUME: 129 ( 0.7 Hz) SYNC100: 32643 ( 189.0 Hz) WR48-63: 32643 ( 189.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 5203662 ( 30124.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 223116 ( 1291.6 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 172.741 s FEE elapsed dead time(s) elapsed idle time(s) 1 2.094 74.418 2 0.927 76.576 3 3.655 26.911 4 0.000 161.752 5 0.023 80.653 6 0.215 91.583 7 1.871 44.002 8 1.038 39.676 9 0.000 166.718 10 8.591 10.747 11 3.917 45.280 12 0.214 88.240 13 0.000 163.408 14 0.280 88.718 15 1.416 44.122 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8604841 3308 0 0 0 0 1138 1138 0 1032 0 12424 1 16359820 4228 0 0 13 13 2101 2101 0 0 0 24974 2 13547296 1610585 0 0 6 6 1930 1930 0 1606713 0 19730 3 25981004 1589167 0 0 14 13 3491 3491 0 1582158 0 28914 4 2316430 549243 0 0 0 0 351 351 0 548541 0 5271 5 15208801 341870 0 0 1 1 1974 1974 0 337920 0 16457 6 12807457 657341 0 0 5 5 1667 1667 0 653997 0 10925 7 22697274 479019 0 0 10 10 2849 2849 0 473301 0 19398 8 22587880 5594 0 0 5 5 2792 2792 0 0 0 24704 9 1078064 266 0 0 0 0 133 133 0 0 0 1963 10 42543713 10740 0 0 43 43 5327 5327 0 0 0 13313 11 20362862 5182 0 0 14 14 2577 2577 0 0 0 4606 12 13994455 3436 0 0 7 7 1711 1711 0 0 0 21818 13 2731986 666 0 0 0 0 333 333 0 0 0 1621 14 13145005 3222 0 0 8 8 1603 1603 0 0 0 13539 15 22683905 5340 0 0 4 4 2666 2666 0 0 0 3459 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:83137.008s ( 0.385 blocks/s, 0.024 Mb/s)
	318	Fri May 14 17:30:56 2021	TD	Analysis of S496 files R5_18, R5_32 & R5_45
S496/R5_18 - attachment 1 LEC fast discriminators - all channels disabled Slow comparator thresholds - p+n FEE64s 0xa except aida09 0xf n+n FEE64s 0x20 Sampling ADCs OFF S496/R5_32 - attachment 2 LEC fast discriminators - all channels disabled Slow comparator thresholds - p+n FEE64s 0xa except aida09 & aida15 0xf n+n FEE64s 0x20 Sampling ADCs ON, trigger threshold 15000 S496/R5_45 - attachment 3 LEC fast discriminators - all channels disabled Slow comparator thresholds - p+n FEE64s 0xc n+n FEE64s 0x20 Sampling ADCs ON, trigger threshold 15000 * N.B. Ignore 'elapsed idle time(s)' metric which does not work with WR timestamps
Attachment 1: R5_18
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 260127700 ( 2662102.7 Hz) Other data format: 1792300 ( 18342.1 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 1843 ( 18.9 Hz) RESUME: 1843 ( 18.9 Hz) SYNC100: 32365 ( 331.2 Hz) WR48-63: 32365 ( 331.2 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 1723884 ( 17641.9 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 65165 ( 666.9 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 97.715 s FEE elapsed dead time(s) elapsed idle time(s) 1 36.653 41.693 2 44.272 48.364 3 46.942 50.228 4 0.000 81.057 5 27.042 29.539 6 41.734 45.821 7 43.168 46.980 8 54.980 58.850 9 4.477 17.487 10 57.700 61.776 11 43.911 47.867 12 48.996 52.920 13 21.829 26.853 14 51.720 56.077 15 53.092 56.077 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 16755449 4921 0 0 122 122 2151 2151 0 375 0 4078 1 15729031 4184 0 0 103 103 1989 1989 0 0 0 7687 2 17390463 502955 0 0 136 135 2294 2294 0 498096 0 5652 3 14745088 476937 0 0 118 118 1907 1907 0 472887 0 8350 4 2857057 295831 0 0 0 0 413 413 0 295005 0 2971 5 19341027 128907 0 0 112 112 2426 2426 0 123831 0 5913 6 17649809 199902 0 0 136 136 2168 2168 0 195294 0 3189 7 17833639 143068 0 0 135 135 2201 2201 0 138396 0 5626 8 17514114 4531 0 0 139 138 2127 2127 0 0 0 6021 9 14922420 3686 0 0 23 23 1820 1820 0 0 0 1039 10 16909910 4387 0 0 140 141 2053 2053 0 0 0 3196 11 18661933 4896 0 0 151 151 2297 2297 0 0 0 1392 12 17768788 4718 0 0 148 148 2211 2211 0 0 0 5274 13 16872590 4258 0 0 96 96 2033 2033 0 0 0 605 14 16747047 4335 0 0 134 135 2033 2033 0 0 0 3273 15 18429335 4784 0 0 150 150 2242 2242 0 0 0 899 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:66231.297s ( 0.483 blocks/s, 0.030 Mb/s)
Attachment 2: R5_32
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 259165945 ( 2447522.6 Hz) Other data format: 2754055 ( 26008.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 1068 ( 10.1 Hz) RESUME: 1070 ( 10.1 Hz) SYNC100: 32395 ( 305.9 Hz) WR48-63: 32395 ( 305.9 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 2687127 ( 25376.8 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 109002 ( 1029.4 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 105.889 s FEE elapsed dead time(s) elapsed idle time(s) 1 2.772 22.665 2 28.336 32.781 3 9.177 18.264 4 0.000 94.253 5 5.882 26.886 6 8.897 16.463 7 18.512 26.355 8 29.893 33.327 9 0.057 79.535 10 45.010 46.522 11 27.988 27.766 12 32.050 37.161 13 1.981 23.343 14 0.002 96.477 15 36.327 39.608 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 17728100 5332 0 0 32 32 2340 2340 0 588 0 6827 1 15060064 3752 0 0 18 18 1858 1858 0 0 0 14360 2 18674317 723732 0 0 99 99 2382 2382 0 718770 0 8046 3 19118104 904863 0 0 39 39 2463 2463 0 899859 0 15890 4 2260998 321658 0 0 0 0 326 326 0 321006 0 3138 5 14506811 189277 0 0 25 25 1756 1756 0 185715 0 9208 6 21798062 340177 0 0 43 43 2689 2689 0 334713 0 5553 7 18561150 231264 0 0 70 70 2324 2324 0 226476 0 9337 8 22060606 5804 0 0 136 136 2766 2766 0 0 0 10232 9 5139884 1246 0 0 5 5 618 618 0 0 0 1276 10 24215101 6255 0 0 190 191 2937 2937 0 0 0 4375 11 22485598 5956 0 0 148 148 2830 2830 0 0 0 2034 12 19527530 4883 0 0 108 109 2333 2333 0 0 0 8243 13 14237341 3538 0 0 9 9 1760 1760 0 0 0 984 14 2260329 600 0 0 1 1 299 299 0 0 0 8235 15 21531950 5718 0 0 145 145 2714 2714 0 0 0 1264 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:65849.211s ( 0.486 blocks/s, 0.030 Mb/s)
Attachment 3: R5_45
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 257173968 ( 1605341.2 Hz) Other data format: 4746032 ( 29625.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 149 ( 0.9 Hz) RESUME: 149 ( 0.9 Hz) SYNC100: 32597 ( 203.5 Hz) WR48-63: 32597 ( 203.5 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 4680540 ( 29217.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 198582 ( 1239.6 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 160.199 s FEE elapsed dead time(s) elapsed idle time(s) 1 1.826 60.560 2 1.857 72.569 3 3.101 18.747 4 0.000 149.092 5 0.725 60.979 6 0.276 85.413 7 1.668 27.289 8 2.719 31.235 9 0.000 157.899 10 6.867 10.073 11 2.634 42.733 12 0.375 69.093 13 0.000 148.656 14 0.255 79.878 15 3.499 32.266 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8318308 3120 0 0 3 3 1077 1077 0 960 0 11152 1 17482671 4536 0 0 10 10 2258 2258 0 0 0 22118 2 12591028 1472109 0 0 7 7 1790 1790 0 1468515 0 17237 3 26467090 1465014 0 0 17 17 3574 3574 0 1457832 0 25917 4 2005748 477188 0 0 0 0 331 331 0 476526 0 4838 5 16244903 298514 0 0 5 5 2012 2012 0 294480 0 14514 6 11816116 567974 0 0 7 7 1554 1554 0 564852 0 9683 7 24437155 423701 0 0 8 8 3155 3155 0 417375 0 17498 8 22741388 5724 0 0 10 10 2852 2852 0 0 0 21729 9 1027419 274 0 0 0 0 137 137 0 0 0 1781 10 39497864 9636 0 0 47 47 4771 4771 0 0 0 12114 11 19835466 4892 0 0 8 8 2438 2438 0 0 0 4107 12 15299334 3708 0 0 8 8 1846 1846 0 0 0 19448 13 3576125 840 0 0 0 0 420 420 0 0 0 1437 14 13120071 3182 0 0 7 7 1584 1584 0 0 0 12115 15 22713282 5620 0 0 12 12 2798 2798 0 0 0 2894 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:63108.355s ( 0.507 blocks/s, 0.032 Mb/s)
	317	Fri May 14 14:57:32 2021	CB + OH, TD, NH	Friday 14th May 16:00-20:00
16:00 Shift taken over from CA 16:00 Temperatures OK. Attach 1. Upon reloading aida11 first returns "No respose" for all values. Upon refreshing, aida11 responds but aida 9 returns "No response". Upon refreshing again, attach 1. Bias OK. Attach 2. Statistics OK. Attach 3-7 System-wide checks Clock OK ADC - all fail Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 32 28 25 7 1 4 1 3 2 3 6 : 36112 aida02 : 43 28 29 5 4 2 3 4 1 3 6 : 36092 aida03 : 39 26 28 5 4 3 2 5 1 3 6 : 36428 aida04 : 9 5 21 8 5 4 4 4 2 2 6 : 35292 aida05 : 46 32 26 7 2 2 2 2 2 3 6 : 35768 aida06 : 4 5 16 1 3 4 2 4 1 3 6 : 35352 aida07 : 47 21 25 5 4 3 1 4 1 3 6 : 35604 aida08 : 62 33 32 6 0 2 3 3 3 2 6 : 35520 aida09 : 33 25 28 7 3 5 2 4 1 3 6 : 36140 aida10 : 47 29 22 6 3 2 2 2 2 3 6 : 35716 aida11 : 39 30 29 6 2 3 1 3 2 3 6 : 36124 aida12 : 38 36 20 6 3 2 3 3 3 2 6 : 35448 aida13 : 42 29 24 3 4 2 3 3 3 2 6 : 35440 aida14 : 31 28 18 7 3 3 4 4 2 2 6 : 35228 aida15 : 48 25 30 5 3 2 4 4 2 2 6 : 35272 aida16 : 37 16 28 6 3 3 2 2 2 3 6 : 35796 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 16:15 Control handed over to OH temporarily. 16:25 AIDA stopped. OH attempts to update Merger. Discriminators off. 16:29 AIDA restart begins. 16:33 AIDA restarted. 16:40 Dead time still very high ~50% in most FEEs. 16:50 Checking effect of threshold on dead time. 16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This reduces deadtime to <10%. Sampling ADCs were disabled (switched off) late Wednesday evening. 17:01 DAQ can be restarted. 17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back. 17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20 17:40 OH hands back control. Temperature OK - attach 10 Bias OK - attach 11 Stats - attach 12-17 System-wide checks Clock - all pass ADC - all pass WR Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 56 35 27 8 1 4 1 2 2 3 6 : 35816 aida02 : 43 31 27 5 3 1 3 3 2 3 6 : 36404 aida03 : 45 28 28 16 4 2 2 4 1 3 6 : 36180 aida04 : 2 2 18 12 4 3 3 4 2 2 6 : 34872 aida05 : 28 28 24 10 2 2 2 2 1 4 6 : 36752 aida06 : 28 11 9 5 2 3 3 3 1 3 6 : 35064 aida07 : 52 24 23 9 4 3 1 2 2 3 6 : 35744 aida08 : 21 26 27 8 0 2 3 3 3 2 6 : 35284 aida09 : 40 32 26 10 3 5 1 3 1 3 6 : 35520 aida10 : 46 23 27 7 3 1 1 3 3 3 6 : 36928 aida11 : 49 31 20 10 2 2 1 2 2 3 6 : 35516 aida12 : 39 35 17 9 2 2 3 4 3 2 6 : 35940 aida13 : 40 32 20 7 4 2 3 4 3 2 6 : 36032 aida14 : 33 33 16 10 3 4 3 4 3 2 6 : 36236 aida15 : 53 27 25 12 3 2 4 4 2 2 6 : 35452 aida16 : 35 22 32 15 3 4 3 3 1 3 6 : 36060 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 18:30 Temperatures - OK (attach 18) Bias OK (attach 19) Stats - attach 20-24 System-wide checks Clock - all pass ADC - all pass Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 41 31 31 8 1 4 1 2 2 3 6 : 35788 aida02 : 42 31 29 5 3 2 4 4 1 3 6 : 36304 aida03 : 29 30 29 16 4 3 2 4 1 3 6 : 36276 aida04 : 28 4 20 12 4 3 4 4 2 2 6 : 35280 aida05 : 25 28 23 10 2 2 2 2 1 4 6 : 36724 aida06 : 5 3 10 5 2 4 2 2 2 3 6 : 35308 aida07 : 38 26 22 9 4 3 1 2 2 3 6 : 35688 aida08 : 52 30 28 8 0 2 2 3 3 2 6 : 35200 aida09 : 22 26 21 10 3 4 1 4 1 3 6 : 35704 aida10 : 37 28 25 7 3 1 1 3 3 3 6 : 36900 aida11 : 53 30 20 10 2 3 2 2 2 3 6 : 35908 aida12 : 37 34 17 10 2 2 3 4 3 2 6 : 35956 aida13 : 47 34 16 7 4 2 3 4 3 2 6 : 36012 aida14 : 37 29 17 10 3 4 3 4 3 2 6 : 36236 aida15 : 44 28 25 12 3 2 4 4 2 2 6 : 35424 aida16 : 40 21 33 15 3 4 1 3 1 3 6 : 35576 18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5% 19:40 Temperatures OK Bias OK - attach 26 Stats - attach 27-32 System-wide checks Clock, ADC - all pass WR Base Current Difference aida01 fault 0xf932 : 0xf934 : 2 aida02 fault 0x62ec : 0x62ee : 2 aida03 fault 0x8679 : 0x867b : 2 aida04 fault 0xf0e4 : 0xf0e6 : 2 aida05 fault 0x9db8 : 0x9dc4 : 12 aida06 fault 0x7f18 : 0x7f1a : 2 aida07 fault 0xdd2c : 0xdd2e : 2 aida08 fault 0x1557 : 0x1559 : 2 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 41 33 30 8 1 4 1 2 2 3 6 : 35788 aida02 : 51 30 30 5 3 2 3 4 1 3 6 : 36092 aida03 : 52 31 28 16 4 2 2 5 1 3 6 : 36744 aida04 : 6 8 20 12 4 4 3 4 2 2 6 : 35096 aida05 : 30 31 22 10 2 2 2 2 1 4 6 : 36752 aida06 : 12 6 10 5 2 3 3 2 2 3 6 : 35488 aida07 : 53 24 21 9 4 3 1 3 2 3 6 : 36228 aida08 : 36 28 28 9 0 2 3 3 3 2 6 : 35408 aida09 : 27 28 25 10 3 4 2 3 1 3 6 : 35548 aida10 : 50 23 26 7 3 1 1 3 3 3 6 : 36928 aida11 : 42 28 22 10 3 3 2 2 2 3 6 : 35944 aida12 : 39 35 15 10 2 2 2 4 3 2 6 : 35684 aida13 : 43 34 17 7 4 2 3 4 3 2 6 : 36012 aida14 : 42 32 16 10 3 4 3 4 3 2 6 : 36264 aida15 : 36 30 25 12 3 2 4 3 3 2 6 : 35920 aida16 : 40 21 25 15 3 4 2 3 1 3 6 : 35704 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 19:40 Analysed R5_86. Attached. Dead time as before. 19:57 Shift handed over to James S
Attachment 1: Temp1.png

Attachment 2: Bias1.png

Attachment 3: Stats1-1.png

Attachment 4: Stats1-2.png

Attachment 5: Stats1-3.png

Attachment 6: Stats1-4.png

Attachment 7: Stats1-5.png

Attachment 8: Stats1-6.png

Attachment 9: R5_45_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 257173968 ( 1605341.2 Hz) Other data format: 4746032 ( 29625.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 149 ( 0.9 Hz) RESUME: 149 ( 0.9 Hz) SYNC100: 32597 ( 203.5 Hz) WR48-63: 32597 ( 203.5 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 4680540 ( 29217.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 198582 ( 1239.6 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 160.199 s FEE elapsed dead time(s) elapsed idle time(s) 1 1.826 60.560 2 1.857 72.569 3 3.101 18.747 4 0.000 149.092 5 0.725 60.979 6 0.276 85.413 7 1.668 27.289 8 2.719 31.235 9 0.000 157.899 10 6.867 10.073 11 2.634 42.733 12 0.375 69.093 13 0.000 148.656 14 0.255 79.878 15 3.499 32.266 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8318308 3120 0 0 3 3 1077 1077 0 960 0 11152 1 17482671 4536 0 0 10 10 2258 2258 0 0 0 22118 2 12591028 1472109 0 0 7 7 1790 1790 0 1468515 0 17237 3 26467090 1465014 0 0 17 17 3574 3574 0 1457832 0 25917 4 2005748 477188 0 0 0 0 331 331 0 476526 0 4838 5 16244903 298514 0 0 5 5 2012 2012 0 294480 0 14514 6 11816116 567974 0 0 7 7 1554 1554 0 564852 0 9683 7 24437155 423701 0 0 8 8 3155 3155 0 417375 0 17498 8 22741388 5724 0 0 10 10 2852 2852 0 0 0 21729 9 1027419 274 0 0 0 0 137 137 0 0 0 1781 10 39497864 9636 0 0 47 47 4771 4771 0 0 0 12114 11 19835466 4892 0 0 8 8 2438 2438 0 0 0 4107 12 15299334 3708 0 0 8 8 1846 1846 0 0 0 19448 13 3576125 840 0 0 0 0 420 420 0 0 0 1437 14 13120071 3182 0 0 7 7 1584 1584 0 0 0 12115 15 22713282 5620 0 0 12 12 2798 2798 0 0 0 2894 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:63108.355s ( 0.507 blocks/s, 0.032 Mb/s)
Attachment 10: Temps2.png

Attachment 11: Bias2.png

Attachment 12: Stats2-1.png

Attachment 13: Stats2-2.png

Attachment 14: Stats2-3.png

Attachment 15: Stats2-4.png

Attachment 16: Stats2-5.png

Attachment 17: Stats2-6.png

Attachment 18: Temp3.png

Attachment 19: Bias3.png

Attachment 20: Stats3-1.png

Attachment 21: Stats3-2.png

Attachment 22: Stats3-3.png

Attachment 23: Stats3-4.png

Attachment 24: Stats3-5.png

Attachment 25: R5_67_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 255938817 ( 1305497.2 Hz) Other data format: 5981183 ( 30508.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 114 ( 0.6 Hz) RESUME: 113 ( 0.6 Hz) SYNC100: 32730 ( 166.9 Hz) WR48-63: 32730 ( 166.9 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 5915496 ( 30173.9 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 257806 ( 1315.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 196.047 s FEE elapsed dead time(s) elapsed idle time(s) 1 2.111 102.399 2 0.513 105.076 3 3.587 39.451 4 0.000 182.105 5 0.792 103.780 6 0.167 115.991 7 1.244 72.680 8 0.805 56.828 9 0.000 190.127 10 2.629 6.967 11 0.295 76.751 12 0.222 114.996 13 0.000 188.225 14 0.031 124.690 15 2.491 63.947 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8223770 3364 0 0 2 2 1092 1092 0 1176 0 13825 1 16628095 4338 0 0 12 12 2157 2157 0 0 0 28636 2 13173045 1831289 0 0 3 3 1864 1864 0 1827555 0 22667 3 27251797 1807380 0 0 13 13 3668 3668 0 1800018 0 33190 4 2564061 622927 0 0 0 0 404 404 0 622119 0 6108 5 14810679 385418 0 0 6 6 1903 1903 0 381600 0 19053 6 12067625 742516 0 0 6 6 1616 1616 0 739272 0 12600 7 21266971 549224 0 0 11 11 2723 2723 0 543756 0 22659 8 23290358 5786 0 0 5 5 2888 2888 0 0 0 28199 9 1157906 324 0 0 0 0 162 162 0 0 0 2443 10 45608185 11489 0 0 27 26 5718 5718 0 0 0 16139 11 20146427 4876 0 0 6 6 2432 2432 0 0 0 5579 12 13411417 3300 0 0 7 7 1643 1643 0 0 0 24922 13 2681820 654 0 0 0 0 327 327 0 0 0 1925 14 11884997 2904 0 0 1 1 1451 1451 0 0 0 15892 15 21771664 5394 0 0 15 15 2682 2682 0 0 0 3969 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:67311.180s ( 0.475 blocks/s, 0.030 Mb/s)
Attachment 26: Stats_RESUME.png

Attachment 27: Stats-CORRELATION.png

Attachment 28: Bias4.png

Attachment 29: Stats-good.png

Attachment 30: Stats-AIDAADC.png

Attachment 31: Stats_DISC.png

Attachment 32: Stats_PAUSE.png

Attachment 33: R5_86_analysis.txt
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 256871525 ( 1549964.4 Hz) Other data format: 5048475 ( 30462.5 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 133 ( 0.8 Hz) RESUME: 132 ( 0.8 Hz) SYNC100: 32617 ( 196.8 Hz) WR48-63: 32617 ( 196.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 4982976 ( 30067.3 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 217412 ( 1311.9 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 165.727 s FEE elapsed dead time(s) elapsed idle time(s) 1 1.819 66.425 2 0.365 78.785 3 3.792 29.131 4 0.000 149.995 5 0.868 74.751 6 0.080 85.583 7 2.138 33.988 8 2.899 39.410 9 0.000 164.566 10 6.499 11.006 11 1.139 43.564 12 0.944 80.601 13 0.000 152.925 14 0.102 87.597 15 2.546 36.680 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 8622288 3316 0 0 4 3 1158 1158 0 993 0 11812 1 17129008 4510 0 0 12 12 2243 2243 0 0 0 24429 2 13474006 1553973 0 0 6 6 1869 1869 0 1550223 0 19028 3 26129234 1524704 0 0 20 20 3374 3374 0 1517916 0 27835 4 2168475 521045 0 0 0 0 352 352 0 520341 0 5247 5 15823578 326080 0 0 4 4 1963 1963 0 322146 0 16074 6 12515224 622754 0 0 2 2 1625 1625 0 619500 0 10516 7 23990039 458079 0 0 12 12 3099 3099 0 451857 0 18924 8 22108276 5660 0 0 13 13 2817 2817 0 0 0 24206 9 1062355 236 0 0 0 0 118 118 0 0 0 1949 10 41223894 10244 0 0 40 40 5082 5082 0 0 0 13136 11 20572413 4974 0 0 7 7 2480 2480 0 0 0 4577 12 14036482 3446 0 0 3 3 1720 1720 0 0 0 21370 13 3191210 818 0 0 0 0 409 409 0 0 0 1573 14 12949121 3198 0 0 3 3 1596 1596 0 0 0 13450 15 21875922 5438 0 0 7 7 2712 2712 0 0 0 3286 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:70812.570s ( 0.452 blocks/s, 0.028 Mb/s)
	316	Fri May 14 07:05:08 2021	CA	Friday 14th May 08:00 - 16:00 shift
08:00 CA takes over 08:28 Problem at UNILAC - no beam 09:30 all system wide checks ok except all FEE64 fail ADC calibration (no waveforms) WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc2 : 10 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp errors: Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 09:36 FEE64 Temperatures ok - attachment 1 Detector bias / leakage currents ok - attachment 2 Statistics - attachments 3-8 09:50 beam is back - writing to file R4_265 09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA 10:10 FRS increase degrader thickness 10.28 all histograms, stats and merger stats zero'd 10.50 all system wide checks ok except Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc2 : 10 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 3 2 2 2 2 2 2 4 3 3 6 : 36860 aida02 : 7 3 5 1 2 5 2 4 3 3 6 : 37284 aida03 : 7 5 3 1 4 4 2 3 3 3 6 : 36756 aida04 : 3 1 20 4 2 3 3 5 1 3 6 : 36052 aida05 : 19 25 26 6 2 2 3 2 2 3 6 : 35828 aida06 : 1 3 16 1 3 4 1 5 1 3 6 : 35580 aida07 : 6 4 1 2 3 3 2 3 3 3 6 : 36552 aida08 : 8 5 2 3 2 3 2 2 2 4 6 : 37064 aida09 : 9 7 1 2 2 4 2 3 3 3 6 : 36652 aida10 : 2 5 3 1 1 4 2 3 3 3 6 : 36544 aida11 : 16 4 4 2 1 2 2 3 3 3 6 : 36384 aida12 : 2 1 1 3 1 2 2 3 3 3 6 : 36288 aida13 : 0 1 1 2 0 4 3 2 3 3 6 : 36184 aida14 : 6 3 2 0 2 3 2 3 3 3 6 : 36432 aida15 : 25 8 2 1 2 3 1 2 2 4 6 : 36836 aida16 : 3 5 1 2 2 2 3 2 3 3 6 : 36100 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021 FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021 FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021 FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021 FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021 FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021 FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021 FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021 FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021 FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021 FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021 10.50 DAQ continues S496/R4_304 11:18 FEE64 Temperatures ok - attachment 9 Detector bias & leakage currents ok - attachment 10 statistics - attachments 11-16 15:00 FEE64 avg. CPU usage (%) 1 52 2 55 3 55 4 60 5 93 6 50 7 53 8 55 9 55 10 51 11 50 12 67 13 56 14 55 15 55 16 58 15:08 all system wide checks ok except all FEE64 fail ADC calibration (no waveform) WR decoder status: Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dc3 : 11 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA errors: Base Current Difference aida05 fault 0x0 : 0x3 : 3 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 15:10 Temperatures ok - attachment 17 Detector bias / leakage current ok - attachment 18 Statistics - attachments 18 -24
Attachment 1: Screenshot_from_2021-05-14_08-32-16.png

Attachment 2: Screenshot_from_2021-05-14_08-32-45.png

Attachment 3: Screenshot_from_2021-05-14_08-33-11.png

Attachment 4: Screenshot_from_2021-05-14_08-33-33.png

Attachment 5: Screenshot_from_2021-05-14_08-34-12.png

Attachment 6: Screenshot_from_2021-05-14_08-34-44.png

Attachment 7: Screenshot_from_2021-05-14_08-35-17.png

Attachment 8: Screenshot_from_2021-05-14_08-35-55.png

Attachment 9: Screenshot_from_2021-05-14_10-17-25.png

Attachment 10: Screenshot_from_2021-05-14_10-17-43.png

Attachment 11: Screenshot_from_2021-05-14_10-22-39.png

Attachment 12: Screenshot_from_2021-05-14_10-23-13.png

Attachment 13: Screenshot_from_2021-05-14_10-23-41.png

Attachment 14: Screenshot_from_2021-05-14_10-24-12.png

Attachment 15: Screenshot_from_2021-05-14_10-24-37.png

Attachment 16: Screenshot_from_2021-05-14_10-25-20.png

Attachment 17: Screenshot_from_2021-05-14_14-07-48.png

Attachment 18: Screenshot_from_2021-05-14_14-11-23.png

Attachment 19: Screenshot_from_2021-05-14_14-13-21.png

Attachment 20: Screenshot_from_2021-05-14_14-13-38.png

Attachment 21: Screenshot_from_2021-05-14_14-14-12.png

Attachment 22: Screenshot_from_2021-05-14_14-14-39.png

Attachment 23: Screenshot_from_2021-05-14_14-15-06.png

Attachment 24: Screenshot_from_2021-05-14_14-15-26.png

Goto page Previous 1, 2, 3 ... 18, 19, 20 ... 33, 34, 35 Next

ELOG V3.1.4-unknown