ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 1 of 35

Full | Summary | Threaded | Hide attachments

687 Entries

Goto page 1, 2, 3 ... 33, 34, 35 Next

ID	Date	Author	Subject
75	Thu Oct 31 15:24:35 2019	TD	waveform spectra issues?

Attachment 1: 20.png

Attachment 2: 21.png

Attachment 3: 22.png

Attachment 4: 23.png

Attachment 5: 30.png

Attachment 6: 31.png

Attachment 7: 32.png

Attachment 8: 33.png

Attachment 9: 40.png

Attachment 10: 41.png

Attachment 11: 42.png

Attachment 12: 43.png

273	Sat Apr 24 04:07:23 2021	ML	ucesb scalers
ucesb is back - see attachment
Attachment 1: 24-04-2021_5am_ucesb.png

228	Fri Apr 16 05:13:09 2021	ML	system wide checks
system wide checks: Done around 4am. ################################# To be noted: -1- White rabbit and FPGA checks failed for 1 FEE and passed for 11 -2- No waveform for AIDA09 in 1.8.L spectrum -3- AIDA 7 is red in the NewMerger control window (?) ################################# Details: Clock status test result: Passed 12, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Calibration test result: Passed 12, Failed 0 White Rabbit checks: Base Current Difference aida07 fault 0xfa8c : 0xfb37 : 171 White Rabbit error counter test result: Passed 11, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp check: Base Current Difference aida07 fault 0x0 : 0xe : 14 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Memory check: Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 45 9 7 3 4 3 2 3 3 3 6 : 36940 aida02 : 23 9 6 7 4 3 2 3 3 3 6 : 36964 aida03 : 40 8 5 4 5 3 2 3 3 3 6 : 36976 aida04 : 37 13 7 3 4 3 4 2 3 3 6 : 36940 aida05 : 35 11 6 3 4 4 2 3 2 4 6 : 38052 aida06 : 42 8 2 3 4 4 2 3 3 3 6 : 36968 aida07 : 42 10 5 5 5 3 2 3 3 3 6 : 37032 aida08 : 40 8 1 6 3 4 2 3 3 3 6 : 36976 aida09 : 18 18 14 8 5 5 5 4 3 2 6 : 36728 aida10 : 1 8 9 6 3 3 1 4 2 3 6 : 36052 aida11 : 24 11 14 6 2 2 4 2 3 3 6 : 36824 aida12 : 17 13 8 5 5 4 2 3 2 3 6 : 36108
Attachment 1: 16-04-2021_3h45am_HV.png

Attachment 2: 48_temperature.png

Attachment 3: 16-04-2021_3h51_StatGoodEvt.png

Attachment 4: 16-04-2021_3h53_StatADCdataItem.png

Attachment 5: 16-04-2021_3h58_StatWR#4.png

Attachment 6: 16-04-2021_4h04_StatWR#5.png

Attachment 7: 16-04-2021_3h59_StatPause#2.png

Attachment 8: 16-04-2021_4h01_StatResume#3.png

Attachment 9: 16-04-2021_4h02_StatCorrelation#8.png

Attachment 10: 16-04-2021_4h06_SpecRate.png

Attachment 11: 16-04-2021_4h08_SpecRate2.png

Attachment 12: 16-04-2021_4h17_Spec1.8L.png

Attachment 13: 16-04-2021_4h29_Spec1.8L_id4.png

Attachment 14: 16-04-2021_4h20_Spec1.8W_id7.png

Attachment 15: 25_Spec1.8W_id8.png

Attachment 16: 16-04-2021_4h48_NewMergerControl.png

240	Sun Apr 18 17:27:44 2021	ML-OH	system wide checks
system wide checks at 6pm CET ################################################## Difference noted from previous shift: FPGS timestamp error 10 passed and 2 failed ################################################## Clock status test result: Passed 12, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ADC Calibration: FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 9, Failed 3 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit checks: Base Current Difference aida01 fault 0x7686 : 0x768a : 4 aida02 fault 0x941d : 0x9421 : 4 aida03 fault 0x7cd7 : 0x7cdb : 4 aida04 fault 0xb86d : 0xb871 : 4 aida05 fault 0x1a59 : 0x1a61 : 8 aida06 fault 0x4f45 : 0x4f4d : 8 aida07 fault 0x3bfc : 0x3c11 : 21 aida08 fault 0xc7ce : 0xc7d5 : 7 aida09 fault 0xb33b : 0xb33c : 1 White Rabbit error counter test result: Passed 3, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp errors checks: Base Current Difference aida09 fault 0x1 : 0x2 : 1 aida12 fault 0xa : 0xc : 2 FPGA Timestamp error counter test result: Passed 10, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Memory information fro FEE64: Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 27 16 4 2 2 2 1 3 3 3 6 : 36332 aida02 : 19 8 9 3 2 2 2 4 2 3 6 : 36092 aida03 : 23 10 8 6 4 4 3 3 2 3 6 : 36332 aida04 : 27 4 8 0 1 3 4 3 2 3 6 : 36044 aida05 : 21 6 5 4 4 2 2 2 2 4 6 : 37204 aida06 : 23 11 5 0 3 4 1 3 3 3 6 : 36548 aida07 : 14 9 5 2 3 3 2 2 4 3 6 : 37200 aida08 : 26 8 7 2 2 1 4 3 2 3 6 : 35928 aida09 : 2 3 4 1 1 3 3 4 2 3 6 : 36160 aida10 : 16 9 2 5 1 1 3 4 2 3 6 : 36104 aida11 : 7 5 0 0 3 3 1 4 2 3 6 : 35716 aida12 : 9 7 3 2 4 4 1 3 3 3 6 : 36556
Attachment 1: 18-04-2021_6pm_CAENHV.png

Attachment 2: 18-04-2021_6pm_StatGoodEvent.png

Attachment 3: 18-04-2021_6pm_ADCdataItem.png

Attachment 4: 18-04-2021_6pm_WR28-47#4.png

Attachment 5: 18-04-2021_6pm_WR48-63#5.png

Attachment 6: 18-04-2021_6pm_PauseInfo.png

Attachment 7: 18-04-2021_6pm_ResumeInfo.png

Attachment 8: 18-04-2021_6pm_DiscInfo.png

Attachment 9: 18-04-2021_6pm_Correlation#8.png

Attachment 10: 18-04-2021_6pm_ucesb.png

Attachment 11: 18-04-2021_645pm_tempScan.png

Attachment 12: 18-04-2021_645pm_Spec1-8L.png

Attachment 13: 18-04-2021_7pm_Spec1-8L_2.png

Attachment 14: 18-04-2021_7pm_Spec1-8_layoutId4.png

Attachment 15: 18-04-2021_7pm_Spec1-8H.png

Attachment 16: 18-04-2021_645pm_SpecRate.png

Attachment 17: 18-04-2021_730pm_AIDA_LC.png

242	Sun Apr 18 21:43:01 2021	ML-OH	system wide checks
New System wide check 10pm CET Clock status test result: Passed 12, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ADC calibration check: FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 9, Failed 3 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0x7686 : 0x768a : 4 aida02 fault 0x941d : 0x9421 : 4 aida03 fault 0x7cd7 : 0x7cdb : 4 aida04 fault 0xb86d : 0xb871 : 4 aida05 fault 0x1a59 : 0x1a61 : 8 aida06 fault 0x4f45 : 0x4f4d : 8 aida07 fault 0x3bfc : 0x3c1d : 33 aida08 fault 0xc7ce : 0xc7d5 : 7 aida09 fault 0xb33b : 0xb33c : 1 White Rabbit error counter test result: Passed 3, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp error checks: Base Current Difference aida09 fault 0x1 : 0x2 : 1 aida12 fault 0xa : 0xf : 5 FPGA Timestamp error counter test result: Passed 10, Failed 2 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Memory from FEE64: Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 22 19 6 4 2 3 2 4 2 3 6 : 36304 aida02 : 23 9 3 3 1 3 1 4 2 3 6 : 35828 aida03 : 15 5 6 3 2 3 3 3 2 3 6 : 35876 aida04 : 20 8 18 1 1 3 4 3 2 3 6 : 36240 aida05 : 21 7 1 2 1 2 4 3 2 3 6 : 35868 aida06 : 18 8 4 1 3 4 1 3 3 3 6 : 36520 aida07 : 4 9 3 4 2 3 2 2 3 3 6 : 36104 aida08 : 15 5 2 4 2 1 3 4 2 3 6 : 36100 aida09 : 2 23 8 1 2 3 2 4 2 3 6 : 36192 aida10 : 10 7 8 5 2 1 3 3 2 3 6 : 35712 aida11 : 1 9 5 1 2 3 1 4 2 3 6 : 35772 aida12 : 14 8 6 3 3 4 1 3 3 3 6 : 36600
Attachment 1: 18-04-2021_10pm_HV.png

Attachment 2: 18-04-2021_10pm_StatGoodEvt.png

Attachment 3: 18-04-2021_10pm_PAUSE.png

Attachment 4: 18-04-2021_10pm_RESUME.png

Attachment 5: 18-04-2021_10pm_ADCdataItem.png

Attachment 6: 18-04-2021_10pm_DISC.png

Attachment 7: 18-04-2021_10pm_correlations.png

Attachment 8: 18-04-2021_10pm_SpecRate.png

Attachment 9: 18-04-2021_10pm_Spec1-8H.png

Attachment 10: 18-04-2021_10pm_Spec1-8HlayoutID6.png

Attachment 11: 18-04-2021_10pm_Spec1-8L.png

Attachment 12: 18-04-2021_10pm_Spec1-8L_layoutid4.png

Attachment 13: 18-04-2021_10pm_Spec1-8W.png

Attachment 14: 18-04-2021_10pm_SpecLayoutID8.png

Attachment 15: 18-04-2021_10pm_WR28-47.png

Attachment 16: 18-04-2021_10pm_WR48-63.png

Attachment 17: 18-04-2021_10pm_temp.png

Attachment 18: 18-04-2021_10pm_ucesb_monitor.png

Attachment 19: 18-04-2021_10pm-AIDA_Alerting_-_Grafana.png

Attachment 20: 18-04-2021_10pm_NewMerger.png

269	Sat Apr 24 01:44:09 2021	ML	system wide checks
system wide checks: Clock status: FEE64 module aida06 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ADC calibration: FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida12 failed Calibration test result: Passed 9, Failed 3 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White Rabbit Base Current Difference aida06 fault 0x679f : 0x67a4 : 5 White Rabbit error counter test result: Passed 11, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp error page not working !! Memory information: Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 19 15 4 6 4 3 2 4 2 3 6 : 36420 aida02 : 36 25 24 6 0 4 2 3 2 3 4 : 28056 aida03 : 17 7 4 3 3 5 2 2 3 3 6 : 36444 aida04 : 4 5 4 5 3 4 2 3 3 3 6 : 36824 aida05 : 23 9 4 2 3 3 1 3 3 3 6 : 36452 aida06 : 26 8 5 1 3 4 2 2 3 3 6 : 36312 aida07 : 21 9 3 3 4 5 1 2 3 3 6 : 36268 aida08 : 27 9 2 2 2 4 1 3 3 3 6 : 36500 aida09 : 13 4 3 3 1 2 4 2 3 3 6 : 36388 aida10 : 1 10 0 3 1 2 3 3 2 3 6 : 35572 aida11 : 23 13 8 3 3 4 3 3 2 3 6 : 36196 aida12 : 19 7 4 3 2 3 2 3 3 3 6 : 36644
Attachment 1: 24-04-2021_230am_AIDA_HV.png

Attachment 2: 24-04-2021_230am_ucesbStat.png

Attachment 3: 24-04-2021_230am_grafana.png

Attachment 4: 24-04-2021_230am_AidaGoodEvent.png

Attachment 5: 24-04-2021_230am_Correlation#8.png

Attachment 6: 24-04-2021_230am_RESUME#3.png

Attachment 7: 24-04-2021_230am_PAUSE#2.png

314	Thu May 13 21:21:25 2021	Philippos and Marc	system wide checks
Wide checks at Time 22:22 CET Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida01 fault 0xf932 : 0xf933 : 1 aida02 fault 0x62ec : 0x62ed : 1 aida03 fault 0x8679 : 0x867a : 1 aida04 fault 0xf0e4 : 0xf0e5 : 1 aida05 fault 0x9db8 : 0x9dbc : 4 aida06 fault 0x7f18 : 0x7f19 : 1 aida07 fault 0xdd2c : 0xdd2d : 1 aida08 fault 0x1557 : 0x1558 : 1 White Rabbit error counter test result: Passed 8, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Base Current Difference aida05 fault 0x0 : 0x2 : 2 FPGA Timestamp error counter test result: Passed 15, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 35 11 3 5 5 4 1 3 2 4 6 : 37876 aida02 : 20 5 4 3 2 5 2 4 3 3 6 : 37400 aida03 : 20 3 4 3 4 4 1 3 2 4 6 : 37640 aida04 : 25 6 2 3 4 4 3 3 3 3 6 : 37140 aida05 : 30 10 5 7 2 6 2 4 2 4 6 : 38776 aida06 : 21 11 4 4 3 5 2 4 3 3 6 : 37548 aida07 : 23 8 4 4 2 6 1 3 2 4 6 : 37852 aida08 : 27 7 2 3 4 3 2 4 3 3 6 : 37284 aida09 : 31 6 5 6 3 3 2 4 3 3 6 : 37372 aida10 : 22 5 3 4 3 6 2 2 2 4 6 : 37616 aida11 : 23 5 4 2 5 2 1 3 2 4 6 : 37444 aida12 : 20 7 4 7 3 5 2 3 3 3 6 : 37096 aida13 : 28 5 3 3 1 5 2 3 3 3 6 : 36840 aida14 : 38 12 2 4 2 4 1 4 3 3 6 : 37144 aida15 : 27 9 5 4 2 5 3 2 3 3 6 : 36740 aida16 : 21 6 4 4 1 4 2 3 3 3 6 : 36740
Attachment 1: 2021-05-13_ucesb_scalers_22h22mn.png

Attachment 2: 2021-05-13_Currents_22h22mn.png

Attachment 3: 2021-05-13_temperatures_22h22.png

Attachment 4: 2021-05-13-22h22mn_Stat-good-events.png

Attachment 5: 2021-05-13-22h22mn_ADC-data-items.png

Attachment 6: 2021-05-13_22h22mn-Correlation-info.png

Attachment 7: 2021-05-13_22h22mn-Pause-info.png

Attachment 8: 2021-05-13_22h22mn-Resume-Info.png

Attachment 9: 2021-05-13_merger_22h22.png

Attachment 10: 2021-05-13_22h22mn-LAY-ID1.png

Attachment 11: 2021-05-13_22h22mn-LAY-ID2.png

Attachment 12: 2021-05-13_22h22mn-LAY-ID3.png

Attachment 13: 2021-05-13_22h22mn-LAY-ID4.png

338	Thu May 20 23:26:11 2021	Marc Labiche (ML)	system wide checks
System Wide check at begining of the night shift: ----- Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ----- ADC calibration: FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module ----- White rabbit status: Base Current Difference aida05 fault 0x500 : 0x56d : 109 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR ----- FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last ----- Memory info from FEE cards Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 20 7 7 6 1 5 2 2 3 3 6 : 36472 aida02 : 19 5 4 4 4 3 3 3 2 3 6 : 36020 aida03 : 22 10 2 2 5 2 2 3 3 3 6 : 36680 aida04 : 12 7 1 0 4 3 1 4 3 3 6 : 36856 aida05 : 16 7 5 3 1 3 2 3 3 3 6 : 36584 aida06 : 20 7 5 6 2 4 4 2 3 3 6 : 36888 aida07 : 13 7 5 2 3 2 1 3 2 2 7 : 37308 aida08 : 18 7 5 3 0 2 3 3 3 3 6 : 36656 aida09 : 21 6 4 4 1 4 2 3 3 3 6 : 36740 aida10 : 21 10 2 3 2 3 2 2 2 4 6 : 37156 aida11 : 23 2 4 4 3 2 2 3 2 4 6 : 37612 aida12 : 24 1 4 3 2 2 1 2 4 3 6 : 36744 aida13 : 26 9 4 2 1 4 3 2 3 3 6 : 36464 aida14 : 19 9 9 2 3 4 1 2 2 4 6 : 37156 aida15 : 10 7 10 0 1 3 1 3 3 3 6 : 36288 aida16 : 28 7 3 2 1 3 3 3 3 3 6 : 36824
Attachment 1: 2021-05-21_0h20_ucesb.png

Attachment 2: 2021-05-21_0h10_Leakcurrent.png

Attachment 3: 2021-05-21_0h10_Stat_adcItem.png

Attachment 4: 2021-05-21_0h10_Stats_Correlation.png

Attachment 5: 2021-05-21_0h10_Stats_GoodEvents.png

Attachment 6: 2021-05-21_0h10_Stats_PAUSE.png

Attachment 7: 2021-05-21_0h10_stats_RESUME.png

Attachment 8: 2021-05-21_0h10_temperature.png

Attachment 9: 2021-05-21_0h30_AIDAgrafana.png

339	Fri May 21 02:17:47 2021	ML	system wide checks
Wide check at 3am - All fine. Same output as at the beginning of the shift (see my previous entry) and attachments.
Attachment 1: 2021-05-21_3h05_Leakcurrent.png

Attachment 2: 2021-05-21_3h05_stats_adcItem.png

Attachment 3: 2021-05-21_3h05_stats_Correlation.png

Attachment 4: 2021-05-21_3h05_stats_GoodEvents.png

Attachment 5: 2021-05-21_3h05_stats_PAUSE.png

Attachment 6: 2021-05-21_3h05_stats_RESUME.png

Attachment 7: 2021-05-21_3h05_temperature.png

Attachment 8: 2021-05-21_3h05_ucesb.png

Attachment 9: 2021-05-21_3h05_aidaGrafana.png

340	Fri May 21 06:09:28 2021	ML	system wide checks
System wide checks at 7am CET All seems fine, nothing significant to report Slight increase of dead time noted in FEE 3, 5 and 7. ----- Clock status test result: Passed 16, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ----- ADC calibration FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed FEE64 module aida09 failed FEE64 module aida10 failed FEE64 module aida11 failed FEE64 module aida12 failed FEE64 module aida13 failed FEE64 module aida14 failed FEE64 module aida15 failed FEE64 module aida16 failed Calibration test result: Passed 0, Failed 16 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module ----- White Rabbit: Base Current Difference aida05 fault 0x500 : 0x56e : 110 White Rabbit error counter test result: Passed 15, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR ----- FPGA Timestamp error counter test result: Passed 16, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last ----- Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 23 5 6 5 0 4 1 3 3 3 6 : 36484 aida02 : 10 3 3 2 3 2 2 2 3 3 6 : 35952 aida03 : 2 5 1 4 5 2 2 3 3 3 6 : 36608 aida04 : 1 3 4 0 4 3 2 3 3 3 6 : 36572 aida05 : 19 9 3 2 2 3 1 3 4 3 6 : 37380 aida06 : 13 11 1 5 3 4 4 2 3 3 6 : 36860 aida07 : 12 2 4 3 3 2 1 3 2 2 7 : 37280 aida08 : 25 5 7 1 1 3 3 3 3 3 6 : 36828 aida09 : 22 5 6 4 1 3 2 3 3 3 6 : 36640 aida10 : 23 8 6 3 2 3 2 2 2 4 6 : 37212 aida11 : 24 14 7 3 3 2 3 3 3 3 6 : 36960 aida12 : 16 10 5 3 1 2 1 2 4 3 6 : 36736 aida13 : 23 6 4 2 2 3 2 3 3 3 6 : 36620 aida14 : 22 7 8 4 3 4 1 2 2 4 6 : 37200 aida15 : 10 12 9 2 1 3 1 3 3 3 6 : 36376 aida16 : 18 5 6 2 0 3 2 4 3 3 6 : 37008
Attachment 1: 2021-05-21_7h00_LeakCurrent.png

Attachment 2: 2021-05-21_7h00_Stats_GoodEvents.png

Attachment 3: 2021-05-21_7h00_Stats_ADCitems.png

Attachment 4: 2021-05-21_7h00_stats_Correlation.png

Attachment 5: 2021-05-21_7h00_Stats_PAUSE.png

Attachment 6: 2021-05-21_7h00_Stats_RESUME.png

Attachment 7: 2021-05-21_7h00_temperature.png

Attachment 8: 2021-05-21_7h00_ucesb.png

141	Wed Mar 11 16:29:17 2020	CA, LS, TD	slow comparator threshold comparison
17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1 17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2 17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3 17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4
Attachment 1: 100keV_stats.png

Attachment 2: 110keVStats.png

Attachment 3: 120kevstats.png

Attachment 4: 80kevDSSD2.png

45	Wed Apr 3 15:24:06 2019	CA, TD	report - low inquire save state
On Save/Restore Module Settings Tab - "inquire save state" returns an error. See attachment 1.
Attachment 1: 030419_1614_savestate.png

44	Wed Apr 3 13:29:20 2019	CA, CB, TD	report - high DAQ stall
aida10 runs very fast for a period, then stops. Have to restart Merger and TapeServer to get it running again. As shown in attachment 1, aida10 records very high number of bad events. Waveform for all FEE64s are disabled (see for example attachment 4) BUT note that 'wave events' are reported in aida10 stats (attachment 1)
Attachment 1: 030419_1416_aida10stats.png

Attachment 2: 030419_1417_aida09stats.png

Attachment 3: 030419_1417_aida11stats.png

Attachment 4: 030419_1418_aida12stats.png

Attachment 5: 030419_1419_aida10_led.png

272	Sat Apr 24 03:32:11 2021	ML	quick checks
Quick system checks after crash Temperature ok. Still cannot connect remotely to /ucesb with my gsi web login and password (?). Clock status test result: Passed 12, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable Calibration test result: Passed 12, Failed 0 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x31a2 : 0x31a2 : 0 aida05 : WR status 0x10 White Rabbit error counter test result: Passed 11, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 9 3 4 2 3 2 1 4 2 3 7 : 39804 aida02 : 29 4 2 3 0 3 1 4 2 3 7 : 39828 aida03 : 20 6 1 2 2 3 1 4 2 3 7 : 39888 aida04 : 19 5 4 2 1 3 1 4 2 3 7 : 39860 aida05 : 27 5 4 1 2 3 2 3 1 4 7 : 40692 aida06 : 17 10 3 2 1 3 1 4 2 3 7 : 39876 aida07 : 22 7 2 2 2 3 1 4 2 3 7 : 39920 aida08 : 12 6 5 1 3 2 1 4 2 3 7 : 39824 aida09 : 0 9 1 3 1 3 1 4 2 3 7 : 39800 aida10 : 2 2 2 3 3 2 3 2 2 3 7 : 39256 aida11 : 21 7 4 3 0 3 2 3 2 3 7 : 39596 aida12 : 18 8 3 0 2 3 2 3 2 3 7 : 39608
Attachment 1: 24-04-2021_430am_GoodEvent.png

160	Tue Jul 7 08:50:27 2020	NH	nnrpi2 does not boot
MIDAS was not working properly on nnrpi2 - power cycled. Now system does not boot properly, it gets stuck trying to login claiming "PAM critical error" - probable file system corruption? Will investigate if SD card can be checked and repaired
212	Fri Apr 9 16:40:36 2021	NH	nnrpi1 Update
nnrpi1 is now back in S4 with a new SD card and new raspberry pi. The freezing originally continued (it was the USB system failing) but was fixed with a change to /boot/cmdline.txt coherent_pool=4M It seems the kernel update to fix one bug introduced another... Now it is running again with all TTYs connected. I think one TTY may not be working but 11 should be
208	Tue Mar 30 13:21:50 2021	NH	nnrpi1 (tty logs)
We had issues with all TTYs reading out on nnrpi1, I have moved them from an apparently faulty USB hub to a new one. Trying to reboot the pi to detect the new USB ports and it didn't recover from boot Trying to repair SD card but I get a lot of errors on my laptop, indications are the SD card is completely dead A new SD card will be ordered and nnrpi1 restored from the original images A new USB hub should also be organised as it seems one is faulty (pi hangs if 3 or more USB devices connected to it)
161	Tue Oct 6 15:48:06 2020	NH	nnrpi kernel update
FYI: The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were huge and filled with the same error many times a second Oct 4 15:01:22 raspberrypi kernel: [1030048.603846] Oct 4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode Oct 4 15:01:22 raspberrypi kernel: [1030048.607742] Oct 4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode Oct 4 15:01:22 raspberrypi kernel: [1030048.667755] Oct 4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode Oct 4 15:01:22 raspberrypi kernel: [1030048.667848] Oct 4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode Oct 4 15:01:22 raspberrypi kernel: [1030048.731753] A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version. now the message doesn't seem to appear so hopefully the pis won't break so rapidly.
503	Sun Jun 26 23:04:30 2022	Marc	new shift - Monday 27 June 0:00 to 8:00
0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature). Stats & Temperatures (VIRTEX,PSU, ASICs) all ok. At 0:30 Stats ok - Attachment 1 Temp ok - Attachement 2 HV-LC -Attachment 3 At 2:20 Stats ok - Attachment 4 Temp ok - Attachement 5 HV-LC -Attachment 6 Wide Checks: Clock status test result: Passed 8, Failed 0 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable ADC Calibration (same as before): FEE64 module aida01 failed FEE64 module aida02 failed FEE64 module aida03 failed FEE64 module aida04 failed FEE64 module aida05 failed FEE64 module aida06 failed FEE64 module aida07 failed FEE64 module aida08 failed Calibration test result: Passed 0, Failed 8 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module WR decoder status: Base Current Difference aida07 fault 0xc53d : 0xc5c9 : 140 aida08 fault 0xf1be : 0xf2b2 : 244 White Rabbit error counter test result: Passed 6, Failed 2 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp check: Base Current Difference aida07 fault 0x2a : 0x41 : 23 FPGA Timestamp error counter test result: Passed 7, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last At 4:15: Stats ok - Attachment 7 Temp ok - Attachement 8 HV-LC -Attachment 9 Wide Checks: No change At 7:15: (no beam since ~6am -> background run) Stats ok - Attachment 10 Temp ok - Attachement 11 HV-LC -Attachment 12 Wide Checks: No change
Attachment 1: Stats-Screenshot_from_2022-06-27_00-30-58.png

Attachment 2: Temp-Screenshot_from_2022-06-27_00-31-44.png

Attachment 3: HV-LC-Screenshot_from_2022-06-27_00-30-18.png

Attachment 4: Stats-Screenshot_from_2022-06-27_02-22-34.png

Attachment 5: Temp-Screenshot_from_2022-06-27_02-19-42.png

Attachment 6: HV-LC-Screenshot_from_2022-06-27_02-19-05.png

Attachment 7: Stats-Screenshot_from_2022-06-27_04-17-22.png

Attachment 8: Temp-Screenshot_from_2022-06-27_04-16-40.png

Attachment 9: HV-LC-Screenshot_from_2022-06-27_04-14-57.png

Attachment 10: Stats-Screenshot_from_2022-06-27_07-15-17.png

Attachment 11: Temp-Screenshot_from_2022-06-27_07-14-02.png

Attachment 12: HV-LC-Screenshot_from_2022-06-27_07-13-09.png

437	Fri May 13 02:28:36 2022	ML	beam dropped
We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz). This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

Goto page 1, 2, 3 ... 33, 34, 35 Next

ELOG V3.1.4-unknown