AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 27 of 36  ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
ID Date Authordown Subject
  708   Tue Jul 22 17:00:26 2025 MP, CCStatus

Test with shielded HV cables for both detectors.

The FWHM for ch1 is now 184 ch.

Yesterday cables for cards 5, 6, 10, 11, 12, 14, 16 were partially connected (only 1 out of 2 flat cables was in), now both are connected.

Attachment 1: Screenshot_from_2025-07-22_17-51-52.png
Screenshot_from_2025-07-22_17-51-52.png
Attachment 2: Screenshot_from_2025-07-22_17-59-20.png
Screenshot_from_2025-07-22_17-59-20.png
  707   Mon Jul 21 13:01:19 2025 MPStatus

Status before grounding checks

Attachment 1: Screenshot_from_2025-07-21_14-00-23.png
Screenshot_from_2025-07-21_14-00-23.png
Attachment 2: Screenshot_from_2025-07-21_14-00-39.png
Screenshot_from_2025-07-21_14-00-39.png
  240   Sun Apr 18 17:27:44 2021 ML-OHsystem wide checks

system wide checks at 6pm CET

##################################################

Difference noted from previous shift:

FPGS timestamp error 10 passed and 2 failed

##################################################


Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit checks:

         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c11 :      21  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA timestamp errors checks:

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0xc :      2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Memory information fro FEE64:

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     16      4      2      2      2      1      3      3      3      6   : 36332
aida02 :     19      8      9      3      2      2      2      4      2      3      6   : 36092
aida03 :     23     10      8      6      4      4      3      3      2      3      6   : 36332
aida04 :     27      4      8      0      1      3      4      3      2      3      6   : 36044
aida05 :     21      6      5      4      4      2      2      2      2      4      6   : 37204
aida06 :     23     11      5      0      3      4      1      3      3      3      6   : 36548
aida07 :     14      9      5      2      3      3      2      2      4      3      6   : 37200
aida08 :     26      8      7      2      2      1      4      3      2      3      6   : 35928
aida09 :      2      3      4      1      1      3      3      4      2      3      6   : 36160
aida10 :     16      9      2      5      1      1      3      4      2      3      6   : 36104
aida11 :      7      5      0      0      3      3      1      4      2      3      6   : 35716
aida12 :      9      7      3      2      4      4      1      3      3      3      6   : 36556

 

 

 

Attachment 1: 18-04-2021_6pm_CAENHV.png
18-04-2021_6pm_CAENHV.png
Attachment 2: 18-04-2021_6pm_StatGoodEvent.png
18-04-2021_6pm_StatGoodEvent.png
Attachment 3: 18-04-2021_6pm_ADCdataItem.png
18-04-2021_6pm_ADCdataItem.png
Attachment 4: 18-04-2021_6pm_WR28-47#4.png
18-04-2021_6pm_WR28-47#4.png
Attachment 5: 18-04-2021_6pm_WR48-63#5.png
18-04-2021_6pm_WR48-63#5.png
Attachment 6: 18-04-2021_6pm_PauseInfo.png
18-04-2021_6pm_PauseInfo.png
Attachment 7: 18-04-2021_6pm_ResumeInfo.png
18-04-2021_6pm_ResumeInfo.png
Attachment 8: 18-04-2021_6pm_DiscInfo.png
18-04-2021_6pm_DiscInfo.png
Attachment 9: 18-04-2021_6pm_Correlation#8.png
18-04-2021_6pm_Correlation#8.png
Attachment 10: 18-04-2021_6pm_ucesb.png
18-04-2021_6pm_ucesb.png
Attachment 11: 18-04-2021_645pm_tempScan.png
18-04-2021_645pm_tempScan.png
Attachment 12: 18-04-2021_645pm_Spec1-8L.png
18-04-2021_645pm_Spec1-8L.png
Attachment 13: 18-04-2021_7pm_Spec1-8L_2.png
18-04-2021_7pm_Spec1-8L_2.png
Attachment 14: 18-04-2021_7pm_Spec1-8_layoutId4.png
18-04-2021_7pm_Spec1-8_layoutId4.png
Attachment 15: 18-04-2021_7pm_Spec1-8H.png
18-04-2021_7pm_Spec1-8H.png
Attachment 16: 18-04-2021_645pm_SpecRate.png
18-04-2021_645pm_SpecRate.png
Attachment 17: 18-04-2021_730pm_AIDA_LC.png
18-04-2021_730pm_AIDA_LC.png
  241   Sun Apr 18 19:05:33 2021 ML-OHAll spectra reset

 All spectra reset at 8pm CET.

 

  242   Sun Apr 18 21:43:01 2021 ML-OHsystem wide checks

New System wide check 10pm CET

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration check:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c1d :      33  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  


White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error checks:

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Memory from FEE64:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22     19      6      4      2      3      2      4      2      3      6   : 36304
aida02 :     23      9      3      3      1      3      1      4      2      3      6   : 35828
aida03 :     15      5      6      3      2      3      3      3      2      3      6   : 35876
aida04 :     20      8     18      1      1      3      4      3      2      3      6   : 36240
aida05 :     21      7      1      2      1      2      4      3      2      3      6   : 35868
aida06 :     18      8      4      1      3      4      1      3      3      3      6   : 36520
aida07 :      4      9      3      4      2      3      2      2      3      3      6   : 36104
aida08 :     15      5      2      4      2      1      3      4      2      3      6   : 36100
aida09 :      2     23      8      1      2      3      2      4      2      3      6   : 36192
aida10 :     10      7      8      5      2      1      3      3      2      3      6   : 35712
aida11 :      1      9      5      1      2      3      1      4      2      3      6   : 35772
aida12 :     14      8      6      3      3      4      1      3      3      3      6   : 36600

Attachment 1: 18-04-2021_10pm_HV.png
18-04-2021_10pm_HV.png
Attachment 2: 18-04-2021_10pm_StatGoodEvt.png
18-04-2021_10pm_StatGoodEvt.png
Attachment 3: 18-04-2021_10pm_PAUSE.png
18-04-2021_10pm_PAUSE.png
Attachment 4: 18-04-2021_10pm_RESUME.png
18-04-2021_10pm_RESUME.png
Attachment 5: 18-04-2021_10pm_ADCdataItem.png
18-04-2021_10pm_ADCdataItem.png
Attachment 6: 18-04-2021_10pm_DISC.png
18-04-2021_10pm_DISC.png
Attachment 7: 18-04-2021_10pm_correlations.png
18-04-2021_10pm_correlations.png
Attachment 8: 18-04-2021_10pm_SpecRate.png
18-04-2021_10pm_SpecRate.png
Attachment 9: 18-04-2021_10pm_Spec1-8H.png
18-04-2021_10pm_Spec1-8H.png
Attachment 10: 18-04-2021_10pm_Spec1-8HlayoutID6.png
18-04-2021_10pm_Spec1-8HlayoutID6.png
Attachment 11: 18-04-2021_10pm_Spec1-8L.png
18-04-2021_10pm_Spec1-8L.png
Attachment 12: 18-04-2021_10pm_Spec1-8L_layoutid4.png
18-04-2021_10pm_Spec1-8L_layoutid4.png
Attachment 13: 18-04-2021_10pm_Spec1-8W.png
18-04-2021_10pm_Spec1-8W.png
Attachment 14: 18-04-2021_10pm_SpecLayoutID8.png
18-04-2021_10pm_SpecLayoutID8.png
Attachment 15: 18-04-2021_10pm_WR28-47.png
18-04-2021_10pm_WR28-47.png
Attachment 16: 18-04-2021_10pm_WR48-63.png
18-04-2021_10pm_WR48-63.png
Attachment 17: 18-04-2021_10pm_temp.png
18-04-2021_10pm_temp.png
Attachment 18: 18-04-2021_10pm_ucesb_monitor.png
18-04-2021_10pm_ucesb_monitor.png
Attachment 19: 18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
Attachment 20: 18-04-2021_10pm_NewMerger.png
18-04-2021_10pm_NewMerger.png
  228   Fri Apr 16 05:13:09 2021 MLsystem wide checks

system wide checks:

Done around 4am.

#################################

To be noted:

-1- White rabbit and FPGA checks failed for 1 FEE and passed for 11

-2- No waveform for AIDA09 in 1.8.L spectrum

-3- AIDA 7 is red in the NewMerger control window (?)

#################################

 

Details:

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

White Rabbit checks:

         Base         Current     Difference
aida07 fault      0xfa8c :      0xfb37 :      171  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp check:


             Base         Current         Difference
aida07 fault      0x0 :      0xe :      14  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Memory check:

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     45      9      7      3      4      3      2      3      3      3      6   : 36940
aida02 :     23      9      6      7      4      3      2      3      3      3      6   : 36964
aida03 :     40      8      5      4      5      3      2      3      3      3      6   : 36976
aida04 :     37     13      7      3      4      3      4      2      3      3      6   : 36940
aida05 :     35     11      6      3      4      4      2      3      2      4      6   : 38052
aida06 :     42      8      2      3      4      4      2      3      3      3      6   : 36968
aida07 :     42     10      5      5      5      3      2      3      3      3      6   : 37032
aida08 :     40      8      1      6      3      4      2      3      3      3      6   : 36976
aida09 :     18     18     14      8      5      5      5      4      3      2      6   : 36728
aida10 :      1      8      9      6      3      3      1      4      2      3      6   : 36052
aida11 :     24     11     14      6      2      2      4      2      3      3      6   : 36824
aida12 :     17     13      8      5      5      4      2      3      2      3      6   : 36108

Attachment 1: 16-04-2021_3h45am_HV.png
16-04-2021_3h45am_HV.png
Attachment 2: 48_temperature.png
48_temperature.png
Attachment 3: 16-04-2021_3h51_StatGoodEvt.png
16-04-2021_3h51_StatGoodEvt.png
Attachment 4: 16-04-2021_3h53_StatADCdataItem.png
16-04-2021_3h53_StatADCdataItem.png
Attachment 5: 16-04-2021_3h58_StatWR#4.png
16-04-2021_3h58_StatWR#4.png
Attachment 6: 16-04-2021_4h04_StatWR#5.png
16-04-2021_4h04_StatWR#5.png
Attachment 7: 16-04-2021_3h59_StatPause#2.png
16-04-2021_3h59_StatPause#2.png
Attachment 8: 16-04-2021_4h01_StatResume#3.png
16-04-2021_4h01_StatResume#3.png
Attachment 9: 16-04-2021_4h02_StatCorrelation#8.png
16-04-2021_4h02_StatCorrelation#8.png
Attachment 10: 16-04-2021_4h06_SpecRate.png
16-04-2021_4h06_SpecRate.png
Attachment 11: 16-04-2021_4h08_SpecRate2.png
16-04-2021_4h08_SpecRate2.png
Attachment 12: 16-04-2021_4h17_Spec1.8L.png
16-04-2021_4h17_Spec1.8L.png
Attachment 13: 16-04-2021_4h29_Spec1.8L_id4.png
16-04-2021_4h29_Spec1.8L_id4.png
Attachment 14: 16-04-2021_4h20_Spec1.8W_id7.png
16-04-2021_4h20_Spec1.8W_id7.png
Attachment 15: 25_Spec1.8W_id8.png
25_Spec1.8W_id8.png
Attachment 16: 16-04-2021_4h48_NewMergerControl.png
16-04-2021_4h48_NewMergerControl.png
  269   Sat Apr 24 01:44:09 2021 MLsystem wide checks

system wide checks:

 

 

Clock status:
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White Rabbit

         Base         Current     Difference
aida06 fault      0x679f :      0x67a4 :      5  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error page not working !!

Memory information:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     15      4      6      4      3      2      4      2      3      6   : 36420
aida02 :     36     25     24      6      0      4      2      3      2      3      4   : 28056
aida03 :     17      7      4      3      3      5      2      2      3      3      6   : 36444
aida04 :      4      5      4      5      3      4      2      3      3      3      6   : 36824
aida05 :     23      9      4      2      3      3      1      3      3      3      6   : 36452
aida06 :     26      8      5      1      3      4      2      2      3      3      6   : 36312
aida07 :     21      9      3      3      4      5      1      2      3      3      6   : 36268
aida08 :     27      9      2      2      2      4      1      3      3      3      6   : 36500
aida09 :     13      4      3      3      1      2      4      2      3      3      6   : 36388
aida10 :      1     10      0      3      1      2      3      3      2      3      6   : 35572
aida11 :     23     13      8      3      3      4      3      3      2      3      6   : 36196
aida12 :     19      7      4      3      2      3      2      3      3      3      6   : 36644

 

Attachment 1: 24-04-2021_230am_AIDA_HV.png
24-04-2021_230am_AIDA_HV.png
Attachment 2: 24-04-2021_230am_ucesbStat.png
24-04-2021_230am_ucesbStat.png
Attachment 3: 24-04-2021_230am_grafana.png
24-04-2021_230am_grafana.png
Attachment 4: 24-04-2021_230am_AidaGoodEvent.png
24-04-2021_230am_AidaGoodEvent.png
Attachment 5: 24-04-2021_230am_Correlation#8.png
24-04-2021_230am_Correlation#8.png
Attachment 6: 24-04-2021_230am_RESUME#3.png
24-04-2021_230am_RESUME#3.png
Attachment 7: 24-04-2021_230am_PAUSE#2.png
24-04-2021_230am_PAUSE#2.png
  270   Sat Apr 24 03:13:05 2021 MLAIDA DAQ Reset

AIDA PC crashe completely at ~3:30am CET.

Tom was called and he Restarted the AIDA system at ~4am with Run 52.

Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.

 

  272   Sat Apr 24 03:32:11 2021 MLquick checks

Quick system checks after crash

Temperature ok.

Still cannot connect remotely to /ucesb with my gsi web login and password (?).

 

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida05 fault      0x31a2 :      0x31a2 :      0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      9      3      4      2      3      2      1      4      2      3      7   : 39804
aida02 :     29      4      2      3      0      3      1      4      2      3      7   : 39828
aida03 :     20      6      1      2      2      3      1      4      2      3      7   : 39888
aida04 :     19      5      4      2      1      3      1      4      2      3      7   : 39860
aida05 :     27      5      4      1      2      3      2      3      1      4      7   : 40692
aida06 :     17     10      3      2      1      3      1      4      2      3      7   : 39876
aida07 :     22      7      2      2      2      3      1      4      2      3      7   : 39920
aida08 :     12      6      5      1      3      2      1      4      2      3      7   : 39824
aida09 :      0      9      1      3      1      3      1      4      2      3      7   : 39800
aida10 :      2      2      2      3      3      2      3      2      2      3      7   : 39256
aida11 :     21      7      4      3      0      3      2      3      2      3      7   : 39596
aida12 :     18      8      3      0      2      3      2      3      2      3      7   : 39608

 

Attachment 1: 24-04-2021_430am_GoodEvent.png
24-04-2021_430am_GoodEvent.png
  273   Sat Apr 24 04:07:23 2021 MLucesb scalers

ucesb is back - see attachment

Attachment 1: 24-04-2021_5am_ucesb.png
24-04-2021_5am_ucesb.png
  274   Sat Apr 24 04:50:01 2021 MLSpectra

Some screenshot of spectra attached

Attachment 1: 24-04-2021_540am_SpecLayoutId3.png
24-04-2021_540am_SpecLayoutId3.png
Attachment 2: 24-04-2021_540am_SpecLayId4.png
24-04-2021_540am_SpecLayId4.png
Attachment 3: 24-04-2021_540am_SpecLayId5.png
24-04-2021_540am_SpecLayId5.png
Attachment 4: 24-04-2021_540am_SpecLayId6.png
24-04-2021_540am_SpecLayId6.png
Attachment 5: 24-04-2021_540am_SpecLayId7.png
24-04-2021_540am_SpecLayId7.png
Attachment 6: 24-04-2021_540am_SpecLayId8.png
24-04-2021_540am_SpecLayId8.png
Attachment 7: 24-04-2021_540am_SpecLayId1.png
24-04-2021_540am_SpecLayId1.png
Attachment 8: 24-04-2021_540am_SpecLayId2.png
24-04-2021_540am_SpecLayId2.png
  275   Sat Apr 24 05:17:51 2021 MLEnd of beam

Beam has now stopped (end of expriment) and background measurements starts.

  339   Fri May 21 02:17:47 2021 MLsystem wide checks

Wide check at 3am -

All fine.

Same output as at the beginning of the shift (see my previous entry) and attachments.

 

Attachment 1: 2021-05-21_3h05_Leakcurrent.png
2021-05-21_3h05_Leakcurrent.png
Attachment 2: 2021-05-21_3h05_stats_adcItem.png
2021-05-21_3h05_stats_adcItem.png
Attachment 3: 2021-05-21_3h05_stats_Correlation.png
2021-05-21_3h05_stats_Correlation.png
Attachment 4: 2021-05-21_3h05_stats_GoodEvents.png
2021-05-21_3h05_stats_GoodEvents.png
Attachment 5: 2021-05-21_3h05_stats_PAUSE.png
2021-05-21_3h05_stats_PAUSE.png
Attachment 6: 2021-05-21_3h05_stats_RESUME.png
2021-05-21_3h05_stats_RESUME.png
Attachment 7: 2021-05-21_3h05_temperature.png
2021-05-21_3h05_temperature.png
Attachment 8: 2021-05-21_3h05_ucesb.png
2021-05-21_3h05_ucesb.png
Attachment 9: 2021-05-21_3h05_aidaGrafana.png
2021-05-21_3h05_aidaGrafana.png
  340   Fri May 21 06:09:28 2021 MLsystem wide checks

System  wide checks at 7am CET

All seems fine, nothing significant to report

Slight increase of dead time noted in FEE 3, 5 and 7.

-----
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


-----

White Rabbit:

         Base         Current     Difference
aida05 fault      0x500 :      0x56e :      110  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     23      5      6      5      0      4      1      3      3      3      6   : 36484
aida02 :     10      3      3      2      3      2      2      2      3      3      6   : 35952
aida03 :      2      5      1      4      5      2      2      3      3      3      6   : 36608
aida04 :      1      3      4      0      4      3      2      3      3      3      6   : 36572
aida05 :     19      9      3      2      2      3      1      3      4      3      6   : 37380
aida06 :     13     11      1      5      3      4      4      2      3      3      6   : 36860
aida07 :     12      2      4      3      3      2      1      3      2      2      7   : 37280
aida08 :     25      5      7      1      1      3      3      3      3      3      6   : 36828
aida09 :     22      5      6      4      1      3      2      3      3      3      6   : 36640
aida10 :     23      8      6      3      2      3      2      2      2      4      6   : 37212
aida11 :     24     14      7      3      3      2      3      3      3      3      6   : 36960
aida12 :     16     10      5      3      1      2      1      2      4      3      6   : 36736
aida13 :     23      6      4      2      2      3      2      3      3      3      6   : 36620
aida14 :     22      7      8      4      3      4      1      2      2      4      6   : 37200
aida15 :     10     12      9      2      1      3      1      3      3      3      6   : 36376
aida16 :     18      5      6      2      0      3      2      4      3      3      6   : 37008

 

Attachment 1: 2021-05-21_7h00_LeakCurrent.png
2021-05-21_7h00_LeakCurrent.png
Attachment 2: 2021-05-21_7h00_Stats_GoodEvents.png
2021-05-21_7h00_Stats_GoodEvents.png
Attachment 3: 2021-05-21_7h00_Stats_ADCitems.png
2021-05-21_7h00_Stats_ADCitems.png
Attachment 4: 2021-05-21_7h00_stats_Correlation.png
2021-05-21_7h00_stats_Correlation.png
Attachment 5: 2021-05-21_7h00_Stats_PAUSE.png
2021-05-21_7h00_Stats_PAUSE.png
Attachment 6: 2021-05-21_7h00_Stats_RESUME.png
2021-05-21_7h00_Stats_RESUME.png
Attachment 7: 2021-05-21_7h00_temperature.png
2021-05-21_7h00_temperature.png
Attachment 8: 2021-05-21_7h00_ucesb.png
2021-05-21_7h00_ucesb.png
  436   Fri May 13 00:50:16 2022 ML2h-Shift Checks

All seems ok  (apart for my virgin network connection usually quite reliable but not so tonight !?!).

Statistics seems ok.

Temperatures appears stable

Leakage current ok.

System Wide Checks:

Clock check: Passed= 14: - Failed=: 0

ADC calibration check:

FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES

White rabbit decode status: Passed=14 ; Failed=0

FPGA timestamp check: Passed=14 ; Failed=0

Memory information: Page loading failed with the error message below:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"

Attachment 1: Stats_Screenshot_from_2022-05-13_01-45-56.png
Stats_Screenshot_from_2022-05-13_01-45-56.png
Attachment 2: Temp_Screenshot_from_2022-05-13_01-47-31.png
Temp_Screenshot_from_2022-05-13_01-47-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_01-48-21.png
HV_LC_Screenshot_from_2022-05-13_01-48-21.png
  437   Fri May 13 02:28:36 2022 MLbeam dropped

We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).

This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

  438   Fri May 13 02:56:36 2022 ML2h-Shift Checks

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

Attachment 1: Stats_Screenshot_from_2022-05-13_03-43-34.png
Stats_Screenshot_from_2022-05-13_03-43-34.png
Attachment 2: Temp_Screenshot_from_2022-05-13_03-44-41.png
Temp_Screenshot_from_2022-05-13_03-44-41.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_03-41-28.png
HV_LC_Screenshot_from_2022-05-13_03-41-28.png
  439   Fri May 13 04:45:23 2022 ML2h-Shift Checks

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

Attachment 1: Stats_Screenshot_from_2022-05-13_05-43-14.png
Stats_Screenshot_from_2022-05-13_05-43-14.png
Attachment 2: Temp_Screenshot_from_2022-05-13_05-42-10.png
Temp_Screenshot_from_2022-05-13_05-42-10.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_05-41-23.png
HV_LC_Screenshot_from_2022-05-13_05-41-23.png
  440   Fri May 13 06:47:33 2022 ML2h-Shift Checks

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

Attachment 1: Stats_Screenshot_from_2022-05-13_07-43-54.png
Stats_Screenshot_from_2022-05-13_07-43-54.png
Attachment 2: Temp_Screenshot_from_2022-05-13_07-43-13.png
Temp_Screenshot_from_2022-05-13_07-43-13.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_07-42-38.png
HV_LC_Screenshot_from_2022-05-13_07-42-38.png
  444   Fri May 13 23:37:59 2022 ML2h-Shift Checks
Took over remotely from CB  about an hour ago

At 0h40 (CET):
      Stats still ok - attach 1
      Leakage still OK - Attach 2
      Temps - attach 3
      Grafana OK
      UCESB rates ok - attach 5
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_00-40-56.png
Stats_Screenshot_from_2022-05-14_00-40-56.png
Attachment 2: HV_LC_Screenshot_from_2022-05-14_00-39-51.png
HV_LC_Screenshot_from_2022-05-14_00-39-51.png
Attachment 3: temp_Screenshot_from_2022-05-14_00-41-44.png
temp_Screenshot_from_2022-05-14_00-41-44.png
Attachment 4: UCESB_Screenshot_from_2022-05-14_01-00-00.png
UCESB_Screenshot_from_2022-05-14_01-00-00.png
ELOG V3.1.4-unknown