AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 9 of 35  ELOG logo
  ID Date Author Subjectdown
  371   Thu Jun 17 11:14:04 2021 TDThursday 17 June
12.15 DAQ continues OK - no storage mode

12.15 System checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d6 : 	 13014  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     31     11      9      4      4      4      2      2      2      4      6   : 37604
aida02 :     30     10     10      3      3      4      3      2      2      4      6   : 37768
aida03 :     35     11      8      6      5      3      2      2      3      3      6   : 36580
aida04 :     33      9      4      4      3      4      2      3      3      3      6   : 36940
aida05 :     41      9      7      3      4      4      2      2      2      4      6   : 37564
aida06 :     38      6      5      3      2      3      2      3      4      3      6   : 37752
aida07 :     25     12      6      5      4      2      3      2      2      4      6   : 37572
aida08 :     32      7     10      5      4      3      3      2      2      4      6   : 37752
aida09 :     41      8      6      4      2      3      3      2      2      4      6   : 37572
aida10 :     31     10     11      5      5      3      3      3      3      3      6   : 37340
aida11 :     29      7     12      3      4      4      3      3      1      4      6   : 37324
aida12 :     42      8      7      5      3      4      3      3      1      4      6   : 37304
aida13 :     43     10      4      5      5      3      3      3      1      4      6   : 37276
aida14 :     40     15      4      5      4      4      3      3      3      3      6   : 37368
aida15 :     13      5      8      2      1      2      3      2      1      4      7   : 40284
aida16 :     39      9      6      6      5      3      3      3      1      4      6   : 37316

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

12.15 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-17_Statistics_aidas-gsi.png
Screenshot_2021-06-17_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1008.png
1008.png
Attachment 8: 1009.png
1009.png
  224   Thu Apr 15 22:38:18 2021 OHThursday 15th April - 16th April
23:38 They are testing the finger detector
      It is a lot of U they are putting into it which is passing through to AIDA ~3500Hz
      Most seems to be punching through

23:45 Rate seems to be reduced to peaking at 1000Hz

00:06 DAQ Running to no storage
      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa


      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse
  223   Thu Apr 15 13:28:07 2021 TDThursday 15 April
13.28 System wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida01 fault 	 0x44c2 : 	 0x44c5 : 	 3  
aida02 fault 	 0xc81f : 	 0xc822 : 	 3  
aida03 fault 	 0x9342 : 	 0x9344 : 	 2  
aida04 fault 	 0xdb3f : 	 0xdb41 : 	 2  
aida05 fault 	 0x4baf : 	 0x4bb3 : 	 4  
aida06 fault 	 0xd2a4 : 	 0xd2a8 : 	 4  
aida07 fault 	 0xc51c : 	 0xc51f : 	 3  
aida08 fault 	 0xe853 : 	 0xe856 : 	 3  
aida09 fault 	 0x5212 : 	 0x5216 : 	 4  
aida10 fault 	 0xf81d : 	 0xf81f : 	 2  
aida11 fault 	 0x810 : 	 0x812 : 	 2  
aida12 fault 	 0xa279 : 	 0xa27a : 	 1  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x5 : 	 5  
aida12 fault 	 0x0 : 	 0xc : 	 12  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     57     33     12     15      3      3      3      2      3      2      6   : 35276
aida02 :     15      3      5      2      1      4      1      3      3      3      6   : 36388
aida03 :     13      4      2      1      4      3      1      3      3      3      6   : 36372
aida04 :     21      5      2      1      2      4      4      2      3      3      6   : 36668
aida05 :     10      5      6      3      2      3      3      3      1      4      6   : 36880
aida06 :     22     12      1      0      3      3      1      3      3      3      6   : 36360
aida07 :     19      3      3      0      3      3      2      4      1      4      6   : 37076
aida08 :     21      6      2      4      2      3      1      3      3      3      6   : 36388
aida09 :      2      7      5      3      2      4      2      3      2      3      6   : 35696
aida10 :      7      3      1      3      1      3      3      3      2      3      6   : 35684
aida11 :      1      2      1      3      2      5      2      2      3      3      6   : 36228
aida12 :     17     10      4      2      3      2      2      4      2      3      6   : 36052



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida02 =>   Options file size is 2114 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

13.30 Output of ~/oh/OptionsCheck.py

02:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&ASIC.se
ttings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

13.37 Detector bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachments 2
      Statistics - good events, ADC data, disc data, info code 4 & 5, pause & resume, correlation - attachments 3-10
      per FEE64 rate spectra - attachments 11 & 12
      per FEE64 1.8.L spectra - attachments 13 & 19
       aida5 peak width 70 ch FWHM, aida06 peak width 89ch FWHM
      per FEE64 1.8.W spectra 20us FSR - attachments 14 & 15
      meger & tape server - attachments 16-18
 
Attachment 1: 4.png
4.png
Attachment 2: Screenshot_2021-04-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-15_Statistics_aidas-gsi.png
Screenshot_2021-04-15_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-15_Statistics_aidas-gsi(1).png
Screenshot_2021-04-15_Statistics_aidas-gsi(1).png
Attachment 5: Screenshot_2021-04-15_Statistics_aidas-gsi(2).png
Screenshot_2021-04-15_Statistics_aidas-gsi(2).png
Attachment 6: Screenshot_2021-04-15_Statistics_aidas-gsi(3).png
Screenshot_2021-04-15_Statistics_aidas-gsi(3).png
Attachment 7: Screenshot_2021-04-15_Statistics_aidas-gsi(4).png
Screenshot_2021-04-15_Statistics_aidas-gsi(4).png
Attachment 8: Screenshot_2021-04-15_Statistics_aidas-gsi(5).png
Screenshot_2021-04-15_Statistics_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-15_Statistics_aidas-gsi(6).png
Screenshot_2021-04-15_Statistics_aidas-gsi(6).png
Attachment 10: Screenshot_2021-04-15_Statistics_aidas-gsi(7).png
Screenshot_2021-04-15_Statistics_aidas-gsi(7).png
Attachment 11: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 13: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(2).png
Attachment 14: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(3).png
Attachment 15: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(4).png
Attachment 16: Screenshot_2021-04-15_NewMerger_Control_aidas-gsi.png
Screenshot_2021-04-15_NewMerger_Control_aidas-gsi.png
Attachment 17: Screenshot_2021-04-15_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-15_Tape_Service_(Expert)_aidas-gsi.png
Attachment 18: 5.png
5.png
Attachment 19: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(5).png
  144   Thu Mar 12 08:04:24 2020 OH, CA, TDThursday 12th March 08:00-16:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

09:00 Bias and leakage currents ok - attachment 1
      Statistics ok - attachment 2
      FEE temperatures ok - attachment 3
      System wide checks all ok

09:08 No beam, not sure exactly how long it has been down.
      FRS team are working on it

09:28 still no beam. issues with SIS. 
      DAQ set to NoStorage mode
      most recent file R7_522

10.27 beam is back
      writing to file R9_0

10.43 attachment 4 - hit rate spectra

11:09 all system wide checks ok
      FEE64 temps ok - attachment 5
      good event statistics ok - attachment 6
      detector bias and leakage currents - attachment 7

11.18 merger ok, ~ 2.7 million data items per second
      TapeServer ok, ~ 25 Mb/s data rate
      Data forwarding to MBS ok

11.20 writing to file R9_39

11.22 beam off

11.24 beam back online (R9_42)

11.41 FRS has reverted to an older setting

13:04 hit rate spectra - attachment 8

13:19 all system wide checks ok

13:30 FEE64 temps ok - attachment 9

      good event statistics ok - attachment 10

      detector bias and leakage currents ok - attachment 11


15:31 good event statistics ok - attachment 13

      detector bias and leakage currents ok - attachment 14

      FEE64 temps ok - attachment 15

      all system wide checks ok

      beam seems to have stopped during 2pm meeting

15.35 TapeServer ok, data rate ~ 23 Mb/s

      Merger ok, rate of 2.5 - 3 million data items per second

      data forwarding to MBS ok

15.38 AIDA writing to file R9_233

16:02 beam back - AIDA writing to file R9_250







      
Attachment 1: 200312_0902_bias.png
200312_0902_bias.png
Attachment 2: 200312_0902_stats.png
200312_0902_stats.png
Attachment 3: 200312_0903_temp.png
200312_0903_temp.png
Attachment 4: 200312_1042_rates.png
200312_1042_rates.png
Attachment 5: 200312_1105_temp.png
200312_1105_temp.png
Attachment 6: 200312_1107_stats.png
200312_1107_stats.png
Attachment 7: 200312_1108_stats.png
200312_1108_stats.png
Attachment 8: 200312_1302_rates.png
200312_1302_rates.png
Attachment 9: 200312_1321_temp.png
200312_1321_temp.png
Attachment 10: 200312_1325_stats.png
200312_1325_stats.png
Attachment 11: 200312_1327_bias.png
200312_1327_bias.png
Attachment 12: 200312_1528_stats.png
200312_1528_stats.png
Attachment 13: 200312_1529_bias.png
200312_1529_bias.png
Attachment 14: 300312_1527_temp.png
300312_1527_temp.png
  143   Wed Mar 11 23:44:27 2020 OHThursday 12th March 00:00-08:000
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00:44 Implantation range in AIDA - attachment 1

01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
      During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
      the AIDA boxes and begin moving files onto that

01:35 System wide checks all ok
      Bias and leakage current ok - attachment 2
      Stats all ok - attachment 3
      FEE temperatures ok - attachment 4

      TapeServer rate around 25-29MB per second
      Merger rate around 2.6-3.2 million items per second

04:11 System wide checks all ok
      Bias and leakage currents ok - attachment 5
      Stats all ok - attachment 6
      FEE temp ok - attachment 7
06:38 System wide checks all ok
      Bias and leakage currents ok - attachment 8
      Stats all ok - attachment 9
      FEE temp ok - attachment 10
Attachment 1: 200311_ImpDis.png
200311_ImpDis.png
Attachment 2: 200312_0135_Bias.png
200312_0135_Bias.png
Attachment 3: 200312_0136_Stats.png
200312_0136_Stats.png
Attachment 4: 200312_0139_FEETemp.png
200312_0139_FEETemp.png
Attachment 5: 200312_0405_BIAS.png
200312_0405_BIAS.png
Attachment 6: 200312_0405_stats.png
200312_0405_stats.png
Attachment 7: 200312_0406_Temp.png
200312_0406_Temp.png
Attachment 8: 200312_0638_bias.png
200312_0638_bias.png
Attachment 9: 200312_0639_stats.png
200312_0639_stats.png
Attachment 10: 200312_0639_temp.png
200312_0639_temp.png
  431   Thu May 12 07:09:43 2022 TDThursday 12 May 08:00-24:00
08.10 AIDA DAQ status
    
      14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
      AD9222 OFF - waveforms disabed
      disc - all disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23

      BNC PB-5
      amplitude 1.000V
      attenuation x1
      tau_d 1ms (tail pulse)
      polarity +
      frequency 22Hz

08.14 all histograms zero'd
2
08.15 system wide check counter baseline

08.17 check ASIC control - all FEE64s all ASICs

Attachment 1 - grafana DSSSD bias & leakage current - OK

Attachment 2 - per FEE64 stat spectra

Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips

Attachments 5 & 6 - per FEE64 1.8.L spectra
 pulser peak width aida01 124ch FWHM, aida04 293ch FWHM

Attachment 7 - per FEE64 rate spectra

Attachments 8-12 - system wide checks - all OK

Attachments 13-16 - statistics - adc, pause, resume & correlation data

Attachment 17 - FEE64 temps OK

Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends

Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages

08.44 DAQ continues - file NOTAPE/R18 - no storage mode

08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc

08.53 no storage mode enabled

Attachment 21 - analysis of file NOTAPE/R18_20

Attachment 22- analysis of file NOTAPE/R19_19
               AIDA12 slow comp up to 0xd improved deadtime drastically for FEE

10:54 Beam on detector. Take another run to test deadtime - attachment 23
      Deadtimes ranging from 10-40% in n+n FEEs

Attachment 23 & 24 aida06 aida02 1.*.H spectra


14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
        Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime)
Attachment 1: Screenshot_from_2022-05-12_08-25-44.png
Screenshot_from_2022-05-12_08-25-44.png
Attachment 2: Screenshot_from_2022-05-12_08-25-20.png
Screenshot_from_2022-05-12_08-25-20.png
Attachment 3: Screenshot_from_2022-05-12_08-24-01.png
Screenshot_from_2022-05-12_08-24-01.png
Attachment 4: Screenshot_from_2022-05-12_08-23-35.png
Screenshot_from_2022-05-12_08-23-35.png
Attachment 5: Screenshot_from_2022-05-12_08-22-24.png
Screenshot_from_2022-05-12_08-22-24.png
Attachment 6: Screenshot_from_2022-05-12_08-21-18.png
Screenshot_from_2022-05-12_08-21-18.png
Attachment 7: Screenshot_from_2022-05-12_08-19-15.png
Screenshot_from_2022-05-12_08-19-15.png
Attachment 8: Screenshot_from_2022-05-12_08-16-40.png
Screenshot_from_2022-05-12_08-16-40.png
Attachment 9: Screenshot_from_2022-05-12_08-16-30.png
Screenshot_from_2022-05-12_08-16-30.png
Attachment 10: Screenshot_from_2022-05-12_08-16-25.png
Screenshot_from_2022-05-12_08-16-25.png
Attachment 11: Screenshot_from_2022-05-12_08-16-17.png
Screenshot_from_2022-05-12_08-16-17.png
Attachment 12: Screenshot_from_2022-05-12_08-16-08.png
Screenshot_from_2022-05-12_08-16-08.png
Attachment 13: Screenshot_from_2022-05-12_08-10-02.png
Screenshot_from_2022-05-12_08-10-02.png
Attachment 14: Screenshot_from_2022-05-12_08-09-44.png
Screenshot_from_2022-05-12_08-09-44.png
Attachment 15: Screenshot_from_2022-05-12_08-09-20.png
Screenshot_from_2022-05-12_08-09-20.png
Attachment 16: Screenshot_from_2022-05-12_08-09-02.png
Screenshot_from_2022-05-12_08-09-02.png
Attachment 17: Screenshot_from_2022-05-12_08-08-39.png
Screenshot_from_2022-05-12_08-08-39.png
Attachment 18: Screenshot_from_2022-05-12_08-08-26.png
Screenshot_from_2022-05-12_08-08-26.png
Attachment 19: Screenshot_from_2022-05-12_08-39-14.png
Screenshot_from_2022-05-12_08-39-14.png
Attachment 20: Screenshot_from_2022-05-12_08-39-05.png
Screenshot_from_2022-05-12_08-39-05.png
Attachment 21: R18_20
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259331519 ( 1693112.6 Hz)
        Other data format:    2588481 (   16899.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1016 (       6.6 Hz)
                               RESUME:       1017 (       6.6 Hz)
                              SYNC100:      32570 (     212.6 Hz)
                              WR48-63:      32570 (     212.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2521308 (   16461.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.169 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.171                0.000
  1                9.417                0.000
  2                0.061                0.000
  3               24.348                0.000
  4                0.008                0.000
  5               72.336                0.000
  6                0.231                0.000
  7               25.570                0.000
  8                0.125                0.000
  9                0.007                0.000
 10                0.000                7.342
 11               41.574                0.000
 12                0.027                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7431923      12514          0          0          8          8        933        933          0      10632          0          0
  1   37380104       9380          0          0         51         51       4639       4639          0          0          0          0
  2    7217279    1365495          0          0          3          3       1056       1056          0    1363377          0          0
  3   38015689    1157497          0          0        179        179       4920       4920          0    1147299          0          0
  4    4440445       1078          0          0          2          2        537        537          0          0          0          0
  5   40380917      10697          0          0        319        320       5029       5029          0          0          0          0
  6   14913071       3788          0          0          5          5       1889       1889          0          0          0          0
  7   41232478      10490          0          0        181        181       5064       5064          0          0          0          0
  8   11952709       2932          0          0          4          4       1462       1462          0          0          0          0
  9    4910914       1250          0          0          1          1        624        624          0          0          0          0
 10    2469929        580          0          0          0          0        290        290          0          0          0          0
 11   39767127      10486          0          0        257        257       4986       4986          0          0          0          0
 12    5557988       1406          0          0          6          6        697        697          0          0          0          0
 13    3660946        888          0          0          0          0        444        444          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.578s (  874.840 blocks/s,  54.677 Mb/s)
Attachment 22: R19_19.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      17139
          ADC data format:  138519076 ( 1563611.9 Hz)
        Other data format:    1763639 (   19908.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        358 (       4.0 Hz)
                               RESUME:        358 (       4.0 Hz)
                              SYNC100:      17469 (     197.2 Hz)
                              WR48-63:      17469 (     197.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1727985 (   19505.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       88.589 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.084                0.000
  1                0.528                0.000
  2                0.008                0.000
  3               10.973                0.000
  4                0.000                0.000
  5               30.421                0.000
  6                0.000                0.000
  7                7.252                0.000
  8                0.055                0.000
  9                0.000                0.000
 10                0.000               12.541
 11                3.332                0.000
 12                0.009                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4390551       7288          0          0          4          4        565        565          0       6150          0          0
  1   21021919       5186          0          0          9          9       2584       2584          0          0          0          0
  2    3174721     918030          0          0          1          1        518        518          0     916992          0          0
  3   22097261     810837          0          0         77         77       2920       2920          0     804843          0          0
  4    2596723        620          0          0          0          0        310        310          0          0          0          0
  5   23543560       6240          0          0        174        174       2946       2946          0          0          0          0
  6    4211356       1020          0          0          0          0        510        510          0          0          0          0
  7   24080430       6296          0          0         63         63       3085       3085          0          0          0          0
  8    4823696       1068          0          0          3          3        531        531          0          0          0          0
  9    1770702        408          0          0          0          0        204        204          0          0          0          0
 10     933213        222          0          0          0          0        111        111          0          0          0          0
 11   21419475       5280          0          0         21         21       2619       2619          0          0          0          0
 12    2929744        698          0          0          6          6        343        343          0          0          0          0
 13    1525725        446          0          0          0          0        223        223          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   17.484s (  980.247 blocks/s,  61.265 Mb/s)
Attachment 23: R20_20.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       4909
          ADC data format:   39765494 ( 1644562.3 Hz)
        Other data format:     414671 (   17149.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        151 (       6.2 Hz)
                               RESUME:        152 (       6.3 Hz)
                              SYNC100:       4999 (     206.7 Hz)
                              WR48-63:       4999 (     206.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     404370 (   16723.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      23252 (     961.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       24.180 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.954                0.000
  2                0.000                0.000
  3                4.569                0.000
  4                0.000                0.000
  5               10.824                0.000
  6                0.000                0.000
  7                5.869                0.000
  8                0.000                0.000
  9                0.008                0.000
 10                0.000                0.000
 11                2.942                0.000
 12                0.007                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    1157062       1976          0          0          0          0        148        148          0       1680          0        159
  1    5834981       1480          0          0         12         12        728        728          0          0          0      22412
  2    1125294     220409          0          0          0          0        178        178          0     220053          0         83
  3    5924210     180469          0          0         32         32        741        741          0     178923          0         86
  4     669291        180          0          0          0          0         90         90          0          0          0         10
  5    6069826       1577          0          0         47         48        741        741          0          0          0         20
  6    1749760       2346          0          0          0          0        222        222          0       1902          0         32
  7    6604292       3590          0          0         38         38        851        851          0       1812          0         32
  8    1794890        416          0          0          0          0        208        208          0          0          0         42
  9     736012        164          0          0          1          1         81         81          0          0          0        124
 10     410664         98          0          0          0          0         49         49          0          0          0         15
 11    6315369       1576          0          0         19         19        769        769          0          0          0        153
 12     852659        248          0          0          2          2        122        122          0          0          0         23
 13     521184        142          0          0          0          0         71         71          0          0          0         61
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    5.102s (  962.254 blocks/s,  60.141 Mb/s)
Attachment 24: Screenshot_from_2022-05-12_11-57-44.png
Screenshot_from_2022-05-12_11-57-44.png
Attachment 25: Screenshot_from_2022-05-12_11-56-42.png
Screenshot_from_2022-05-12_11-56-42.png
Attachment 26: Screenshot_from_2022-05-12_22-32-53.png
Screenshot_from_2022-05-12_22-32-53.png
Attachment 27: Screenshot_from_2022-05-12_22-32-22.png
Screenshot_from_2022-05-12_22-32-22.png
  430   Thu May 12 00:37:52 2022 OH, AMThursday 12 May
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
      That is now complete.
      Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc

      R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
      Will lower 2 and 4 to 200 and raise 6 and 8 to 300

      R17 2 good, 4 shakey 6 and 8 poor - attachment 2

      R18 2 and 4 at 220 and 6 and 7 at 350
      2 and 8 low deadtime, 4 and 6 could be better. - attachment 3


      Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
      Statistics at this rate - attachment 4




02:18 Statistics - attachment 7
      Temperatures - attachment 6
      Bias & leakage current - attachment 5

      Clock Status - OK
      White Rabbit - OK
      The other 3 checks currently aren't working




04:11 Statistics - attachment 10
      Temperatures - attachment 9
      Bias & leakage current - attachment 8

      Clock Status - OK
      White Rabbit - OK




06:05 Statistics - attachment 13
      Temperatures - attachment 12
      Bias & leakage current - attachment 11

      Clock Status - OK
      White Rabbit - OK
Attachment 1: R16_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      18197
          ADC data format:  144240941 ( 1037416.0 Hz)
        Other data format:    4701506 (   33814.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        651 (       4.7 Hz)
                               RESUME:        651 (       4.7 Hz)
                              SYNC100:      18715 (     134.6 Hz)
                              WR48-63:      18715 (     134.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4662774 (   33535.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       9906 (      71.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      139.039 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               96.976
  1                1.224                0.000
  2                0.000                2.784
  3                4.445                0.000
  4                0.000              108.587
  5               78.372                0.000
  6                0.000               48.067
  7               67.687                0.000
  8                0.000              100.043
  9                0.000              102.969
 10                0.000              122.226
 11                0.000               21.170
 12                0.000                2.713
 13                0.000              116.657
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     346857       9766          0          0          0          0         47         47          0       9672          0       2994
  1   28465912       7268          0          0         10         10       3624       3624          0          0          0       3890
  2     235746    2352664          0          0          0          0        299        299          0    2352066          0        528
  3   33196911    2283279          0          0         20         20       4471       4471          0    2274297          0        644
  4     346134         72          0          0          0          0         36         36          0          0          0          4
  5   37750759       9970          0          0        309        309       4676       4676          0          0          0        836
  6     855292      17907          0          0          0          0        114        114          0      17679          0        522
  7   38520306      19374          0          0        312        312       4845       4845          0       9060          0        313
  8     312768         68          0          0          0          0         34         34          0          0          0         23
  9     342945         90          0          0          0          0         45         45          0          0          0         14
 10     176833         46          0          0          0          0         23         23          0          0          0          7
 11    1559416        390          0          0          0          0        195        195          0          0          0         62
 12    1848830        538          0          0          0          0        269        269          0          0          0         19
 13     282232         74          0          0          0          0         37         37          0          0          0         50
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   18.576s (  979.614 blocks/s,  61.226 Mb/s)
Attachment 2: R17_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9844
          ADC data format:   78473161 ( 1119546.3 Hz)
        Other data format:    2099979 (   29959.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        470 (       6.7 Hz)
                               RESUME:        470 (       6.7 Hz)
                              SYNC100:      10105 (     144.2 Hz)
                              WR48-63:      10105 (     144.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2078829 (   29657.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       70.094 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               61.805
  1                4.886                0.000
  2                0.000                0.000
  3               17.776                0.000
  4                0.000               55.325
  5               34.711                0.000
  6                0.000               20.226
  7               27.353                0.000
  8                0.000               57.388
  9                0.000               53.302
 10                0.000               59.261
 11                0.000                7.690
 12                0.000               11.252
 13                0.000               58.364
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     137192       4907          0          0          0          0         16         16          0       4875          0          0
  1   18540829       4668          0          0         40         40       2294       2294          0          0          0          0
  2     107308    1183296          0          0          0          0        177        177          0    1182942          0          0
  3   18083643     896022          0          0        126        126       2379       2379          0     891012          0          0
  4     171162         40          0          0          0          0         20         20          0          0          0          0
  5   19343787       5154          0          0        155        155       2422       2422          0          0          0          0
  6     395908        126          0          0          0          0         63         63          0          0          0          0
  7   19401210       5206          0          0        149        149       2454       2454          0          0          0          0
  8     159836         40          0          0          0          0         20         20          0          0          0          0
  9     172050         38          0          0          0          0         19         19          0          0          0          0
 10      94357         20          0          0          0          0         10         10          0          0          0          0
 11     792883        200          0          0          0          0        100        100          0          0          0          0
 12     933311        234          0          0          0          0        117        117          0          0          0          0
 13     139685         28          0          0          0          0         14         14          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.115s (  973.186 blocks/s,  60.824 Mb/s)
Attachment 3: R18_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9700
          ADC data format:   70825654 ( 1030660.0 Hz)
        Other data format:    8568846 (  124694.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        268 (       3.9 Hz)
                               RESUME:        268 (       3.9 Hz)
                              SYNC100:       9956 (     144.9 Hz)
                              WR48-63:       9956 (     144.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8548398 (  124396.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       68.719 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.039
  1                0.526                0.000
  2                0.044                0.000
  3               16.252                0.000
  4                0.000               41.343
  5               16.542                0.000
  6                0.000               37.115
  7                4.958                0.000
  8                0.000               62.160
  9                0.000               48.394
 10                0.000               57.328
 11                0.000               39.469
 12                0.000                0.000
 13                0.000               54.952
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     123341       4797          0          0          0          0          9          9          0       4779          0          0
  1   16458296       4022          0          0          5          5       2006       2006          0          0          0          0
  2      67098    4844172          0          0          2          2        649        649          0    4842870          0          2
  3   14877427    3705657          0          0        108        108       2346       2346          0    3700749          0          0
  4     201730         54          0          0          0          0         27         27          0          0          0          0
  5   18757776       4968          0          0        118        118       2366       2366          0          0          0          0
  6     275379         62          0          0          0          0         31         31          0          0          0          0
  7   18591077       4758          0          0         35         35       2344       2344          0          0          0          0
  8      87837         12          0          0          0          0          6          6          0          0          0          0
  9     156274         28          0          0          0          0         14         14          0          0          0          0
 10      61114         18          0          0          0          0          9          9          0          0          0          0
 11     292464         64          0          0          0          0         32         32          0          0          0          0
 12     736144        204          0          0          0          0        102        102          0          0          0          0
 13     139697         30          0          0          0          0         15         15          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.125s (  958.025 blocks/s,  59.877 Mb/s)
Attachment 4: 220512_0155_stats.png
220512_0155_stats.png
Attachment 5: 220512_0217_biasleakage.png
220512_0217_biasleakage.png
Attachment 6: 220512_0215_temps.png
220512_0215_temps.png
Attachment 7: 220512_0214_stats.png
220512_0214_stats.png
Attachment 8: 220512_0410_biasLeak.png
220512_0410_biasLeak.png
Attachment 9: 220512_0410_temps.png
220512_0410_temps.png
Attachment 10: 220512_0408_stats.png
220512_0408_stats.png
Attachment 11: 220512_0604_biasLeakage.png
220512_0604_biasLeakage.png
Attachment 12: 220512_0603_temps.png
220512_0603_temps.png
Attachment 13: 220512_0601_stats.png
220512_0601_stats.png
  703   Thu Jun 12 13:15:25 2025 GB, CC, NK, MP, TDThursday 12 June 2025
EMC tests with NK (Orsay)

Stack configuration (upstream->downstream)

bPlas 
AIDA DSSSD #0
AIDA DSSSD #1
bPlas
3x MSL type BB7(DS)-1000 (1x aida10, 2x Mesytec preamplifiers)

AIDA DSSSD #0 bias -100V leakage current -13.23uA
AIDA DSSSD #1 bias -100V leakage current -9.68uA

BNC PB-5 settings
Amplitude 1.0V
Attenuator x1
Frequency 25Hz
Polarity +
tau_d 1ms
Tail pulse

14.39 per p+n FEE64 1.8.W spectra - 20us FSR - attachment 1

      per p+n FEE64 1.8.L spectra - attachment 2

      Pulser peak widths ch FWHM

      DSSSD #0 (DSSSD - FEE64 cables with heavy duty braid - electrically connected to ground of adaptor PCB and internally connected to ground within snout)

9  1   5
120 201  21
15  3  12
114 180  broad

      DSSSD #1 (DSSSD - FEE64 cables with mesh - currently not connected i.e. floating to adaptor PCB and internally connected to ground within snout)
10 14  13
-   22   374
11  7  16
broad 417 20

       Pulser peak widths c. 20 ch FWHM probably indicate cable from DSSSD *not* connected to FEE64 adaptor PCB

        
Attachment 1: Screenshot_from_2025-06-12_14-39-35.png
Screenshot_from_2025-06-12_14-39-35.png
Attachment 2: Screenshot_from_2025-06-12_14-33-26.png
Screenshot_from_2025-06-12_14-33-26.png
  189   Thu Mar 11 07:09:25 2021 OH, LSThursday 11th of March
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
      N.B. wrong date given to the screenshots by accident (It's early)
      Statistics - attachment 1
      FEE temperatures (AIDA10 unavailable) - attachment 2
      FEE temperature - attachment 3
      Bias and leakage currents ok - attachment 4

08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
      Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
      Restarted but no stats in aida04
      It did not recover from a reset
      Telnet in and did a reboot command

09:10 Finally recovered issue was it was not linking to merger
      There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
      Manually set the ASIC settings for aida12. Rates look as expected after
      Will try to reset once beam is back
      

10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
      Analysis of R30_21 looks fine
      
10:38 System wide checks ok
      Statistics - attachment 5
      Temperature - attachment 6
      Bias and leakage - attachment 7

12.10 (LS)
      System wide checks all okay, no fails
      Statistics (attachment8)
      Rate spectra (attachment9)
      FEE temps (attament10)
      Leakage currents written to sheet (attachment11)
      Merger ~45M items/s
      Tapeserver ~14MB/s

13.45 no beam (around file R30_98)

14.00 System wide checks all okay, no fails
      Statistics (attachment12)
      FEE temps (attament13)
      Leakage currents written to sheet (attachment14)
      Merger ~44M items/s
      Tapeserver ~14MB/s
      Beam still down

14.11 beam back aida file R30_112

14.13 beam back down,  back at 14.16

14.20 no beam

14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
      double by analysing R30_101,102,103 no timewarps
      beam is back but low intensity at the moment

14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)

16.00 System wide checks all okay, no fails
      Statistics (attachment16)
      Rate spectra (attachment17)
      FEE temps (attament18)
      Leakage currents written to sheet (attachment19)
      Merger ~45M items/s
      Tapeserver ~14MB/s

18.00 System wide checks all okay, no fails
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE temps (attament22)
      Leakage currents written to sheet (attachment23)
      Merger ~45M items/s
      Tapeserver ~15MB/s

      bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
      now see error in MBS relay!
      "Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
      It triggered 63 times.
      I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser

19:10 MBS DAQ stopped to allow FRS tests

19:38 MBS DAQ is back

20:15 Aida07 fails FPGA check
			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
   
      All other system wide checks ok
      Statistics - attachment 24 (Again ignore the year in the filename)
      Temperatures - attachment 25
      Bias and leakage currents - 26

21:36 They are resetting the time sorter due to FATIMA issues

22:38 System wide checks error on WR in addition to the previous FPGA error
		 Base 		Current 	Difference
aida05 fault 	 0x5cc9 : 	 0x5cca : 	 1  
aida06 fault 	 0xde76 : 	 0xde77 : 	 1  
aida07 fault 	 0xa566 : 	 0xa567 : 	 1  
aida08 fault 	 0x6ab0 : 	 0x6ab1 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4
      Statistics - attachment 27
      Temperature - attachment 28
      Bias and leakage current - attachment 29

23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain.
Attachment 1: 210310_0805_Stats.png
210310_0805_Stats.png
Attachment 2: 210310_0806_Temp.png
210310_0806_Temp.png
Attachment 3: 210310_0806_Temp2.png
210310_0806_Temp2.png
Attachment 4: 210310_0808_Bias.png
210310_0808_Bias.png
Attachment 5: 210311_1039_Stats.png
210311_1039_Stats.png
Attachment 6: 210311_1039_Temp.png
210311_1039_Temp.png
Attachment 7: 210311_1041_Bias.png
210311_1041_Bias.png
Attachment 8: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 11: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 12: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 13: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 14: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 15: 1450_SpectraRate.PNG
1450_SpectraRate.PNG
Attachment 16: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 17: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 18: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 19: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
Attachment 20: 1800_Statistics.PNG
1800_Statistics.PNG
Attachment 21: 1800_SpectraRate.PNG
1800_SpectraRate.PNG
Attachment 22: 1800_FEETemps.PNG
1800_FEETemps.PNG
Attachment 23: 1800_LeakageCurrents.PNG
1800_LeakageCurrents.PNG
Attachment 24: 200311_2014_Stats.png
200311_2014_Stats.png
Attachment 25: 200311_2014_Temp.png
200311_2014_Temp.png
Attachment 26: 210311_2017_Bias.png
210311_2017_Bias.png
Attachment 27: 210311_2225_sTATS.png
210311_2225_sTATS.png
Attachment 28: 210311_2226_Temp.png
210311_2226_Temp.png
Attachment 29: 210311_2227_Bias.png
210311_2227_Bias.png
  363   Thu Jun 10 11:32:27 2021 TDThursday 10 June
12.26 DAQ continues file S496/R32_428
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     11      6      3      4      4      2      2      2      4      6   : 37540
aida02 :     30      8      7      3      4      4      3      2      2      4      6   : 37768
aida03 :     38      8     13      3      4      3      1      3      3      3      6   : 36744
aida04 :     37      9      7      2      3      4      2      3      3      3      6   : 36940
aida05 :     29      5      9      3      3      3      1      3      2      4      6   : 37580
aida06 :     38      6      2      3      3      3      2      3      2      4      6   : 37768
aida07 :     31      9      2      7      4      2      3      2      2      4      6   : 37572
aida08 :     34      6      3      5      4      4      3      2      2      4      6   : 37768
aida09 :     29      2     12      5      1      3      3      2      2      4      6   : 37540
aida10 :     24      8     15      3      5      3      3      3      1      4      6   : 37296
aida11 :     35      4     15      3      3      4      3      3      1      4      6   : 37308
aida12 :     36      5      3      5      6      4      3      3      1      4      6   : 37384
aida13 :     31     10      4      3      6      4      3      3      1      4      6   : 37356
aida14 :     37      7      1      4      5      4      3      3      1      4      6   : 37276
aida15 :     34     10      6      3      4      1      2      4      2      4      6   : 38168
aida16 :     31      9      3      3      5      4      3      3      1      4      6   : 37268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 202

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.07 analysis of file S496/R32_431 - attachment 9
      max dead time aida04 0.33%
Attachment 1: Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-10_Statistics_aidas-gsi.png
Screenshot_2021-06-10_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 996.png
996.png
Attachment 8: 997.png
997.png
Attachment 9: R32_431
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   54180182 (   19758.0 Hz)
        Other data format:  207739819 (   75756.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        105 (       0.0 Hz)
                               RESUME:        105 (       0.0 Hz)
                              SYNC100:      42213 (      15.4 Hz)
                              WR48-63:      42213 (      15.4 Hz)
                           FEE64 disc:    1515114 (     552.5 Hz)
                             MBS info:  206140069 (   75173.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          6 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2742.196 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2662.854
  2                0.384                0.000
  3                9.009                0.000
  4                0.000                0.000
  5                0.000             1805.119
  6                0.000                0.000
  7                0.000             1797.783
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        559          0          0          0          0          0          0          0          0          0          0          0
  1     413404        177          0          0          0          0         84         84          9          0          0          0
  2       1153  103268358          0          0         19         19      17963      17963          0  103232394          0          0
  3   42054837  104466440          0          0         86         86      21757      21757    1515079  102907675          0          0
  4        515          0          0          0          0          0          0          0          0          0          0          0
  5    5561752       2359          0          0          0          0       1175       1175          9          0          0          0
  6       1738          2          0          0          0          0          1          1          0          0          0          0
  7    6142236       2472          0          0          0          0       1232       1232          8          0          0          0
  8        565          1          0          0          0          0          0          0          1          0          0          2
  9        190          0          0          0          0          0          0          0          0          0          0          0
 10        694          0          0          0          0          0          0          0          0          0          0          0
 11        417          6          0          0          0          0          0          0          6          0          0          4
 12        967          4          0          0          0          0          1          1          2          0          0          0
 13        244          0          0          0          0          0          0          0          0          0          0          0
 14        285          0          0          0          0          0          0          0          0          0          0          0
 15        626          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.160s ( 1097.388 blocks/s,  68.587 Mb/s)
  335   Thu May 20 01:00:30 2021 MuneerahThu May 20 00:00-08:00

02:00 System check

The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!

The rates on ucesb got back to work by around 01:12

Deat Time check ok R14_870, attachment 1

Statistics are ok attachment 2

Temperatures are ok attacment 3

Voltages are ok attachment 4

 

system wide check

All ok except

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White rabbite

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      8      8      5      0      6      1      2      2      2      7   : 37296
aida02 :     19      8     11      2      3      4      2      2      3      3      6   : 36412
aida03 :     14      6      3      4      1      3      1      3      3      3      6   : 36312
aida04 :     25     12     12      4      3      4      1      4      3      3      6   : 37316
aida05 :      4      8      3      4      1      2      2      4      3      3      6   : 36928
aida06 :     42     12      8      7      2      3      3      4      3      3      6   : 37736
aida07 :     24      6      7      3      3      3      1      3      2      2      7   : 37536
aida08 :     10     15      7      2      1      4      2      3      3      3      6   : 36752
aida09 :     29     13      6      6      4      3      3      3      3      3      6   : 37244
aida10 :     18     12      7      2      2      3      3      2      2      4      6   : 37464
aida11 :     20      8      0      5      5      4      2      3      3      3      6   : 36976
aida12 :      7      2      4      3      2      3      2      2      4      3      6   : 37068
aida13 :      2      5      5      4      2      3      2      3      3      3      6   : 36608
aida14 :      9     13      8      2      2      2      3      3      2      4      6   : 37836
aida15 :     24     12     12      1      0      3      2      3      3      3      6   : 36640
aida16 :     12      9      9      2      1      3      2      4      3      3      6   : 37128

6:45 System check

Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9

System wide check are all ok except:

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit 1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20     11      3      5      0      6      1      2      3      2      7   : 38264
aida02 :     14      9      9      2      3      3      3      2      3      3      6   : 36496
aida03 :     13      7      3      5      2      3      2      3      3      3      6   : 36668
aida04 :     30     15     13      4      3      4      1      3      4      3      6   : 37888
aida05 :     17      9      3      3      1      2      2      4      3      3      6   : 36956
aida06 :     40     16      5      6      2      3      2      3      2      4      6   : 37936
aida07 :      3      2      3      2      3      4      2      2      2      2      7   : 37196
aida08 :     17     10      6      2      1      3      3      3      3      3      6   : 36852
aida09 :     29     11      7      6      3      3      2      4      3      3      6   : 37436
aida10 :      1     10      5      1      1      3      3      2      2      4      6   : 37252
aida11 :     30      9      0      5      4      4      2      3      3      3      6   : 36960
aida12 :     19     10      5      3      2      3      2      2      4      3      6   : 37196
aida13 :     22      9      6      3      2      3      3      3      3      3      6   : 36960
aida14 :     19      6      9      2      2      3      3      2      2      4      6   : 37452
aida15 :      6      8     11      2      1      3      2      3      3      3      6   : 36616
aida16 :     19     14      5      3      1      3      1      4      3      3      6   : 36908

 

07:33 system check

Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13

System wide check

all ok except

ADC all failed

White rabbit

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      9      4      6      0      5      1      2      2      2      7   : 37152
aida02 :     19      6     13      1      3      3      4      2      3      3      6   : 36780
aida03 :     23      5      6      4      2      3      1      3      3      3      6   : 36452
aida04 :     24     11     14      3      3      4      2      3      3      3      6   : 37048
aida05 :     17      3      6      3      1      2      2      3      3      3      6   : 36444
aida06 :     36     15      4      9      2      4      2      4      3      3      6   : 37608
aida07 :     20      8      5      3      3      4      2      2      2      2      7   : 37376
aida08 :     21     11     10      2      1      4      2      2      2      4      6   : 37324
aida09 :     27     14      7      6      3      3      2      4      3      3      6   : 37452
aida10 :     25      9      5      2      2      3      3      2      2      4      6   : 37436
aida11 :     30     14      2      4      5      3      2      3      3      3      6   : 36936
aida12 :     22      3      5      3      2      3      2      3      3      3      6   : 36640
aida13 :     10      7      1      4      2      3      3      2      3      3      6   : 36336
aida14 :     13      9     12      2      3      3      2      3      2      4      6   : 37820
aida15 :     20      8     11      3      0      3      2      3      3      3      6   : 36640
aida16 :     23      6      6      2      0      2      1      3      4      3      6   : 37164

 

 

 

 

 

Attachment 1: 210520_0156_DT.png
210520_0156_DT.png
Attachment 2: 210520_0157_Stats.png
210520_0157_Stats.png
Attachment 3: 210520_0157_Temps.png
210520_0157_Temps.png
Attachment 4: 210520_0157_Voltages.png
210520_0157_Voltages.png
Attachment 5: 210520_0537_Temps.png
210520_0537_Temps.png
Attachment 6: 210520_0539_VOltages.png
210520_0539_VOltages.png
Attachment 7: 210520_0540_Histo.png
210520_0540_Histo.png
Attachment 8: 210520_0541_Rates.png
210520_0541_Rates.png
Attachment 9: 210520_0542_DT.png
210520_0542_DT.png
Attachment 10: 210520_0730_DT.png
210520_0730_DT.png
Attachment 11: 210520_0730_Voltages.png
210520_0730_Voltages.png
Attachment 12: 210520_0731_Stats.png
210520_0731_Stats.png
Attachment 13: 210520_0731_Tems.png
210520_0731_Tems.png
  544   Thu Mar 21 15:41:49 2024 NH, AM, MP, CCThu Mar 21
Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference
Attachment 1: Screenshot_from_2024-03-21_16-17-59.png
Screenshot_from_2024-03-21_16-17-59.png
Attachment 2: Screenshot_from_2024-03-21_16-18-25.png
Screenshot_from_2024-03-21_16-18-25.png
Attachment 3: Screenshot_from_2024-03-21_16-18-42.png
Screenshot_from_2024-03-21_16-18-42.png
Attachment 4: Screenshot_from_2024-03-21_16-39-12.png
Screenshot_from_2024-03-21_16-39-12.png
Attachment 5: Screenshot_from_2024-03-21_16-39-26.png
Screenshot_from_2024-03-21_16-39-26.png
Attachment 6: Screenshot_from_2024-03-21_16-39-39.png
Screenshot_from_2024-03-21_16-39-39.png
  546   Fri Mar 22 08:22:43 2024 NH, AM, MP, CCThu Mar 21
> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.
  128   Thu Feb 20 16:35:55 2020 TD, NHThu 20 February 2020
DSSD#3 (most downstream) was replaced

Noted that old DSSD wafer was not sitting properly - detached - in PCB, PCB flexing and/or inadequate epoxy gluing

New DSSD#3 - 2998-22 - 1006 um - 100 V - ~4 uA

Old Detector removed was 3208-14

Fig 1. HV Control

Fig 2. Temps

Fig 3. Stats

Fig 4. Rates

Fig 5-6: Waves

Fig 7-8: Pulser Peaks

Start alpha run at 17:48, 20FEB20, R1
 Slow comparator 0x64
 Pulser off

Fig 9 : Alpha stats

Fig 10: Merger stats

Fig 11: Merger main page

Fig 12: Timestamp errors in merger console
Attachment 1: hv.png
hv.png
Attachment 2: temps.png
temps.png
Attachment 3: stats.png
stats.png
Attachment 4: rates.png
rates.png
Attachment 5: waves_even.png
waves_even.png
Attachment 6: waves_odd.png
waves_odd.png
Attachment 7: pulser_even.png
pulser_even.png
Attachment 8: pulser_odd.png
pulser_odd.png
Attachment 9: alpha_stats.png
alpha_stats.png
Attachment 10: merger.png
merger.png
Attachment 11: merger_stats.png
merger_stats.png
Attachment 12: merger_errors.png
merger_errors.png
  312   Thu May 13 17:05:42 2021 Muneerah, MArcThu 13 May 04:00-00:00

19:08 System Check

Rates from ucesb below 1 kHz

Atachment 1 Stats

Atachment 2 Temperatures

Atachment 3 Voltages

System wide checks are all ok except

ADC calibration all failed  
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White rabbit  
         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dba :      2  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA all passed

Memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6      2      2      1      2      4      3      4      2      3      7   : 40424
aida02 :      3      4      2      2      1      2      1      3      3      3      7   : 40140
aida03 :     10     10      0      3      2      4      2      4      2      3      7   : 40280
aida04 :      3      2      0      2      2      3      2      3      1      4      7   : 40540
aida05 :      4      9      2      3      2      4      3      3      1      4      7   : 41048
aida06 :      8      6      1      1      2      3      2      2      3      3      7   : 40064
aida07 :      1      4      0      1      1      3      1      3      3      3      7   : 40196
aida08 :      1      4      2      2      2      4      3      4      2      3      7   : 40452
aida09 :      8      5      2      2      1      2      1      3      3      3      7   : 40168
aida10 :     22      5      4      2      1      3      1      4      2      3      7   : 39872
aida11 :      2      1      2      1      2      2      3      3      2      3      7   : 39632
aida12 :      8      5      2      2      1      3      1      4      2      3      7   : 39784
aida13 :     18      8      2      1      0      2      3      3      2      3      7   : 39624
aida14 :     23     12      2      2      1      2      2      4      2      3      7   : 40028
aida15 :      3      5      3      2      2      3      1      4      2      3      7   : 39844
aida16 :      0      2      1      2      1      4      2      4      2      3      7   : 40096

 

 

 

Attachment 1: 130521_1802_Stats.png
130521_1802_Stats.png
Attachment 2: 130521_1802_Temp.png
130521_1802_Temp.png
Attachment 3: 130521_1804_Voltages.png
130521_1804_Voltages.png
  310   Thu May 13 00:51:40 2021 Muneerah and OHThu 13 May 00:00-08:00
01:50 Beam on
The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing.
Tempetratures OK

Rates OK

Attachment 1 Voltages

Atachment 2 Rates

Atachment 3 Temperatures

Attachment 4 ucesb rates less than 1 kHz


System wide check are ok except:

Base Current Difference
aida05 fault  0x36ca :  0x36ce :  4 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base Current Difference
aida13 fault 0xa :  0xe9 :  223 
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760
aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644
aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188
aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404
aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104
aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128
aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500
aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672
aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588
aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460
aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348
aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000
aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840
aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996
aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280
aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716



<strong>05:00 OH Took over at 04:00</strong>

System wide checks ok except

   Base   Current  Difference
aida05 fault   0x36ca :   0x36d0 :   6 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


    Base   Current   Difference
aida05 fault   0x0 :   0x1 :   1 
aida13 fault   0xa :   0xe9 :   223 
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



05:41 AIDA 05 has dropped out. Will restart the system

05:58 DAQ recovered from powercycle
 p+n strips at 0xa
 n+n strips at 0x20
 ADC control register at 0xFF
 All waveforms turned off in LED

 Statistics - attachment 5
 Correlation scaler in FEE 3 and 4 are mostly tracking 1:1

06:34 Local users have been unable to recover MBS
      And have been unable to contact local experts

07:22 Bias and leakage currents ok - attachment 6
 Stats attachment -7
Attachment 1: 130521_0155_Voltages.png
130521_0155_Voltages.png
Attachment 2: 130521_0157_Stats.png
130521_0157_Stats.png
Attachment 3: 130521_0158_Temp.png
130521_0158_Temp.png
Attachment 4: 130521_0204_ucesb.png
130521_0204_ucesb.png
Attachment 5: 210513_0600_Stats.png
210513_0600_Stats.png
Attachment 6: 210513_0722_bias.png
210513_0722_bias.png
Attachment 7: 210513_0722_Stats.png
210513_0722_Stats.png
  674   Mon Dec 9 12:28:01 2024 NH, CC, MPTest of BB7 on AIDA Electronics
BB7 connected to aida07, sitting in a cardboard box

Update Tcl/Tml files in ASIC, ASIC4 and sys.tml frm PJCS email to allow setting per-ASIC polarity on FEE64
Restart MIDAS@8015

Power on all FEEs

Note nnrpi is not responding to ssh... MP investigates if it ahs been unplugged
 No, but a powercycle worked and fixed it

See error: /MIDAS/FEE_ASIC/aida06/ASICs.txt  not found, create the folders

We see some data in the L spectra, and changing the settings from one ASIC seems to work properly

Stopped for CC/MP to determine the which ASICs correspond to p+n and n+n

Follow up: Alpha data overnight or source?
  302   Sun May 9 19:29:23 2021 TDSunday 9 May
Attachment 1: Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-09_Statistics_aidas-gsi.png
Screenshot_2021-05-09_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-09_Statistics_aidas-gsi(1).png
Screenshot_2021-05-09_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-09_Statistics_aidas-gsi(2).png
Screenshot_2021-05-09_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-09_Statistics_aidas-gsi(3).png
Screenshot_2021-05-09_Statistics_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png
Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png
Attachment 7: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 10: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png
Attachment 11: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png
Attachment 12: Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png
Attachment 13: 503.png
503.png
Attachment 14: 504.png
504.png
  643   Sat Jun 8 23:00:52 2024 TDSunday 9 June
23.58 8.6.24

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

03.43 slow comparator 0x64

      DSSSD bias & leakage current OK - attachments 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 1.8.H spectra - attachments 8-9

      per FEE64 Rate spectra - attachment 10


09.11 DSSSD bias & leakage current OK - attachments 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats - attachment 13

      per FEE64 Rate spectra - attachment 14


15.05 online spectra
      per DSSSD HEC E_p versus E_n and x versus y
      observe flat field of c. 500MeV at per spull rate c. 5Hz - fission fragments?
Attachment 1: Screenshot_from_2024-06-08_23-58-55.png
Screenshot_from_2024-06-08_23-58-55.png
Attachment 2: Screenshot_from_2024-06-08_23-59-16.png
Screenshot_from_2024-06-08_23-59-16.png
Attachment 3: Screenshot_from_2024-06-09_00-00-02.png
Screenshot_from_2024-06-09_00-00-02.png
Attachment 4: Screenshot_from_2024-06-09_00-00-32.png
Screenshot_from_2024-06-09_00-00-32.png
Attachment 5: Screenshot_from_2024-06-09_03-43-32.png
Screenshot_from_2024-06-09_03-43-32.png
Attachment 6: Screenshot_from_2024-06-09_03-44-19.png
Screenshot_from_2024-06-09_03-44-19.png
Attachment 7: Screenshot_from_2024-06-09_03-46-03.png
Screenshot_from_2024-06-09_03-46-03.png
Attachment 8: Screenshot_from_2024-06-09_03-46-37.png
Screenshot_from_2024-06-09_03-46-37.png
Attachment 9: Screenshot_from_2024-06-09_03-47-40.png
Screenshot_from_2024-06-09_03-47-40.png
Attachment 10: Screenshot_from_2024-06-09_03-49-06.png
Screenshot_from_2024-06-09_03-49-06.png
Attachment 11: Screenshot_from_2024-06-09_11-15-26.png
Screenshot_from_2024-06-09_11-15-26.png
Attachment 12: Screenshot_from_2024-06-09_11-27-22.png
Screenshot_from_2024-06-09_11-27-22.png
Attachment 13: Screenshot_from_2024-06-09_11-27-52.png
Screenshot_from_2024-06-09_11-27-52.png
Attachment 14: Screenshot_from_2024-06-09_11-28-19.png
Screenshot_from_2024-06-09_11-28-19.png
Attachment 15: Capture_2.PNG
Capture_2.PNG
Attachment 16: Capture.PNG
Capture.PNG
  424   Sun May 8 15:10:45 2022 TDSunday 8 May
DSSSD bias and FEE64 power OFF during Ge install



Photographs of AIDA (triple) + DEGAS setup as of Sunday pm.

Attachments 1-4 side/front/oblique views - note AIDA snout droops and twists

Attachments 5 & 6 - position of the AIDA stand vertical adjustment threads (brass) 

The asymmetry upstream-downstream pre-dates this setup. To lift the end of the snout clear of the of lower DEGAS detectors all 3x 1/4 turns clockwise (viewed from above) were applied to each of the four threads.
Attachment 1: 20220508_160103.jpg
20220508_160103.jpg
Attachment 2: 20220508_160115.jpg
20220508_160115.jpg
Attachment 3: 20220508_160128.jpg
20220508_160128.jpg
Attachment 4: 20220508_160140.jpg
20220508_160140.jpg
Attachment 5: 20220508_160720.jpg
20220508_160720.jpg
Attachment 6: 20220508_160804.jpg
20220508_160804.jpg
ELOG V3.1.4-unknown