ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Thu Mar 11 07:09:25 2021

Author*:

Subject*:

08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
      N.B. wrong date given to the screenshots by accident (It's early)
      Statistics - attachment 1
      FEE temperatures (AIDA10 unavailable) - attachment 2
      FEE temperature - attachment 3
      Bias and leakage currents ok - attachment 4

08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
      Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
      Restarted but no stats in aida04
      It did not recover from a reset
      Telnet in and did a reboot command

09:10 Finally recovered issue was it was not linking to merger
      There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
      Manually set the ASIC settings for aida12. Rates look as expected after
      Will try to reset once beam is back

10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
      Analysis of R30_21 looks fine
      
10:38 System wide checks ok
      Statistics - attachment 5
      Temperature - attachment 6
      Bias and leakage - attachment 7

12.10 (LS)
      System wide checks all okay, no fails
      Statistics (attachment8)
      Rate spectra (attachment9)
      FEE temps (attament10)
      Leakage currents written to sheet (attachment11)
      Merger ~45M items/s
      Tapeserver ~14MB/s

13.45 no beam (around file R30_98)

14.00 System wide checks all okay, no fails
      Statistics (attachment12)
      FEE temps (attament13)
      Leakage currents written to sheet (attachment14)
      Merger ~44M items/s
      Tapeserver ~14MB/s
      Beam still down

14.11 beam back aida file R30_112

14.13 beam back down,  back at 14.16

14.20 no beam

14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
      double by analysing R30_101,102,103 no timewarps
      beam is back but low intensity at the moment

14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)

16.00 System wide checks all okay, no fails
      Statistics (attachment16)
      Rate spectra (attachment17)
      FEE temps (attament18)
      Leakage currents written to sheet (attachment19)
      Merger ~45M items/s
      Tapeserver ~14MB/s

18.00 System wide checks all okay, no fails
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE temps (attament22)
      Leakage currents written to sheet (attachment23)
      Merger ~45M items/s
      Tapeserver ~15MB/s

bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
      now see error in MBS relay!
      "Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
      It triggered 63 times.
      I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser

19:10 MBS DAQ stopped to allow FRS tests

19:38 MBS DAQ is back

20:15 Aida07 fails FPGA check
			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
   
      All other system wide checks ok
      Statistics - attachment 24 (Again ignore the year in the filename)
      Temperatures - attachment 25
      Bias and leakage currents - 26

21:36 They are resetting the time sorter due to FATIMA issues

22:38 System wide checks error on WR in addition to the previous FPGA error
		 Base 		Current 	Difference
aida05 fault 	 0x5cc9 : 	 0x5cca : 	 1  
aida06 fault 	 0xde76 : 	 0xde77 : 	 1  
aida07 fault 	 0xa566 : 	 0xa567 : 	 1  
aida08 fault 	 0x6ab0 : 	 0x6ab1 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4
      Statistics - attachment 27
      Temperature - attachment 28
      Bias and leakage current - attachment 29

23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain.

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1: