ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Mon Apr 19 23:01:02 2021

Author*:

Subject*:

23:52 Noticed AIDA07 has dropped out of the merger and is producing no statistics
      Powercycling to recover

00:14 All FEEs recovered from powercycle
      aida09 running again at HEC threshold of 0x2 (Was set to 0x1 earlier in the day while testing the HEC event disparity in DSSD3)
      Now writing to R34
      Following the powercycle the rates in the FEEs are much improved and back to where they were before the 08:30 crash on the 19th. - Attachment 1
      Waveforms also look improved - attachmens 2 and 3

Current system parameters
      ASIC settings Dec19 (All LEC comparator 0xa)
      Pulser at 2V and 2Hz
      DSSD 1 and 2 Bias 160V
      DSSD 3 at 120V (Will lower to 100 again during the beam off period tomorrow)

00:28 In analysis of R34_13 the number of HEC items in DSSD3 appears to make sense again. The number in aida09 are consistent with the number in aida05 and the same for aida11 and aida07.
      This appears to confirm that the earlier HEC issues we were observing were hardware related but we are unsure what/why/how these issues were caused.
      Analysis - attachment 4

00:40 Issue with one of the FRS magnets so they stop recording data. Leaving AIDA running.

01:08 While beam was down have reset DSSD3 bias to 100V as was agreed with Tom and Nic in email
      Leakage current decreased as expected - attachment 5
      Once beam is back will check HEC still as expected.

01:14 Last magnet in FRS has an issue with the water flowing. Either sensor issue or water flow is to low.
      Hakke have increased the water flow to hopefully keep the water running.
      Beam is back.

From ucesb it seems that the HEC events in DSSD3 are still appearing as expected. Will check a file once we have a complete file.
      Checked R34_36 and the HEC events in DSSD3 are still consistent - attachment 6

2:20 Statistics ok - attachment 7
     Temp - attachment 8
     Bias and leakage current ok - attachment 9

All system wide checks ok except for WR error check:
	
		 Base 		Current 	Difference
aida07 fault 	 0x9154 : 	 0x915b : 	 7  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 10, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Note aida07 reports two bad timestamps in the merger
     MERGE Data Link (3762): bad timestamp  6 3 0xc1817ea7 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
     MERGE Data Link (3762): bad timestamp  6 3 0xc1b77e25 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56

02:25 Remembered during a pre-experiment meeting with Vic, Patrick and Carl they mentioned the merger would track timestamp sequence errors
      Can confirm this is true. Can see the two in aida07 in the counter in the merger stats page - attachment 10

02:45 Now up to 7 timestamp errors on aida07 in the merger
      
02:49 Ran R34_45 through AIDASort - It reports a greater number of implant events are able to have a front back match performed
      Ratio between aida07 and 09 is 0.71 -> Double what it was before
      Energy response of the HEC channels in aida07 and aida09 looks equivalent (left group aida09 right group aida11) - attahchment 11

04:40 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents - attachment 14
      Clock status, ADC calibration and memory infomration all ok
      aida07 fails both WR and FPGA
	
		 Base 		Current 	Difference
aida07 fault 	 0x915b : 	 0x9161 : 	 6  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 10, Failed 2

Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

TS errors in the merger still only at 7 with all in aida07

06:25 Statistics - attachment 15
      Temps ok - attachment 16
      Bias and leakage currents ok - attachment 17

System wide checks: clock status, adc calibration and memory information all ok
      WR multiple fails
	
		 Base 		Current 	Difference
aida05 fault 	 0xc9ca : 	 0xc9cb : 	 1  
aida06 fault 	 0xe900 : 	 0xe901 : 	 1  
aida07 fault 	 0x915b : 	 0x9166 : 	 11  
aida08 fault 	 0x641e : 	 0x641f : 	 1  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 7, Failed 5

FPGA aida07 fails with difference of 1

No new TS sequence errors in merger

07:10 Beam has dropped out.
07:14 Beam is back

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1:

210420_0016_Stats.png Original size: 1907x478

Attachment 2:

210420_0021_Layout7.png Original size: 1908x877

Attachment 3:

210420_0022_Layout8.png Original size: 1913x859

Attachment 4:

210420_0026_R34_13_Analysis.txt

 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255975543 ( 1356743.5 Hz)
        Other data format:    5944457 (   31507.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32703 (     173.3 Hz)
                              WR48-63:      32703 (     173.3 Hz)
                           FEE64 disc:    1073243 (    5688.5 Hz)
                             MBS info:    4805808 (   25472.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     345437 (    1830.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      188.669 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               46.534
  2                0.000               63.789
  3                0.000               54.847
  4                0.000              174.519
  5                0.000              123.945
  6                0.000              113.898
  7                0.000              135.923
  8                0.000               59.774
  9                0.000               27.296
 10                0.000               12.263
 11                0.000               65.598
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14400003     100475          0          0          0          0       1767       1767      95813       1128          0      35602
  1   23251868      63483          0          0          0          0       2988       2988      57507          0          0      21204
  2   17192461    2025458          0          0          0          0       2415       2415     200396    1820232          0      34093
  3   20019932    1908259          0          0          0          0       2815       2815      81941    1820688          0      32715
  4    3059738     405674          0          0          0          0        403        403     100680     304188          0      35774
  5   10595698     355900          0          0          0          0       1322       1322      58122     295134          0      21319
  6   12037952     388178          0          0          0          0       1554       1554     105230     279840          0      34521
  7    9390918     368608          0          0          0          0       1205       1205      81600     284598          0      31165
  8   46854628      72459          0          0          0          0       5872       5872      60715          0          0      21758
  9   43874737      60599          0          0          0          0       5426       5426      49747          0          0      17382
 10   36184433     112868          0          0          0          0       4528       4528     103812          0          0      31745
 11   19113175      82496          0          0          0          0       2408       2408      77680          0          0      28159
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 1641.051s (   19.500 blocks/s,   1.219 Mb/s)

Attachment 5:

210420_0109_Bias.png Original size: 496x333

Attachment 6:

210420_0026_R34_36_Analysis.txt

 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255999603 ( 1309621.8 Hz)
        Other data format:    5920397 (   30287.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32728 (     167.4 Hz)
                              WR48-63:      32728 (     167.4 Hz)
                           FEE64 disc:    1044504 (    5343.4 Hz)
                             MBS info:    4810437 (   24608.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     327887 (    1677.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      195.476 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               51.325
  2                0.000               68.599
  3                0.000               55.115
  4                0.000              180.174
  5                0.000              140.295
  6                0.000              123.427
  7                0.000              138.446
  8                0.000               55.510
  9                0.000               13.463
 10                0.000               24.357
 11                0.000               74.336
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14611253      94841          0          0          0          0       1936       1936      89802       1167          0      33511
  1   23423897      61346          0          0          0          0       3006       3006      55334          0          0      20222
  2   17311487    2050104          0          0          0          0       2477       2477     195362    1849788          0      32021
  3   20533060    1944396          0          0          0          0       2862       2862      77493    1861179          0      30741
  4    2942569     381933          0          0          0          0        417        417      93930     287169          0      33667
  5    9776355     338395          0          0          0          0       1239       1239      56110     279807          0      20096
  6   11887130     369619          0          0          0          0       1555       1555     103862     262647          0      32503
  7    9239609     348065          0          0          0          0       1200       1200      76985     268680          0      29470
  8   49507336      72711          0          0          0          0       6129       6129      60453          0          0      21190
  9   44170393      65100          0          0          0          0       5430       5430      54240          0          0      18016
 10   32977762     112783          0          0          0          0       4022       4022     104739          0          0      30280
 11   19618752      81104          0          0          0          0       2455       2455      76194          0          0      26170
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 5995.524s (    5.337 blocks/s,   0.334 Mb/s)

Attachment 7:

210420_0220_Stats.png Original size: 1911x472