AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 11 of 35  ELOG logo
  ID Date Author Subject
  497   Fri Jun 24 15:04:06 2022 MarcFriday 24th June - evening shift

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

 

 

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

 

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

 

 

Attachment 1: Stats-Screenshot_from_2022-06-24_16-53-47.png
Stats-Screenshot_from_2022-06-24_16-53-47.png
Attachment 2: Temp-Screenshot_from_2022-06-24_16-52-58.png
Temp-Screenshot_from_2022-06-24_16-52-58.png
Attachment 3: HV_LC-Screenshot_from_2022-06-24_16-51-39.png
HV_LC-Screenshot_from_2022-06-24_16-51-39.png
Attachment 4: ucesb-Screenshot_from_2022-06-24_16-57-32.png
ucesb-Screenshot_from_2022-06-24_16-57-32.png
Attachment 5: GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
Attachment 6: Stats-Screenshot_from_2022-06-24_19-21-32.png
Stats-Screenshot_from_2022-06-24_19-21-32.png
Attachment 7: Temp-Screenshot_from_2022-06-24_19-20-48.png
Temp-Screenshot_from_2022-06-24_19-20-48.png
Attachment 8: HV-LC-Screenshot_from_2022-06-24_19-19-43.png
HV-LC-Screenshot_from_2022-06-24_19-19-43.png
Attachment 9: Stats-Screenshot_from_2022-06-24_22-01-09.png
Stats-Screenshot_from_2022-06-24_22-01-09.png
Attachment 10: Temp-Screenshot_from_2022-06-24_22-01-50.png
Temp-Screenshot_from_2022-06-24_22-01-50.png
Attachment 11: HV_LC-Screenshot_from_2022-06-24_21-58-33.png
HV_LC-Screenshot_from_2022-06-24_21-58-33.png
  496   Fri Jun 24 06:56:06 2022 OHFriday 24 June 008:00-16:00
07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16
Attachment 1: 220624_0753_stats.png
220624_0753_stats.png
Attachment 2: 220624_0754_Temp.png
220624_0754_Temp.png
Attachment 3: 220624_0755_Bias.png
220624_0755_Bias.png
Attachment 4: R3_359_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260715255 (  897318.9 Hz)
        Other data format:    1204745 (    4146.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1323 (       4.6 Hz)
                               RESUME:       1323 (       4.6 Hz)
                              SYNC100:      33082 (     113.9 Hz)
                              WR48-63:      33082 (     113.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1135935 (    3909.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     353220 (    1215.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.549 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.171                0.000
  1               46.409                0.000
  2                1.006                0.000
  3                4.601                0.000
  4                0.000               37.574
  5                0.031                0.000
  6                0.191                0.000
  7                0.254                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903987      28078          0          0        236        236       4023       4023          0      19560          0      86851
  1   61208192      17252          0          0        718        718       7908       7908          0          0          0      52897
  2   26785987     291397          0          0         72         72       3425       3425          0     284403          0      56573
  3   43119726     289440          0          0        153        153       5319       5319          0     278496          0      29859
  4    4604412       1194          0          0          0          0        597        597          0          0          0      28791
  5   18336996       4656          0          0         16         16       2312       2312          0          0          0      34152
  6   36059875     277258          0          0         42         42       4514       4514          0     268146          0      48935
  7   39696080     295470          0          0         86         86       4984       4984          0     285330          0      15162
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.877s (  917.511 blocks/s,  57.344 Mb/s)
Attachment 5: 220624_0842_Stats_Stop.png
220624_0842_Stats_Stop.png
Attachment 6: 220624_1111_Stats.png
220624_1111_Stats.png
Attachment 7: 220624_1111_Temp.png
220624_1111_Temp.png
Attachment 8: 220624_1112_Bias.png
220624_1112_Bias.png
Attachment 9: 220624_1303_stats.png
220624_1303_stats.png
Attachment 10: 220624_1303_Temp.png
220624_1303_Temp.png
Attachment 11: 220624_1304_Bias.png
220624_1304_Bias.png
Attachment 12: R5_33_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260717099 (  869523.1 Hz)
        Other data format:    1202901 (    4011.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1344 (       4.5 Hz)
                               RESUME:       1344 (       4.5 Hz)
                              SYNC100:      33117 (     110.4 Hz)
                              WR48-63:      33117 (     110.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1133979 (    3782.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354749 (    1183.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      299.839 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.504                0.000
  1               42.102                0.000
  2                0.275                0.000
  3                6.677                0.000
  4                0.000               31.976
  5                0.051                0.000
  6                0.250                0.000
  7                0.412                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31758241      28867          0          0        238        238       4156       4156          0      20079          0      87480
  1   58453105      16104          0          0        635        635       7417       7417          0          0          0      53128
  2   20104679     289945          0          0         41         41       2565       2565          0     284733          0      56849
  3   41709424     287290          0          0        188        188       5262       5262          0     276390          0      29572
  4    5133297       1264          0          0          0          0        632        632          0          0          0      29278
  5   17857346       4470          0          0         22         22       2213       2213          0          0          0      34539
  6   40732502     278221          0          0         59         59       5175       5175          0     267753          0      48740
  7   44968505     296740          0          0        161        161       5697       5697          0     285024          0      15163
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.742s (  921.070 blocks/s,  57.567 Mb/s)
Attachment 13: 220624_1538_Stats.png
220624_1538_Stats.png
Attachment 14: 220624_1538_Temp.png
220624_1538_Temp.png
Attachment 15: 220624_1538_Bias.png
220624_1538_Bias.png
Attachment 16: R5_63_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260796374 (  834020.0 Hz)
        Other data format:    1123626 (    3593.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1191 (       3.8 Hz)
                               RESUME:       1191 (       3.8 Hz)
                              SYNC100:      33165 (     106.1 Hz)
                              WR48-63:      33165 (     106.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1054914 (    3373.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     350507 (    1120.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.698 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.889                0.000
  1               39.498                0.000
  2                0.306                0.000
  3                5.270                0.000
  4                0.000               26.707
  5                0.036                0.000
  6                0.217                0.000
  7                0.175                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903831      29785          0          0        188        188       4074       4074          0      21261          0      87155
  1   61666403      16932          0          0        646        646       7820       7820          0          0          0      55280
  2   20140383     267008          0          0         42         42       2575       2575          0     261774          0      55290
  3   41652352     266495          0          0        164        164       5306       5306          0     255555          0      28879
  4    4871487       1204          0          0          0          0        602        602          0          0          0      28340
  5   17605547       4430          0          0         11         11       2204       2204          0          0          0      33538
  6   41056242     264471          0          0         55         55       5126       5126          0     254109          0      47278
  7   42900129     273301          0          0         85         85       5458       5458          0     262215          0      14747
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.305s (  774.730 blocks/s,  48.421 Mb/s)
  495   Fri Jun 24 00:34:55 2022 AMFriday 24 June 00:00-08:00

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good

Attachment 1: 220624_0130_fpgatime.png
220624_0130_fpgatime.png
Attachment 2: 220624_0130_whiterabbit.png
220624_0130_whiterabbit.png
Attachment 3: 220624_0130_temps.png
220624_0130_temps.png
Attachment 4: 220624_0130_stats.png
220624_0130_stats.png
Attachment 5: 220624_0130_bias.png
220624_0130_bias.png
Attachment 6: 220624_0530_fpgatime.png
220624_0530_fpgatime.png
Attachment 7: 220624_0530_whiterabbit.png
220624_0530_whiterabbit.png
Attachment 8: 220624_0530_temps.png
220624_0530_temps.png
Attachment 9: 220624_0530_stats.png
220624_0530_stats.png
Attachment 10: 220624_0530_bias.png
220624_0530_bias.png
Attachment 11: 220624_0330_fpgatime.png
220624_0330_fpgatime.png
Attachment 12: 220624_0330_whiterabbit.png
220624_0330_whiterabbit.png
Attachment 13: 220624_0330_temps.png
220624_0330_temps.png
Attachment 14: 220624_0330_stats.png
220624_0330_stats.png
Attachment 15: 220624_0330_bias.png
220624_0330_bias.png
Attachment 16: 220624_0730_fpga.png
220624_0730_fpga.png
Attachment 17: 220624_0730_whiterabbit.png
220624_0730_whiterabbit.png
Attachment 18: 220624_0730_temps.png
220624_0730_temps.png
Attachment 19: 220624_0730_stats.png
220624_0730_stats.png
Attachment 20: 220624_0730_bias.png
220624_0730_bias.png
  494   Thu Jun 23 15:10:17 2022 Philippos PapadakisThursday 23 June 16:00-24:00
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc525 : 	 39  
aida08 fault 	 0xf0e9 : 	 0xf166 : 	 125  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
                 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x23 : 	 9  

Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)


18:05
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc527 : 	 41  
aida08 fault 	 0xf0e9 : 	 0xf177 : 	 142  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x24 : 	 10  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)


20:22
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf17f : 	 150  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x20 : 	 15  
aida08 fault 	 0x1a : 	 0x25 : 	 11  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)

21:56
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf186 : 	 157  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x26 : 	 12  

Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)

23:50
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc52e : 	 48  
aida08 fault 	 0xf0e9 : 	 0xf18c : 	 163  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
	
			 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x27 : 	 13  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15)
Attachment 1: Temperatures_2022-06-23_16-14-38.png
Temperatures_2022-06-23_16-14-38.png
Attachment 2: stats_2022-06-23_16-16-30.png
stats_2022-06-23_16-16-30.png
Attachment 3: IV2022-06-23_16-17-13.png
IV2022-06-23_16-17-13.png
Attachment 4: Temps_2022-06-23_18-08-04.png
Temps_2022-06-23_18-08-04.png
Attachment 5: Stats_2022-06-23_18-08-38.png
Stats_2022-06-23_18-08-38.png
Attachment 6: IV_2022-06-23_18-09-41.png
IV_2022-06-23_18-09-41.png
Attachment 7: Temps_2022-06-23_20-24-48.png
Temps_2022-06-23_20-24-48.png
Attachment 8: Stats_2022-06-23_20-25-18.png
Stats_2022-06-23_20-25-18.png
Attachment 9: IV2022-06-23_20-25-39.png
IV2022-06-23_20-25-39.png
Attachment 10: Temps_2022-06-23_21-59-26.png
Temps_2022-06-23_21-59-26.png
Attachment 11: Stats_2022-06-23_22-00-01.png
Stats_2022-06-23_22-00-01.png
Attachment 12: IV_2022-06-23_22-00-20.png
IV_2022-06-23_22-00-20.png
Attachment 13: Temps_2022-06-23_23-52-04.png
Temps_2022-06-23_23-52-04.png
Attachment 14: Stats_2022-06-23_23-52-32.png
Stats_2022-06-23_23-52-32.png
Attachment 15: IV_2022-06-23_23-52-50.png
IV_2022-06-23_23-52-50.png
  493   Thu Jun 23 06:56:02 2022 OHThursday 23 June 08:00-16:00
07:56 Taken over from AM
      Current White Rabbit Error status:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51a : 	 28  
      aida08 fault 	 0xf0e9 : 	 0xf132 : 	 73  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA error status:
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link

      Statistics ok - attachment 1
      FEE temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      Analysis of file R3_93 - attachment 4
      Still deadtime in aida02 of ~15%
      Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
      
      Currently 2.2 TB of space on /media/SecondDrive
      Current data rate of 6.23MB/s
      At current data rate this should last 4 days

08:27 Currently no beam
      Problem with Alvarez 3 now
      Tomorrow will have a 1 hour break due to changing of the source

09:36 Beam back

09:59 System wide checks
		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51b : 	 29  
      aida08 fault 	 0xf0e9 : 	 0xf138 : 	 79  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  

      Stats ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7

11:14 FRS group is changing from Hg setting back to Pt

11:36 Back on Pt R3_129

14:14 Stats ok - attachment 8
      Temperature ok - attachment 9
      Bias and leakage currents ok - attachment 10
      System wide checks:
      WR:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc523 : 	 37  
      aida08 fault 	 0xf0e9 : 	 0xf156 : 	 109  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x1b : 	 10  
      aida08 fault 	 0x1a : 	 0x20 : 	 6

      Analysis of R3_156 - attachment 11  
Attachment 1: 220623_0758_Stats.png
220623_0758_Stats.png
Attachment 2: 220623_0759_Temp.png
220623_0759_Temp.png
Attachment 3: 220623_0759_Bias.png
220623_0759_Bias.png
Attachment 4: R3_95_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260243782 (  817307.6 Hz)
        Other data format:    1676218 (    5264.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1230 (       3.9 Hz)
                               RESUME:       1231 (       3.9 Hz)
                              SYNC100:      33186 (     104.2 Hz)
                              WR48-63:      33186 (     104.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1607385 (    5048.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     440709 (    1384.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.416 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.258                0.000
  1               47.390                0.000
  2                1.040                0.000
  3                6.094                0.000
  4                0.003               59.113
  5                0.094                0.000
  6                0.304                0.000
  7                0.013                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20778195      27751          0          0         46         46       2749       2749          0      22161          0      72140
  1   62611625      17407          0          0        705        706       7998       7998          0          0          0      66280
  2   31916579     410233          0          0         83         83       4122       4122          0     401823          0      94767
  3   52868439     408659          0          0        312        312       6874       6874          0     394287          0      28791
  4    5535248       1334          0          0          1          1        666        666          0          0          0      39988
  5   19704728       4912          0          0         25         25       2431       2431          0          0          0      52543
  6   35689179     392725          0          0         47         47       4446       4446          0     383739          0      67737
  7   31139789     413197          0          0         11         11       3900       3900          0     405375          0      18463
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.666s (  923.094 blocks/s,  57.693 Mb/s)
Attachment 5: 220623_0958_Stats.png
220623_0958_Stats.png
Attachment 6: 220623_0959_Temp.png
220623_0959_Temp.png
Attachment 7: 220623_1000_Bias.png
220623_1000_Bias.png
Attachment 8: 220623_1413_Stats.png
220623_1413_Stats.png
Attachment 9: 220623_1413_Temp.png
220623_1413_Temp.png
Attachment 10: 220623_1414_bias.png
220623_1414_bias.png
Attachment 11: R3_156_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260453729 (  852518.1 Hz)
        Other data format:    1466271 (    4799.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1362 (       4.5 Hz)
                               RESUME:       1362 (       4.5 Hz)
                              SYNC100:      33138 (     108.5 Hz)
                              WR48-63:      33138 (     108.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1397271 (    4573.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     435545 (    1425.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      305.511 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.365                0.000
  1               54.806                0.000
  2                0.562                0.000
  3                5.946                0.000
  4                0.000               35.352
  5                0.068                0.000
  6                0.334                0.000
  7                0.054                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30150644      28922          0          0        254        254       3938       3938          0      20538          0     108754
  1   64977250      18108          0          0        757        757       8297       8297          0          0          0      62895
  2   28023912     358551          0          0         83         83       3523       3523          0     351339          0      69361
  3   44222269     352853          0          0        180        180       5536       5536          0     341421          0      37017
  4    5005333       1272          0          0          0          0        636        636          0          0          0      36369
  5   19466878       5006          0          0         20         20       2483       2483          0          0          0      42187
  6   36422600     339861          0          0         37         37       4700       4700          0     330387          0      59710
  7   32184843     361698          0          0         31         31       4025       4025          0     353586          0      19252
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.418s (  903.496 blocks/s,  56.469 Mb/s)
  492   Wed Jun 22 23:18:45 2022 AM, OH, TD, MAThursday 23 June
00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
      Attempted to recover restarting merger but this caused 01 to drop out.

00:10 DAQ recovered and AIDA01 recovered with a reboot


01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good

03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good

05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good

07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good
Attachment 1: 220623_0100_whiterabbit.png
220623_0100_whiterabbit.png
Attachment 2: 220623_0100_temps.png
220623_0100_temps.png
Attachment 3: 220623_0100_stats.png
220623_0100_stats.png
Attachment 4: 220623_0100_bias.png
220623_0100_bias.png
Attachment 5: 220623_0300_whiterabbit.png
220623_0300_whiterabbit.png
Attachment 6: 220623_0300_fpgatime.png
220623_0300_fpgatime.png
Attachment 7: 220623_0300_temps.png
220623_0300_temps.png
Attachment 8: 220623_0300_stats.png
220623_0300_stats.png
Attachment 9: 220623_0300_bias.png
220623_0300_bias.png
Attachment 10: 220623_0500_fpgatime.png
220623_0500_fpgatime.png
Attachment 11: 220623_0500_whiterabbit.png
220623_0500_whiterabbit.png
Attachment 12: 220623_0500_temps.png
220623_0500_temps.png
Attachment 13: 220623_0500_stats.png
220623_0500_stats.png
Attachment 14: 220623_0500_bias.png
220623_0500_bias.png
Attachment 15: 220623_0700_fpgatime.png
220623_0700_fpgatime.png
Attachment 16: 220623_0700_whiterabbit.png
220623_0700_whiterabbit.png
Attachment 17: 220623_0700_temps.png
220623_0700_temps.png
Attachment 18: 220623_0700_stats.png
220623_0700_stats.png
Attachment 19: 220623_0700_bias.png
220623_0700_bias.png
  491   Wed Jun 22 15:09:37 2022 MAWednesday 22 June 16:00-00:00

16:00 Take over from Magda mrning shift

Beam was off of sometime for tunning to Mercurey setting.

17:00 attached 1-3

19:00 attached 4-6

21:00 attached 7-9

23:30 10-12

 

Attachment 1: BiasCurrent2022-06-22_17-11-07.png
BiasCurrent2022-06-22_17-11-07.png
Attachment 2: Statistics_2022-06-22_17-13-52.png
Statistics_2022-06-22_17-13-52.png
Attachment 3: Temperature2022-06-22_17-12-35.png
Temperature2022-06-22_17-12-35.png
Attachment 4: BiasCurrent2022-06-22_19-00-31.png
BiasCurrent2022-06-22_19-00-31.png
Attachment 5: Temperature2022-06-22_19-02-18.png
Temperature2022-06-22_19-02-18.png
Attachment 6: Statics2022-06-22_19-03-11.png
Statics2022-06-22_19-03-11.png
Attachment 7: BiasCurrent2022-06-22_21-04-38.png
BiasCurrent2022-06-22_21-04-38.png
Attachment 8: Statistics2022-06-22_21-07-07.png
Statistics2022-06-22_21-07-07.png
Attachment 9: Temperature2022-06-22_21-05-51.png
Temperature2022-06-22_21-05-51.png
Attachment 10: BiasCurrent2022-06-22_23-29-40.png
BiasCurrent2022-06-22_23-29-40.png
Attachment 11: Statistics2022-06-22_23-31-30.png
Statistics2022-06-22_23-31-30.png
Attachment 12: Temperature2022-06-22_23-30-28.png
Temperature2022-06-22_23-30-28.png
  490   Wed Jun 22 10:09:12 2022 PJCSINFO: FEE64 supply Voltages

Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

  489   Wed Jun 22 07:12:46 2022 MSWednesday 22 June 8:00-16:00

Took over the night shift from Tom.

8:10 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.

12:19 OH Disabled all discriminators to reduce the data rate.

13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10

14:00 : attachments 11-13

16:00 : attachments 14-16

 

Attachment 1: Screenshot_from_2022-06-22_08-11-30.png
Screenshot_from_2022-06-22_08-11-30.png
Attachment 2: Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-06-22_10-00-13.png
Screenshot_from_2022-06-22_10-00-13.png
Attachment 7: Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_from_2022-06-22_11-59-55.png
Screenshot_from_2022-06-22_11-59-55.png
Attachment 10: R2_104_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260553751 (  867725.3 Hz)
        Other data format:    1366249 (    4550.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1464 (       4.9 Hz)
                               RESUME:       1464 (       4.9 Hz)
                              SYNC100:      33119 (     110.3 Hz)
                              WR48-63:      33119 (     110.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1297083 (    4319.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     518974 (    1728.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      300.272 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.776                0.000
  1               47.563                0.000
  2                0.466                0.000
  3                4.438                0.000
  4                0.149               14.780
  5               16.612                0.000
  6                0.972                0.000
  7                0.380                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27837591      27876          0          0        187        187       3557       3557          0      20388          0     104437
  1   57538232      16015          0          0        651        652       7356       7356          0          0          0      64114
  2   23047755     331545          0          0         63         63       2949       2949          0     325521          0      64002
  3   44324924     329582          0          0        146        145       5671       5671          0     317949          0      37242
  4   11203550       2766          0          0         57         57       1326       1326          0          0          0      90620
  5   28839527       7636          0          0        219        219       3599       3599          0          0          0      65648
  6   37663072     317251          0          0         87         87       4772       4772          0     307533          0      63360
  7   30099100     333578          0          0         54         54       3889       3889          0     325692          0      29551
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.320s (  932.392 blocks/s,  58.275 Mb/s)
Attachment 11: Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
Attachment 12: Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 13: Screenshot_from_2022-06-22_14-03-14.png
Screenshot_from_2022-06-22_14-03-14.png
Attachment 14: Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_from_2022-06-22_15-58-16.png
Screenshot_from_2022-06-22_15-58-16.png
  488   Tue Jun 21 23:29:16 2022 TDWednesday 22 June 00:00-08:00
00.28 all histograms & stats zero'd
      baseline counters

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

BNC PB-5 pulser
 to p+n FEE64s only
  amplitude 1.0V
  attenuator x1
  decay time 1ms
  polarity +
  frequency 2Hz

all AD9252 waveform ADCs disabled

Tape Server -> no storage mode

00.36 check ASIC control all FEE64s all ASICs

      all system wide checks OK

00.41 BNC PB-5 pulser rate increased from 2Hz to 22Hz per request by HA/NH - to ensure DTAS correlation checker has sufficient rate of identified pulser events 

per p+n FEE64 1.8.L spectra - attachment 1
 aida01 pulser peak width 88 ch FWHM

per FEE64 rate spectra - attachment 2

adc, pause and correlation scaler data items - push, aida01 and aida02 stats - attachments 3-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage current OK - attachments 10 & 11

02.28 aida05 zero adc data items, unable to halt aida05 - all other FEE64s OK

      restart NewMerger -> now able to halt aida05

      DAQ go OK, merge rate c. 1.4M data items/s

per FEE64 1.8.H spectra - attachments 12 & 13
 approx flat distribution to c. 10GeV


adc data items stats OK - attachments 14

FEE64 temps OK - attachment 15

DSSSD bias & leakage current OK - attachments 16 & 17

02.55 Some problems with GO4 data analysis for PID (FRS PID looks OK). 204Pt setting. Will commence writing data to disk file S505/R2_0

04.12 per FEE64 1.8.H spectra - attachments 18 & 19
      c. 9GeV peak in upstream DSSSDs (aida01, 2, 3 & 4) indicating (high energy) heavy fragments are stopping
      Low energy events near c. 200MeV lower mass, high energy, low energy loss - fission fragments?

04.20 per FEE64 stats spectra - attachment 20
      indicates implant x-y distribution - off centre (more significantly in x than in y) but DSSSDs appear to see most of distribution
      remember raw distro will have significant fraction of fission fragments - may be misleading - need to gate on high energy events to see x-y distro of ions of 
interest

adc data items stats OK - attachments 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage current OK - attachments 23



06.10 per FEE64 rate spectra - attachment 24

per FEE64 1.8.H spectra - attachments 25 & 26

per FEE64 stats spectra - attachment 27
     
adc data items stats OK - attachments 28

FEE64 temps OK - attachment 29

system wide checks OK *except* WR & FPGA errors - see attachments 30 & 31

DSSSD bias & leakage current OK - attachments 32

07.52 analysis of file S505/R2_47 - attachment 33
      all FEE64 dead times < 1% *except* aida02 c. 7%
Attachment 1: Screenshot_from_2022-06-22_00-50-20.png
Screenshot_from_2022-06-22_00-50-20.png
Attachment 2: Screenshot_from_2022-06-22_00-49-07.png
Screenshot_from_2022-06-22_00-49-07.png
Attachment 3: Screenshot_from_2022-06-22_00-47-56.png
Screenshot_from_2022-06-22_00-47-56.png
Attachment 4: Screenshot_from_2022-06-22_00-46-57.png
Screenshot_from_2022-06-22_00-46-57.png
Attachment 5: Screenshot_from_2022-06-22_00-46-17.png
Screenshot_from_2022-06-22_00-46-17.png
Attachment 6: Screenshot_from_2022-06-22_00-45-33.png
Screenshot_from_2022-06-22_00-45-33.png
Attachment 7: Screenshot_from_2022-06-22_00-44-58.png
Screenshot_from_2022-06-22_00-44-58.png
Attachment 8: Screenshot_from_2022-06-22_00-44-13.png
Screenshot_from_2022-06-22_00-44-13.png
Attachment 9: Screenshot_from_2022-06-22_00-36-06.png
Screenshot_from_2022-06-22_00-36-06.png
Attachment 10: Screenshot_from_2022-06-22_00-35-17.png
Screenshot_from_2022-06-22_00-35-17.png
Attachment 11: Screenshot_from_2022-06-22_00-34-01.png
Screenshot_from_2022-06-22_00-34-01.png
Attachment 12: Screenshot_from_2022-06-22_02-44-02.png
Screenshot_from_2022-06-22_02-44-02.png
Attachment 13: Screenshot_from_2022-06-22_02-41-13.png
Screenshot_from_2022-06-22_02-41-13.png
Attachment 14: Screenshot_from_2022-06-22_02-40-14.png
Screenshot_from_2022-06-22_02-40-14.png
Attachment 15: Screenshot_from_2022-06-22_02-39-42.png
Screenshot_from_2022-06-22_02-39-42.png
Attachment 16: Screenshot_from_2022-06-22_02-39-01.png
Screenshot_from_2022-06-22_02-39-01.png
Attachment 17: Screenshot_from_2022-06-22_02-38-24.png
Screenshot_from_2022-06-22_02-38-24.png
Attachment 18: Screenshot_from_2022-06-22_04-10-47.png
Screenshot_from_2022-06-22_04-10-47.png
Attachment 19: Screenshot_from_2022-06-22_04-10-00.png
Screenshot_from_2022-06-22_04-10-00.png
Attachment 20: Screenshot_from_2022-06-22_04-23-13.png
Screenshot_from_2022-06-22_04-23-13.png
Attachment 21: Screenshot_from_2022-06-22_04-20-06.png
Screenshot_from_2022-06-22_04-20-06.png
Attachment 22: Screenshot_from_2022-06-22_04-19-34.png
Screenshot_from_2022-06-22_04-19-34.png
Attachment 23: Screenshot_from_2022-06-22_04-19-00.png
Screenshot_from_2022-06-22_04-19-00.png
Attachment 24: Screenshot_from_2022-06-22_06-10-41.png
Screenshot_from_2022-06-22_06-10-41.png
Attachment 25: Screenshot_from_2022-06-22_06-09-33.png
Screenshot_from_2022-06-22_06-09-33.png
Attachment 26: Screenshot_from_2022-06-22_06-08-08.png
Screenshot_from_2022-06-22_06-08-08.png
Attachment 27: Screenshot_from_2022-06-22_06-06-50.png
Screenshot_from_2022-06-22_06-06-50.png
Attachment 28: Screenshot_from_2022-06-22_06-05-58.png
Screenshot_from_2022-06-22_06-05-58.png
Attachment 29: Screenshot_from_2022-06-22_06-05-20.png
Screenshot_from_2022-06-22_06-05-20.png
Attachment 30: Screenshot_from_2022-06-22_06-04-27.png
Screenshot_from_2022-06-22_06-04-27.png
Attachment 31: Screenshot_from_2022-06-22_06-03-53.png
Screenshot_from_2022-06-22_06-03-53.png
Attachment 32: Screenshot_from_2022-06-22_06-03-05.png
Screenshot_from_2022-06-22_06-03-05.png
Attachment 33: R2_47
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259428432 (  714533.4 Hz)
        Other data format:    2491568 (    6862.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        881 (       2.4 Hz)
                               RESUME:        880 (       2.4 Hz)
                              SYNC100:      33352 (      91.9 Hz)
                              WR48-63:      33352 (      91.9 Hz)
                           FEE64 disc:    1422279 (    3917.3 Hz)
                             MBS info:    1000824 (    2756.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     416179 (    1146.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      363.074 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.857                0.000
  1               26.452                0.000
  2                0.132                0.000
  3                1.570                0.000
  4                0.002               46.041
  5                0.661                0.000
  6                0.312                0.000
  7                0.027                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   25701314     249377          0          0         56         56       3411       3411     217219      25224          0      83012
  1   63351849     247459          0          0        567        566       8361       8361     229604          0          0      62244
  2   23166597     387869          0          0         37         37       3010       3010     134065     247710          0      48973
  3   48876539     363237          0          0        105        105       6171       6171     104940     245745          0      28921
  4    8006429     267186          0          0          3          3       1045       1045     265090          0          0      68788
  5   22095663     170195          0          0         36         36       2789       2789     164545          0          0      52757
  6   40483353     472437          0          0         66         66       5106       5106     227985     234108          0      48544
  7   27746688     333808          0          0         11         11       3459       3459      78831     248037          0      22940
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.072s (  912.402 blocks/s,  57.025 Mb/s)
  487   Tue Jun 21 10:19:05 2022 OH MLTuesday 21st June 08:00-00:00
11:19 Stats ok 0xa on all FEE - attachment 1
      Current pulser settings - attachment 2
      FEE Temperatures ok - attachment 3
      Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
      Layout 1 after recovering channels - attachment 4
      New stats (Note increased rate on 3 and 7) attachment 5
      Bias and leakage currents ok - attachment 6

      S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.

14:05 Have had it confirmed that the flange beam dump is in position in S4

15:43 Stats ok -attachment 7
      Temp ok - attachment 8
      Bias and Leakage current ok - attachment 9


17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
      As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
      During the experiment this will be changed to SC1 trigger to solve this.

19:49: Stats ok - attachment 10
       temperatures ok - attachment 11
       Bias - Leakage current ok - attachment 12
Attachment 1: 220621_1119_Stats.png
220621_1119_Stats.png
Attachment 2: 220621_1120_Pulser.png
220621_1120_Pulser.png
Attachment 3: 220621_1120_Temp.png
220621_1120_Temp.png
Attachment 4: 220621_1123_Layout1.png
220621_1123_Layout1.png
Attachment 5: 220621_1123_Stats.png
220621_1123_Stats.png
Attachment 6: 220621_1123_Bias.png
220621_1123_Bias.png
Attachment 7: 220621_1542_Stats.png
220621_1542_Stats.png
Attachment 8: 220621_1542_Temp.png
220621_1542_Temp.png
Attachment 9: 220621_1543_Bias.png
220621_1543_Bias.png
Attachment 10: Stats-Screenshot_from_2022-06-21_19-50-19.png
Stats-Screenshot_from_2022-06-21_19-50-19.png
Attachment 11: Temp-Screenshot_from_2022-06-21_19-48-56.png
Temp-Screenshot_from_2022-06-21_19-48-56.png
Attachment 12: HV_LC-Screenshot_from_2022-06-21_19-47-49.png
HV_LC-Screenshot_from_2022-06-21_19-47-49.png
  486   Mon Jun 20 09:26:23 2022 OH, TDMonday 20 June
10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11
Attachment 1: 220620_1116_stats_no_waveform.png
220620_1116_stats_no_waveform.png
Attachment 2: 220620_1223_stats_no_waveform.png
220620_1223_stats_no_waveform.png
Attachment 3: Screenshot_from_2022-06-20_21-17-41.png
Screenshot_from_2022-06-20_21-17-41.png
Attachment 4: Screenshot_from_2022-06-20_21-14-12.png
Screenshot_from_2022-06-20_21-14-12.png
Attachment 5: Screenshot_from_2022-06-20_21-12-06.png
Screenshot_from_2022-06-20_21-12-06.png
Attachment 6: Screenshot_from_2022-06-20_21-10-55.png
Screenshot_from_2022-06-20_21-10-55.png
Attachment 7: Screenshot_from_2022-06-20_21-08-58.png
Screenshot_from_2022-06-20_21-08-58.png
Attachment 8: Screenshot_from_2022-06-20_21-04-58.png
Screenshot_from_2022-06-20_21-04-58.png
Attachment 9: Screenshot_from_2022-06-20_21-04-00.png
Screenshot_from_2022-06-20_21-04-00.png
Attachment 10: Screenshot_from_2022-06-20_21-02-07.png
Screenshot_from_2022-06-20_21-02-07.png
Attachment 11: Screenshot_from_2022-06-20_21-01-07.png
Screenshot_from_2022-06-20_21-01-07.png
  485   Sun Jun 19 19:55:20 2022 NHaida03 and aida04
aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour
Attachment 1: Screenshot_2022-06-19_205528.png
Screenshot_2022-06-19_205528.png
Attachment 2: Screenshot_2022-06-19_205544.png
Screenshot_2022-06-19_205544.png
  484   Sun Jun 19 12:36:10 2022 OHSunday 19th June
Yesterday the sum/inverter was found to be a considerable source of noise in the system.

Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace

Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow

Put inverter back in CAEN crate in S4 and confirm similar behavior.

Move CAEN crate to new mains port, same behaviour

Move CAEN crate to AC voltage stabiliser, same behaviour

Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears  - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears

Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.

MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd

CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?

MACBs and inverter checked on another crate - no signal

Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected

Tested adding bias filters - No change

15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
      We get ok stats even with waveforms on at 0xa- attachment 5

      Waveforms still have the 100kHz oscillation in them though attachments 6 and 7

      Without waveforms we get similar performance to before the switch to the new mains - attachment 8

15:56 Move CAEN NIM crate across to this mains network
      300+kHz on all FEEs with waveforms on. Can recover waveform off performance though

16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.

17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9
Attachment 1: 1.jpg
1.jpg
Attachment 2: 2.jpg
2.jpg
Attachment 3: 3.jpg
3.jpg
Attachment 4: 4.jpg
4.jpg
Attachment 5: 220618_1753_stats_no_waveform.png
220618_1753_stats_no_waveform.png
  483   Sun Jun 19 00:22:24 2022 TDSunday 19 June
01.20 

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

All waveform ADCs disabled

BNC PB-5 pulser
 to p+n FEE64s only

all system wide checks OK *except* ADC calibration, WR decoder & FPGA timestamp errors - see attachments 1 & 2

DSSSD bias & leakage current OK - see attachments 3 & 4

FEE64 temps OK - see attachment 5

statistics OK - ADC data, pause, resume & correlation scaler data items
                data push, data flush, aida01, aida02 - see attachments 6-13

per FEE64 1.8.L spectra - see attachment 14
 aida01 pulser peak width 73 ch FWHM
Attachment 1: Screenshot_from_2022-06-19_01-24-50.png
Screenshot_from_2022-06-19_01-24-50.png
Attachment 2: Screenshot_from_2022-06-19_01-24-24.png
Screenshot_from_2022-06-19_01-24-24.png
Attachment 3: Screenshot_from_2022-06-19_01-20-50.png
Screenshot_from_2022-06-19_01-20-50.png
Attachment 4: Screenshot_from_2022-06-19_01-20-09.png
Screenshot_from_2022-06-19_01-20-09.png
Attachment 5: Screenshot_from_2022-06-19_01-19-37.png
Screenshot_from_2022-06-19_01-19-37.png
Attachment 6: Screenshot_from_2022-06-19_01-18-35.png
Screenshot_from_2022-06-19_01-18-35.png
Attachment 7: Screenshot_from_2022-06-19_01-18-05.png
Screenshot_from_2022-06-19_01-18-05.png
Attachment 8: Screenshot_from_2022-06-19_01-17-28.png
Screenshot_from_2022-06-19_01-17-28.png
Attachment 9: Screenshot_from_2022-06-19_01-15-44.png
Screenshot_from_2022-06-19_01-15-44.png
Attachment 10: Screenshot_from_2022-06-19_01-14-59.png
Screenshot_from_2022-06-19_01-14-59.png
Attachment 11: Screenshot_from_2022-06-19_01-14-24.png
Screenshot_from_2022-06-19_01-14-24.png
Attachment 12: Screenshot_from_2022-06-19_01-13-43.png
Screenshot_from_2022-06-19_01-13-43.png
Attachment 13: Screenshot_from_2022-06-19_01-13-03.png
Screenshot_from_2022-06-19_01-13-03.png
Attachment 14: Screenshot_from_2022-06-19_01-12-16.png
Screenshot_from_2022-06-19_01-12-16.png
  482   Sat Jun 18 09:46:57 2022 OH, NHFriday 17th June
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23
Attachment 1: 220617_1322_stats.png
220617_1322_stats.png
Attachment 2: 220617_1420_stats.png
220617_1420_stats.png
Attachment 3: 220617_11428_stats.png
220617_11428_stats.png
Attachment 4: 220617_1428_layout1.png
220617_1428_layout1.png
Attachment 5: 220617_1440_stats_waveform_on.png
220617_1440_stats_waveform_on.png
Attachment 6: 220617_1444_stats_no_waveform.png
220617_1444_stats_no_waveform.png
Attachment 7: 220617_1459_stats_waveform_on.png
220617_1459_stats_waveform_on.png
Attachment 8: 220617_1517_stats_no_waveform.png
220617_1517_stats_no_waveform.png
Attachment 9: 220617_1624_stats_waveform_on.png
220617_1624_stats_waveform_on.png
Attachment 10: 220617_1626_stats_waveform_off.png
220617_1626_stats_waveform_off.png
Attachment 11: 220617_1649_stats_waveform_on.png
220617_1649_stats_waveform_on.png
Attachment 12: 220617_1652_no_waveform.png
220617_1652_no_waveform.png
Attachment 13: 220617_1709_waveform_on.png
220617_1709_waveform_on.png
Attachment 14: 220617_1724_stats_no_waveform.png
220617_1724_stats_no_waveform.png
Attachment 15: 220617_1729_stats_waveform_on.png
220617_1729_stats_waveform_on.png
Attachment 16: 220617_1733_stats_no_waveform.png
220617_1733_stats_no_waveform.png
Attachment 17: 220617_1736_0x21.png
220617_1736_0x21.png
Attachment 18: 220617_1738_0x14.png
220617_1738_0x14.png
Attachment 19: 220617_1738_0x12.png
220617_1738_0x12.png
Attachment 20: 220617_1807_stats_waveform_on.png
220617_1807_stats_waveform_on.png
Attachment 21: 220617_1832_stats_no_waveform.png
220617_1832_stats_no_waveform.png
Attachment 22: 220617_1844_0xf.png
220617_1844_0xf.png
Attachment 23: 220617_1845_stats_waveform_on.png
220617_1845_stats_waveform_on.png
  481   Tue Jun 14 12:15:33 2022 NHS450 Archived
All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 
Attachment 1: s450_aida_tape.txt
R1_0.gz: (0x00000009) exists archived, archive_id:13
R1_100.gz: (0x00000009) exists archived, archive_id:13
R1_101.gz: (0x00000009) exists archived, archive_id:13
R1_102.gz: (0x00000009) exists archived, archive_id:13
R1_103.gz: (0x00000009) exists archived, archive_id:13
R1_104.gz: (0x00000009) exists archived, archive_id:13
R1_105.gz: (0x00000009) exists archived, archive_id:13
R1_106.gz: (0x00000009) exists archived, archive_id:13
R1_107.gz: (0x00000009) exists archived, archive_id:13
R1_108.gz: (0x00000009) exists archived, archive_id:13
R1_109.gz: (0x00000009) exists archived, archive_id:13
R1_10.gz: (0x00000009) exists archived, archive_id:13
R1_110.gz: (0x00000009) exists archived, archive_id:13
R1_111.gz: (0x00000009) exists archived, archive_id:13
R1_112.gz: (0x00000009) exists archived, archive_id:13
R1_113.gz: (0x00000009) exists archived, archive_id:13
R1_114.gz: (0x00000009) exists archived, archive_id:13
R1_115.gz: (0x00000009) exists archived, archive_id:13
R1_116.gz: (0x00000009) exists archived, archive_id:13
R1_117.gz: (0x00000009) exists archived, archive_id:13
R1_118.gz: (0x00000009) exists archived, archive_id:13
R1_119.gz: (0x00000009) exists archived, archive_id:13
R1_11.gz: (0x00000009) exists archived, archive_id:13
R1_120.gz: (0x00000009) exists archived, archive_id:13
R1_121.gz: (0x00000009) exists archived, archive_id:13
R1_122.gz: (0x00000009) exists archived, archive_id:13
R1_123.gz: (0x00000009) exists archived, archive_id:13
R1_124.gz: (0x00000009) exists archived, archive_id:13
R1_125.gz: (0x00000009) exists archived, archive_id:13
R1_126.gz: (0x00000009) exists archived, archive_id:13
R1_127.gz: (0x00000009) exists archived, archive_id:13
R1_128.gz: (0x00000009) exists archived, archive_id:13
R1_129.gz: (0x00000009) exists archived, archive_id:13
R1_12.gz: (0x00000009) exists archived, archive_id:13
R1_130.gz: (0x00000009) exists archived, archive_id:13
R1_131.gz: (0x00000009) exists archived, archive_id:13
R1_132.gz: (0x00000009) exists archived, archive_id:13
R1_133.gz: (0x00000009) exists archived, archive_id:13
R1_134.gz: (0x00000009) exists archived, archive_id:13
R1_135.gz: (0x00000009) exists archived, archive_id:13
R1_136.gz: (0x00000009) exists archived, archive_id:13
R1_137.gz: (0x00000009) exists archived, archive_id:13
R1_138.gz: (0x00000009) exists archived, archive_id:13
R1_139.gz: (0x00000009) exists archived, archive_id:13
R1_13.gz: (0x00000009) exists archived, archive_id:13
R1_140.gz: (0x00000009) exists archived, archive_id:13
R1_141.gz: (0x00000009) exists archived, archive_id:13
R1_142.gz: (0x00000009) exists archived, archive_id:13
R1_143.gz: (0x00000009) exists archived, archive_id:13
R1_144.gz: (0x00000009) exists archived, archive_id:13
R1_145.gz: (0x00000009) exists archived, archive_id:13
R1_146.gz: (0x00000009) exists archived, archive_id:13
R1_147.gz: (0x00000009) exists archived, archive_id:13
R1_148.gz: (0x00000009) exists archived, archive_id:13
R1_149.gz: (0x00000009) exists archived, archive_id:13
R1_14.gz: (0x00000009) exists archived, archive_id:13
R1_150.gz: (0x00000009) exists archived, archive_id:13
R1_151.gz: (0x00000009) exists archived, archive_id:13
R1_152.gz: (0x00000009) exists archived, archive_id:13
R1_153.gz: (0x00000009) exists archived, archive_id:13
R1_154.gz: (0x00000009) exists archived, archive_id:13
R1_155.gz: (0x00000009) exists archived, archive_id:13
R1_156.gz: (0x00000009) exists archived, archive_id:13
R1_157.gz: (0x00000009) exists archived, archive_id:13
R1_158.gz: (0x00000009) exists archived, archive_id:13
R1_159.gz: (0x00000009) exists archived, archive_id:13
R1_15.gz: (0x00000009) exists archived, archive_id:13
R1_160.gz: (0x00000009) exists archived, archive_id:13
R1_161.gz: (0x00000009) exists archived, archive_id:13
R1_162.gz: (0x00000009) exists archived, archive_id:13
R1_163.gz: (0x00000009) exists archived, archive_id:13
R1_164.gz: (0x00000009) exists archived, archive_id:13
R1_165.gz: (0x00000009) exists archived, archive_id:13
R1_166.gz: (0x00000009) exists archived, archive_id:13
R1_167.gz: (0x00000009) exists archived, archive_id:13
R1_168.gz: (0x00000009) exists archived, archive_id:13
R1_169.gz: (0x00000009) exists archived, archive_id:13
R1_16.gz: (0x00000009) exists archived, archive_id:13
R1_170.gz: (0x00000009) exists archived, archive_id:13
R1_171.gz: (0x00000009) exists archived, archive_id:13
R1_172.gz: (0x00000009) exists archived, archive_id:13
R1_173.gz: (0x00000009) exists archived, archive_id:13
R1_174.gz: (0x00000009) exists archived, archive_id:13
R1_175.gz: (0x00000009) exists archived, archive_id:13
R1_176.gz: (0x00000009) exists archived, archive_id:13
R1_177.gz: (0x00000009) exists archived, archive_id:13
R1_178.gz: (0x00000009) exists archived, archive_id:13
R1_179.gz: (0x00000009) exists archived, archive_id:13
R1_17.gz: (0x00000009) exists archived, archive_id:13
R1_180.gz: (0x00000009) exists archived, archive_id:13
R1_181.gz: (0x00000009) exists archived, archive_id:13
R1_182.gz: (0x00000009) exists archived, archive_id:13
R1_183.gz: (0x00000009) exists archived, archive_id:13
R1_184.gz: (0x00000009) exists archived, archive_id:13
R1_185.gz: (0x00000009) exists archived, archive_id:13
R1_186.gz: (0x00000009) exists archived, archive_id:13
R1_187.gz: (0x00000009) exists archived, archive_id:13
R1_188.gz: (0x00000009) exists archived, archive_id:13
R1_189.gz: (0x00000009) exists archived, archive_id:13
R1_18.gz: (0x00000009) exists archived, archive_id:13
R1_190.gz: (0x00000009) exists archived, archive_id:13
R1_191.gz: (0x00000009) exists archived, archive_id:13
R1_192.gz: (0x00000009) exists archived, archive_id:13
R1_193.gz: (0x00000009) exists archived, archive_id:13
R1_194.gz: (0x00000009) exists archived, archive_id:13
R1_195.gz: (0x00000009) exists archived, archive_id:13
R1_196.gz: (0x00000009) exists archived, archive_id:13
R1_197.gz: (0x00000009) exists archived, archive_id:13
R1_198.gz: (0x00000009) exists archived, archive_id:13
R1_199.gz: (0x00000009) exists archived, archive_id:13
R1_19.gz: (0x00000009) exists archived, archive_id:13
R1_1.gz: (0x00000009) exists archived, archive_id:13
R1_200.gz: (0x00000009) exists archived, archive_id:13
R1_201.gz: (0x00000009) exists archived, archive_id:13
R1_202.gz: (0x00000009) exists archived, archive_id:13
R1_203.gz: (0x00000009) exists archived, archive_id:13
R1_204.gz: (0x00000009) exists archived, archive_id:13
R1_205.gz: (0x00000009) exists archived, archive_id:13
R1_206.gz: (0x00000009) exists archived, archive_id:13
R1_207.gz: (0x00000009) exists archived, archive_id:13
R1_208.gz: (0x00000009) exists archived, archive_id:13
R1_209.gz: (0x00000009) exists archived, archive_id:13
R1_20.gz: (0x00000009) exists archived, archive_id:13
R1_210.gz: (0x00000009) exists archived, archive_id:13
R1_211.gz: (0x00000009) exists archived, archive_id:13
R1_212.gz: (0x00000009) exists archived, archive_id:13
R1_213.gz: (0x00000009) exists archived, archive_id:13
R1_214.gz: (0x00000009) exists archived, archive_id:13
R1_215.gz: (0x00000009) exists archived, archive_id:13
R1_216.gz: (0x00000009) exists archived, archive_id:13
R1_217.gz: (0x00000009) exists archived, archive_id:13
R1_218.gz: (0x00000009) exists archived, archive_id:13
R1_219.gz: (0x00000009) exists archived, archive_id:13
R1_21.gz: (0x00000009) exists archived, archive_id:13
R1_220.gz: (0x00000009) exists archived, archive_id:13
R1_221.gz: (0x00000009) exists archived, archive_id:13
R1_222.gz: (0x00000009) exists archived, archive_id:13
R1_223.gz: (0x00000009) exists archived, archive_id:13
R1_224.gz: (0x00000009) exists archived, archive_id:13
R1_225.gz: (0x00000009) exists archived, archive_id:13
R1_226.gz: (0x00000009) exists archived, archive_id:13
R1_227.gz: (0x00000009) exists archived, archive_id:13
R1_228.gz: (0x00000009) exists archived, archive_id:13
R1_229.gz: (0x00000009) exists archived, archive_id:13
R1_22.gz: (0x00000009) exists archived, archive_id:13
R1_230.gz: (0x00000009) exists archived, archive_id:13
R1_231.gz: (0x00000009) exists archived, archive_id:13
R1_232.gz: (0x00000009) exists archived, archive_id:13
R1_233.gz: (0x00000009) exists archived, archive_id:13
R1_234.gz: (0x00000009) exists archived, archive_id:13
R1_235.gz: (0x00000009) exists archived, archive_id:13
R1_236.gz: (0x00000009) exists archived, archive_id:13
R1_237.gz: (0x00000009) exists archived, archive_id:13
R1_238.gz: (0x00000009) exists archived, archive_id:13
R1_239.gz: (0x00000009) exists archived, archive_id:13
R1_23.gz: (0x00000009) exists archived, archive_id:13
R1_240.gz: (0x00000009) exists archived, archive_id:13
R1_241.gz: (0x00000009) exists archived, archive_id:13
R1_242.gz: (0x00000009) exists archived, archive_id:13
R1_243.gz: (0x00000009) exists archived, archive_id:13
R1_244.gz: (0x00000009) exists archived, archive_id:13
R1_245.gz: (0x00000009) exists archived, archive_id:13
R1_246.gz: (0x00000009) exists archived, archive_id:13
R1_247.gz: (0x00000009) exists archived, archive_id:13
R1_248.gz: (0x00000009) exists archived, archive_id:13
R1_249.gz: (0x00000009) exists archived, archive_id:13
R1_24.gz: (0x00000009) exists archived, archive_id:13
R1_250.gz: (0x00000009) exists archived, archive_id:13
R1_251.gz: (0x00000009) exists archived, archive_id:13
R1_252.gz: (0x00000009) exists archived, archive_id:13
R1_253.gz: (0x00000009) exists archived, archive_id:13
R1_254.gz: (0x00000009) exists archived, archive_id:13
R1_255.gz: (0x00000009) exists archived, archive_id:13
R1_256.gz: (0x00000009) exists archived, archive_id:13
R1_257.gz: (0x00000009) exists archived, archive_id:13
R1_258.gz: (0x00000009) exists archived, archive_id:13
R1_259.gz: (0x00000009) exists archived, archive_id:13
R1_25.gz: (0x00000009) exists archived, archive_id:13
R1_260.gz: (0x00000009) exists archived, archive_id:13
R1_261.gz: (0x00000009) exists archived, archive_id:13
R1_262.gz: (0x00000009) exists archived, archive_id:13
R1_263.gz: (0x00000009) exists archived, archive_id:13
R1_264.gz: (0x00000009) exists archived, archive_id:13
R1_265.gz: (0x00000009) exists archived, archive_id:13
R1_266.gz: (0x00000009) exists archived, archive_id:13
R1_267.gz: (0x00000009) exists archived, archive_id:13
R1_268.gz: (0x00000009) exists archived, archive_id:13
R1_269.gz: (0x00000009) exists archived, archive_id:13
R1_26.gz: (0x00000009) exists archived, archive_id:13
R1_270.gz: (0x00000009) exists archived, archive_id:13
R1_271.gz: (0x00000009) exists archived, archive_id:13
R1_272.gz: (0x00000009) exists archived, archive_id:13
R1_273.gz: (0x00000009) exists archived, archive_id:13
R1_274.gz: (0x00000009) exists archived, archive_id:13
R1_275.gz: (0x00000009) exists archived, archive_id:13
R1_276.gz: (0x00000009) exists archived, archive_id:13
R1_277.gz: (0x00000009) exists archived, archive_id:13
R1_278.gz: (0x00000009) exists archived, archive_id:13
R1_279.gz: (0x00000009) exists archived, archive_id:13
R1_27.gz: (0x00000009) exists archived, archive_id:13
R1_280.gz: (0x00000009) exists archived, archive_id:13
R1_281.gz: (0x00000009) exists archived, archive_id:13
R1_282.gz: (0x00000009) exists archived, archive_id:13
R1_283.gz: (0x00000009) exists archived, archive_id:13
R1_284.gz: (0x00000009) exists archived, archive_id:13
R1_285.gz: (0x00000009) exists archived, archive_id:13
R1_286.gz: (0x00000009) exists archived, archive_id:13
R1_287.gz: (0x00000009) exists archived, archive_id:13
R1_288.gz: (0x00000009) exists archived, archive_id:13
R1_289.gz: (0x00000009) exists archived, archive_id:13
R1_28.gz: (0x00000009) exists archived, archive_id:13
R1_290.gz: (0x00000009) exists archived, archive_id:13
R1_291.gz: (0x00000009) exists archived, archive_id:13
R1_292.gz: (0x00000009) exists archived, archive_id:13
R1_293.gz: (0x00000009) exists archived, archive_id:13
R1_294.gz: (0x00000009) exists archived, archive_id:13
R1_295.gz: (0x00000009) exists archived, archive_id:13
R1_296.gz: (0x00000009) exists archived, archive_id:13
R1_297.gz: (0x00000009) exists archived, archive_id:13
R1_298.gz: (0x00000009) exists archived, archive_id:13
R1_299.gz: (0x00000009) exists archived, archive_id:13
R1_29.gz: (0x00000009) exists archived, archive_id:13
R1_2.gz: (0x00000009) exists archived, archive_id:13
R1_300.gz: (0x00000009) exists archived, archive_id:13
R1_301.gz: (0x00000009) exists archived, archive_id:13
R1_302.gz: (0x00000009) exists archived, archive_id:13
R1_303.gz: (0x00000009) exists archived, archive_id:13
R1_304.gz: (0x00000009) exists archived, archive_id:13
R1_305.gz: (0x00000009) exists archived, archive_id:13
R1_306.gz: (0x00000009) exists archived, archive_id:13
R1_307.gz: (0x00000009) exists archived, archive_id:13
R1_308.gz: (0x00000009) exists archived, archive_id:13
R1_309.gz: (0x00000009) exists archived, archive_id:13
R1_30.gz: (0x00000009) exists archived, archive_id:13
R1_310.gz: (0x00000009) exists archived, archive_id:13
R1_311.gz: (0x00000009) exists archived, archive_id:13
R1_312.gz: (0x00000009) exists archived, archive_id:13
R1_313.gz: (0x00000009) exists archived, archive_id:13
R1_314.gz: (0x00000009) exists archived, archive_id:13
R1_315.gz: (0x00000009) exists archived, archive_id:13
R1_316.gz: (0x00000009) exists archived, archive_id:13
R1_317.gz: (0x00000009) exists archived, archive_id:13
R1_318.gz: (0x00000009) exists archived, archive_id:13
R1_319.gz: (0x00000009) exists archived, archive_id:13
R1_31.gz: (0x00000009) exists archived, archive_id:13
R1_320.gz: (0x00000009) exists archived, archive_id:13
R1_321.gz: (0x00000009) exists archived, archive_id:13
R1_322.gz: (0x00000009) exists archived, archive_id:13
R1_323.gz: (0x00000009) exists archived, archive_id:13
R1_324.gz: (0x00000009) exists archived, archive_id:13
R1_325.gz: (0x00000009) exists archived, archive_id:13
R1_326.gz: (0x00000009) exists archived, archive_id:13
R1_327.gz: (0x00000009) exists archived, archive_id:13
R1_328.gz: (0x00000009) exists archived, archive_id:13
R1_329.gz: (0x00000009) exists archived, archive_id:13
R1_32.gz: (0x00000009) exists archived, archive_id:13
R1_330.gz: (0x00000009) exists archived, archive_id:13
R1_331.gz: (0x00000009) exists archived, archive_id:13
R1_332.gz: (0x00000009) exists archived, archive_id:13
R1_333.gz: (0x00000009) exists archived, archive_id:13
R1_334.gz: (0x00000009) exists archived, archive_id:13
R1_335.gz: (0x00000009) exists archived, archive_id:13
R1_336.gz: (0x00000009) exists archived, archive_id:13
R1_337.gz: (0x00000009) exists archived, archive_id:13
R1_338.gz: (0x00000009) exists archived, archive_id:13
R1_339.gz: (0x00000009) exists archived, archive_id:13
R1_33.gz: (0x00000009) exists archived, archive_id:13
R1_340.gz: (0x00000009) exists archived, archive_id:13
R1_341.gz: (0x00000009) exists archived, archive_id:13
R1_342.gz: (0x00000009) exists archived, archive_id:13
R1_343.gz: (0x00000009) exists archived, archive_id:13
R1_344.gz: (0x00000009) exists archived, archive_id:13
R1_345.gz: (0x00000009) exists archived, archive_id:13
R1_346.gz: (0x00000009) exists archived, archive_id:13
R1_347.gz: (0x00000009) exists archived, archive_id:13
R1_348.gz: (0x00000009) exists archived, archive_id:13
R1_349.gz: (0x00000009) exists archived, archive_id:13
R1_34.gz: (0x00000009) exists archived, archive_id:13
R1_350.gz: (0x00000009) exists archived, archive_id:13
R1_351.gz: (0x00000009) exists archived, archive_id:13
R1_352.gz: (0x00000009) exists archived, archive_id:13
R1_353.gz: (0x00000009) exists archived, archive_id:13
R1_354.gz: (0x00000009) exists archived, archive_id:13
R1_355.gz: (0x00000009) exists archived, archive_id:13
R1_356.gz: (0x00000009) exists archived, archive_id:13
R1_357.gz: (0x00000009) exists archived, archive_id:13
R1_358.gz: (0x00000009) exists archived, archive_id:13
R1_359.gz: (0x00000009) exists archived, archive_id:13
R1_35.gz: (0x00000009) exists archived, archive_id:13
R1_360.gz: (0x00000009) exists archived, archive_id:13
R1_361.gz: (0x00000009) exists archived, archive_id:13
R1_362.gz: (0x00000009) exists archived, archive_id:13
R1_363.gz: (0x00000009) exists archived, archive_id:13
R1_364.gz: (0x00000009) exists archived, archive_id:13
R1_365.gz: (0x00000009) exists archived, archive_id:13
R1_366.gz: (0x00000009) exists archived, archive_id:13
R1_367.gz: (0x00000009) exists archived, archive_id:13
R1_368.gz: (0x00000009) exists archived, archive_id:13
R1_369.gz: (0x00000009) exists archived, archive_id:13
R1_36.gz: (0x00000009) exists archived, archive_id:13
... 2178 more lines ...
  480   Mon Jun 13 16:38:37 2022 NHMon 13 Jun
Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change
  479   Fri Jun 10 10:32:46 2022 NHFri 10 Jun
Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 
Attachment 1: scopePSU1.jpg
scopePSU1.jpg
Attachment 2: scopePSU2.jpg
scopePSU2.jpg
Attachment 3: scoepPSU3.jpg
scoepPSU3.jpg
Attachment 4: scope5V_DC.jpg
scope5V_DC.jpg
Attachment 5: scopeN6V_DC.jpg
scopeN6V_DC.jpg
Attachment 6: scope_7VDC.jpg
scope_7VDC.jpg
Attachment 7: scope_RFAnt.jpg
scope_RFAnt.jpg
Attachment 8: scopeRF_pickup.jpg
scopeRF_pickup.jpg
  478   Wed Jun 8 09:55:28 2022 NHWeds 8 May
10.30 : Turn off AIDA FEE64s 

        Prepare a ground cable for snout - M10 terminal on one end for copper bar. Will affix with Cu tape for now (no free screws)

Plan for today:

Remove extraneous ground cables
Check ground of all FEE64s esp. 05 and 07 - include on FEE64 side.
 14:42 - Done all grounds seem snug, extra ground cables removed. Try power on and check

Try powering just DSSD1 FEE64s
 - No change (aida01 125k, aida02,03,04 - 400k)

Try grounding snout
 14:59 Connection to snout very bad as tape does not fix well to thick copper in braid. nO difference either way.

Probe PSU output with Battery DSO if available
 - See figure 2

Try shuffling FEE64 power allocation
 - 15:32 - Now as 1 X 3 X 2 X 4 X
                  5 X 7 X 6 X 8 X
(Also FEEs on own pair)
   No difference to rates

Multimeter voltages for last socket of PSU1 (paired with aida04):
  1 : 6.02 V
  2:  6.02 V
  3:  6.02 V
  4: -6.44 V
  5:  7.74 V

Similar on PSU2

Voltage between right (GND) pin 5 and 1 (pin 5 is COM) is -0.4 V, similar if pin 4 is COM. Pin1-3 are 0 V when paired
  
Attachment 1: 2022-06-08_14-48-21_goodevents.png
2022-06-08_14-48-21_goodevents.png
Attachment 2: rn_image_picker_lib_temp_97373248-bdea-4864-b8a9-a13d0d016e30.jpg
rn_image_picker_lib_temp_97373248-bdea-4864-b8a9-a13d0d016e30.jpg
ELOG V3.1.4-unknown