AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 24 of 36  ELOG logo
ID Date Author Subjectdown
  496   Fri Jun 24 06:56:06 2022 OHFriday 24 June 008:00-16:00
07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16
Attachment 1: 220624_0753_stats.png
220624_0753_stats.png
Attachment 2: 220624_0754_Temp.png
220624_0754_Temp.png
Attachment 3: 220624_0755_Bias.png
220624_0755_Bias.png
Attachment 4: R3_359_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260715255 (  897318.9 Hz)
        Other data format:    1204745 (    4146.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1323 (       4.6 Hz)
                               RESUME:       1323 (       4.6 Hz)
                              SYNC100:      33082 (     113.9 Hz)
                              WR48-63:      33082 (     113.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1135935 (    3909.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     353220 (    1215.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.549 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.171                0.000
  1               46.409                0.000
  2                1.006                0.000
  3                4.601                0.000
  4                0.000               37.574
  5                0.031                0.000
  6                0.191                0.000
  7                0.254                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903987      28078          0          0        236        236       4023       4023          0      19560          0      86851
  1   61208192      17252          0          0        718        718       7908       7908          0          0          0      52897
  2   26785987     291397          0          0         72         72       3425       3425          0     284403          0      56573
  3   43119726     289440          0          0        153        153       5319       5319          0     278496          0      29859
  4    4604412       1194          0          0          0          0        597        597          0          0          0      28791
  5   18336996       4656          0          0         16         16       2312       2312          0          0          0      34152
  6   36059875     277258          0          0         42         42       4514       4514          0     268146          0      48935
  7   39696080     295470          0          0         86         86       4984       4984          0     285330          0      15162
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.877s (  917.511 blocks/s,  57.344 Mb/s)
Attachment 5: 220624_0842_Stats_Stop.png
220624_0842_Stats_Stop.png
Attachment 6: 220624_1111_Stats.png
220624_1111_Stats.png
Attachment 7: 220624_1111_Temp.png
220624_1111_Temp.png
Attachment 8: 220624_1112_Bias.png
220624_1112_Bias.png
Attachment 9: 220624_1303_stats.png
220624_1303_stats.png
Attachment 10: 220624_1303_Temp.png
220624_1303_Temp.png
Attachment 11: 220624_1304_Bias.png
220624_1304_Bias.png
Attachment 12: R5_33_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260717099 (  869523.1 Hz)
        Other data format:    1202901 (    4011.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1344 (       4.5 Hz)
                               RESUME:       1344 (       4.5 Hz)
                              SYNC100:      33117 (     110.4 Hz)
                              WR48-63:      33117 (     110.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1133979 (    3782.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354749 (    1183.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      299.839 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.504                0.000
  1               42.102                0.000
  2                0.275                0.000
  3                6.677                0.000
  4                0.000               31.976
  5                0.051                0.000
  6                0.250                0.000
  7                0.412                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31758241      28867          0          0        238        238       4156       4156          0      20079          0      87480
  1   58453105      16104          0          0        635        635       7417       7417          0          0          0      53128
  2   20104679     289945          0          0         41         41       2565       2565          0     284733          0      56849
  3   41709424     287290          0          0        188        188       5262       5262          0     276390          0      29572
  4    5133297       1264          0          0          0          0        632        632          0          0          0      29278
  5   17857346       4470          0          0         22         22       2213       2213          0          0          0      34539
  6   40732502     278221          0          0         59         59       5175       5175          0     267753          0      48740
  7   44968505     296740          0          0        161        161       5697       5697          0     285024          0      15163
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.742s (  921.070 blocks/s,  57.567 Mb/s)
Attachment 13: 220624_1538_Stats.png
220624_1538_Stats.png
Attachment 14: 220624_1538_Temp.png
220624_1538_Temp.png
Attachment 15: 220624_1538_Bias.png
220624_1538_Bias.png
Attachment 16: R5_63_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260796374 (  834020.0 Hz)
        Other data format:    1123626 (    3593.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1191 (       3.8 Hz)
                               RESUME:       1191 (       3.8 Hz)
                              SYNC100:      33165 (     106.1 Hz)
                              WR48-63:      33165 (     106.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1054914 (    3373.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     350507 (    1120.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.698 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.889                0.000
  1               39.498                0.000
  2                0.306                0.000
  3                5.270                0.000
  4                0.000               26.707
  5                0.036                0.000
  6                0.217                0.000
  7                0.175                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903831      29785          0          0        188        188       4074       4074          0      21261          0      87155
  1   61666403      16932          0          0        646        646       7820       7820          0          0          0      55280
  2   20140383     267008          0          0         42         42       2575       2575          0     261774          0      55290
  3   41652352     266495          0          0        164        164       5306       5306          0     255555          0      28879
  4    4871487       1204          0          0          0          0        602        602          0          0          0      28340
  5   17605547       4430          0          0         11         11       2204       2204          0          0          0      33538
  6   41056242     264471          0          0         55         55       5126       5126          0     254109          0      47278
  7   42900129     273301          0          0         85         85       5458       5458          0     262215          0      14747
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.305s (  774.730 blocks/s,  48.421 Mb/s)
  264   Fri Apr 23 07:09:57 2021 LJWFriday 23rd April 08:00-12:00

08:30

System Checks

 

Clock Status error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status error:


         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

 

Statistics - See attacnment 1

Temperatures - See attachment 2

Bias & Leakage Currents - See attachment 3

 

10:02

System Checks:

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration Error:

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 4

Temperatures - See attachment 5

Bias & Leakage Currents - See attachment 6

 

10:45

System Checks

Clock Status Error:


FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


ADC Calibration check Error:


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 7

Temperatures - See attachment 8

Bias & Leakage Currents - See attachment 9

 

Attachment 1: 2304211_1.png
2304211_1.png
Attachment 2: 230421_2.png
230421_2.png
Attachment 3: 230421_3.png
230421_3.png
Attachment 4: 230421_4.png
230421_4.png
Attachment 5: 230421_5.png
230421_5.png
Attachment 6: 230421_6.png
230421_6.png
Attachment 7: 230421_7.png
230421_7.png
Attachment 8: 230421_8.png
230421_8.png
Attachment 9: 230421_9.png
230421_9.png
  263   Thu Apr 22 23:17:46 2021 DJ Friday 23rd April 00:00-08:00

Took over from TD ... All is well.

00:45 System Wide Checks

Clock: 12 passed, 0 failed.

ADC Calibration: Passed 12, 0 failed.

White Rabbit Decoder: Passed 4,failed 8. - See screenshot.

FPGA Passed 11, failed 1. - See screenshot.

FEE64 Linux: See screenshot.

-----

02:49 run closed. Checking beam.

03:00 File 55 opened.

03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.

File 56 opened at 4:24.

---------

04:30 System Wide Checks (See screenshots)

White Rabbit Decoder: Passed 12, failed 0.

FPGA: page not loading properly.

----

05:37 System wide checks. See screenshots

Clock: Status: 11 passed. 1 fail.

ADC calibration: Passed 9, failed 3.

White Rabbit decoder: Passed 12, failed 0.

-----

07:32 System wide checks (See screenshots).

 

 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi.png
Screenshot_2021-04-23_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot-CAEN_HV_Supply.png
Screenshot-CAEN_HV_Supply.png
Attachment 4: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Attachment 5: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 60.png
60.png
Attachment 10: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Attachment 12: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Attachment 13: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Attachment 17: 61.png
61.png
Attachment 18: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Attachment 19: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Attachment 20: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Attachment 21: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 22: Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Attachment 23: 63.png
63.png
Attachment 24: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Attachment 25: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Attachment 26: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Attachment 27: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
  268   Fri Apr 23 20:19:52 2021 TDFriday 23 April 20.00-00.00
21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378
Attachment 1: 201.png
201.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Attachment 4: Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Attachment 5: 202.png
202.png
Attachment 6: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
  266   Fri Apr 23 13:13:46 2021 MA, TDFriday 23 April 12.00-16.00
14.15 System wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors - reports 'server internal error'

14.16 DSSSD bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachment 2
      good events stats - attachment 3

14.22 DAQ continues file S460/R51_205
      beam intensity 2.2e+09/s


15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6


Clock check

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC check

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



White Rabbit


		 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




FPGA
Loge page error!

Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      6      2      1      2      3      2      4      2      3      6   : 36044
aida02 :     15      3      2      2      0      3      2      3      2      3      4   : 27188
aida03 :     29      4      0      3      2      4      2      2      3      3      6   : 36212
aida04 :      4      6      4      4      2      3      3      3      3      3      6   : 36864
aida05 :     19      5      4      2      3      3      1      3      3      3      6   : 36404
aida06 :     24      9      5      2      3      3      3      2      3      3      6   : 36472
aida07 :     20      4      9      0      2      3      2      2      3      3      6   : 36096
aida08 :     22      9      4      3      1      3      1      3      3      3      6   : 36352
aida09 :     20      7      3      2      2      3      3      2      3      3      6   : 36344
aida10 :     25      9      9      5      2      3      2      3      3      3      6   : 36828
aida11 :     24     11      2      0      1      3      1      3      3      3      6   : 36248
aida12 :     26      5      4      2      1      3      2      2      3      3      6   : 36048
Attachment 1: 200.png
200.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Attachment 4: 210423_1550_Voltages.png
210423_1550_Voltages.png
Attachment 5: 210423_1550_Stats.png
210423_1550_Stats.png
Attachment 6: 210423_1550_Temp.png
210423_1550_Temp.png
  549   Fri Mar 22 08:31:39 2024 TDFriday 22 March
09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39
Attachment 1: Screenshot_from_2024-03-22_09-32-58.png
Screenshot_from_2024-03-22_09-32-58.png
Attachment 2: Screenshot_from_2024-03-22_09-34-54.png
Screenshot_from_2024-03-22_09-34-54.png
Attachment 3: Screenshot_from_2024-03-22_09-39-03.png
Screenshot_from_2024-03-22_09-39-03.png
Attachment 4: Screenshot_from_2024-03-22_09-40-10.png
Screenshot_from_2024-03-22_09-40-10.png
Attachment 5: Screenshot_from_2024-03-22_09-47-26.png
Screenshot_from_2024-03-22_09-47-26.png
Attachment 6: Screenshot_from_2024-03-22_09-49-33.png
Screenshot_from_2024-03-22_09-49-33.png
Attachment 7: Screenshot_from_2024-03-22_09-49-00.png
Screenshot_from_2024-03-22_09-49-00.png
Attachment 8: Screenshot_from_2024-03-22_09-48-25.png
Screenshot_from_2024-03-22_09-48-25.png
Attachment 9: Screenshot_from_2024-03-22_09-47-55.png
Screenshot_from_2024-03-22_09-47-55.png
Attachment 10: Screenshot_from_2024-03-22_10-00-47.png
Screenshot_from_2024-03-22_10-00-47.png
Attachment 11: Screenshot_from_2024-03-22_10-00-01.png
Screenshot_from_2024-03-22_10-00-01.png
Attachment 12: Screenshot_from_2024-03-22_09-58-56.png
Screenshot_from_2024-03-22_09-58-56.png
Attachment 13: Screenshot_from_2024-03-22_09-57-50.png
Screenshot_from_2024-03-22_09-57-50.png
Attachment 14: Screenshot_from_2024-03-22_09-56-45.png
Screenshot_from_2024-03-22_09-56-45.png
Attachment 15: Screenshot_from_2024-03-22_09-55-52.png
Screenshot_from_2024-03-22_09-55-52.png
Attachment 16: Screenshot_from_2024-03-22_09-54-18.png
Screenshot_from_2024-03-22_09-54-18.png
Attachment 17: Screenshot_from_2024-03-22_09-52-45.png
Screenshot_from_2024-03-22_09-52-45.png
Attachment 18: Screenshot_from_2024-03-22_09-52-13.png
Screenshot_from_2024-03-22_09-52-13.png
Attachment 19: Screenshot_from_2024-03-22_09-51-47.png
Screenshot_from_2024-03-22_09-51-47.png
Attachment 20: Screenshot_from_2024-03-22_10-04-22.png
Screenshot_from_2024-03-22_10-04-22.png
Attachment 21: Screenshot_from_2024-03-22_10-03-53.png
Screenshot_from_2024-03-22_10-03-53.png
Attachment 22: Screenshot_from_2024-03-22_14-29-18.png
Screenshot_from_2024-03-22_14-29-18.png
Attachment 23: Screenshot_from_2024-03-22_14-30-00.png
Screenshot_from_2024-03-22_14-30-00.png
Attachment 24: Screenshot_from_2024-03-22_14-31-19.png
Screenshot_from_2024-03-22_14-31-19.png
Attachment 25: Screenshot_from_2024-03-22_14-31-01.png
Screenshot_from_2024-03-22_14-31-01.png
Attachment 26: Screenshot_from_2024-03-22_14-31-57.png
Screenshot_from_2024-03-22_14-31-57.png
Attachment 27: Screenshot_from_2024-03-22_14-32-35.png
Screenshot_from_2024-03-22_14-32-35.png
Attachment 28: Screenshot_from_2024-03-22_14-40-04.png
Screenshot_from_2024-03-22_14-40-04.png
Attachment 29: Screenshot_from_2024-03-22_14-39-30.png
Screenshot_from_2024-03-22_14-39-30.png
Attachment 30: Screenshot_from_2024-03-22_14-38-22.png
Screenshot_from_2024-03-22_14-38-22.png
Attachment 31: Screenshot_from_2024-03-22_14-37-19.png
Screenshot_from_2024-03-22_14-37-19.png
Attachment 32: Screenshot_from_2024-03-22_14-36-28.png
Screenshot_from_2024-03-22_14-36-28.png
Attachment 33: Screenshot_from_2024-03-22_14-35-23.png
Screenshot_from_2024-03-22_14-35-23.png
Attachment 34: Screenshot_from_2024-03-22_14-34-38.png
Screenshot_from_2024-03-22_14-34-38.png
Attachment 35: Screenshot_from_2024-03-22_14-33-55.png
Screenshot_from_2024-03-22_14-33-55.png
Attachment 36: Screenshot_from_2024-03-22_14-41-21.png
Screenshot_from_2024-03-22_14-41-21.png
Attachment 37: Screenshot_from_2024-03-22_14-40-57.png
Screenshot_from_2024-03-22_14-40-57.png
Attachment 38: Capture_2.PNG
Capture_2.PNG
Attachment 39: Capture_3.PNG
Capture_3.PNG
  342   Fri May 21 15:03:48 2021 CAFriday 21st May 16:00 - 00:00 shift
16:00 CA takes over
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

16:14 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x570 : 	 112  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

16:17 temperatures ok - attachment 1
      statistics ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      analysis of R14_1625 - FEE7 deadtime at 4.5%

18:06 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x571 : 	 113  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:07 temperatures ok - attachment 4
      statistics ok - attachment 5
      detector bias / leakage currents ok - attachment 6
      analysis of R14_1662 - FEE7 deadtime at 4.3%

19:07 DAQ continues ok - writing to file R14_1682

20:07 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

20:10 temperatures ok - attachment 7
      statistics ok - attachment 8
      detector bias / leakage currents ok - attachment 9
      analysis of R14_1703 - max deadtime at 4.0% - attachment 10

20:35 rate spectra - attachment 11
      HEC spectra - attachment 12 & 13

20:36 all histograms zeroed

20:37 DAQ continues ok - writing to file R14_1712

20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.

      ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files

21:12 no beam - writing to file R14_1723

21:17 beam is back

22:07 system wide checks all ok - WR decoder fault same as entry at 20:07

22:08 temperatures ok - attachment 14
      statistics ok - attachment 15
      detector bias / leakage currents ok - attachment 16
      analysis of R14_1740 - max deadtime at 4.2% - attachment 17








   
Attachment 1: Screenshot_from_2021-05-21_15-14-59.png
Screenshot_from_2021-05-21_15-14-59.png
Attachment 2: Screenshot_from_2021-05-21_15-15-38.png
Screenshot_from_2021-05-21_15-15-38.png
Attachment 3: Screenshot_from_2021-05-21_15-15-54.png
Screenshot_from_2021-05-21_15-15-54.png
Attachment 4: Screenshot_from_2021-05-21_17-06-56.png
Screenshot_from_2021-05-21_17-06-56.png
Attachment 5: Screenshot_from_2021-05-21_17-07-18.png
Screenshot_from_2021-05-21_17-07-18.png
Attachment 6: Screenshot_from_2021-05-21_17-07-47.png
Screenshot_from_2021-05-21_17-07-47.png
Attachment 7: Screenshot_from_2021-05-21_19-09-19.png
Screenshot_from_2021-05-21_19-09-19.png
Attachment 8: Screenshot_from_2021-05-21_19-09-59.png
Screenshot_from_2021-05-21_19-09-59.png
Attachment 9: Screenshot_from_2021-05-21_19-10-15.png
Screenshot_from_2021-05-21_19-10-15.png
Attachment 10: Screenshot_from_2021-05-21_19-14-45.png
Screenshot_from_2021-05-21_19-14-45.png
Attachment 11: Screenshot_from_2021-05-21_19-25-51.png
Screenshot_from_2021-05-21_19-25-51.png
Attachment 12: Screenshot_from_2021-05-21_19-30-14.png
Screenshot_from_2021-05-21_19-30-14.png
Attachment 13: Screenshot_from_2021-05-21_19-32-13.png
Screenshot_from_2021-05-21_19-32-13.png
Attachment 14: Screenshot_from_2021-05-21_21-08-28.png
Screenshot_from_2021-05-21_21-08-28.png
Attachment 15: Screenshot_from_2021-05-21_21-08-51.png
Screenshot_from_2021-05-21_21-08-51.png
Attachment 16: Screenshot_from_2021-05-21_21-09-06.png
Screenshot_from_2021-05-21_21-09-06.png
Attachment 17: Screenshot_from_2021-05-21_21-12-03.png
Screenshot_from_2021-05-21_21-12-03.png
  341   Fri May 21 07:03:50 2021 OH, TDFriday 21 May 08:00-16:00
08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png
210521_0829_Stats.png
Attachment 2: 210521_0829_Temp.png
210521_0829_Temp.png
Attachment 3: 210521_0830_Bias.png
210521_0830_Bias.png
Attachment 4: 210521_0957.png
210521_0957.png
Attachment 5: 210521_0957_Temp.png
210521_0957_Temp.png
Attachment 6: 210521_0958_Bias.png
210521_0958_Bias.png
Attachment 7: 210521_1232_Stats.png
210521_1232_Stats.png
Attachment 8: 210521_1233_Bias.png
210521_1233_Bias.png
Attachment 9: 210521_1233_Temp.png
210521_1233_Temp.png
  130   Fri Feb 21 09:39:38 2020 TD, NHFriday 21 February
Attachment 1 - layout 2 - stat spectra
Attachment 2 - detector biases & leakage currents - OK
Attachment 3 - FEE64 temperatures - OK
Attachments 4-7 - TapeServer, Merger, Merger Statistics, good event statistics - OK

10:40 ASIC check load x2


13.12 DAQ stopped c. 10.50
      unable to connect messages for multiple FEE64s

      After lunch powwer-cycled FEE64s and restarted DAQ
      aida03 crashed (see https://elog.ph.ed.ac.uk/DESPEC/131 ) and rebooted by issuing reboot command as root

      Data file 20FEB20/R1
      ASIC settings 2019Oct31-13.24.23
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      continue alpha background run
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
  468   Fri May 20 16:22:01 2022 TDFriday 20 May
17.20 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-20_17-19-21.png
Screenshot_from_2022-05-20_17-19-21.png
Attachment 2: Screenshot_from_2022-05-20_17-19-35.png
Screenshot_from_2022-05-20_17-19-35.png
Attachment 3: Screenshot_from_2022-05-20_17-20-22.png
Screenshot_from_2022-05-20_17-20-22.png
  119   Fri Dec 20 12:04:03 2019 NH, TDFriday 20 December 2019
Fig 1: DSSD Leakage currents ok

Fig 2: FEE Temps ok

Fig 3: aida08 has a few timestamp errors in merger

Fig 4, 5: GSI WR OK

Fig 6: All System wide checks passed 

FIg 7: Stats good

13:11 - Alpha Run Stopped, Final File: R9_6

FEEs powered off, DSSD bias off.
Attachment 1: Screenshot-CAEN_HV_Control.png
Screenshot-CAEN_HV_Control.png
Attachment 2: FeeTemps.png
FeeTemps.png
Attachment 3: mergerTSerrors.png
mergerTSerrors.png
Attachment 4: gsiwr1.png
gsiwr1.png
Attachment 5: gsiwr2.png
gsiwr2.png
Attachment 6: checks.png
checks.png
Attachment 7: stats.png
stats.png
  80   Fri Nov 1 10:46:13 2019 CA, TD, NHFriday 1st November 2019
11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding
Attachment 1: biaswfoil.png
biaswfoil.png
Attachment 2: statswfoil.png
statswfoil.png
Attachment 3: tempwfoil.png
tempwfoil.png
Attachment 4: biaswofoil.png
biaswofoil.png
Attachment 5: statswofoil.png
statswofoil.png
Attachment 6: tempswofoil.png
tempswofoil.png
  205   Fri Mar 19 17:16:45 2021 TDFriday 19 March
Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04.
Attachment 1: Screenshot_2021-03-19_Options_aidas-gsi.png
Screenshot_2021-03-19_Options_aidas-gsi.png
Attachment 2: Screenshot_2021-03-19_Options_aidas-gsi(4).png
Screenshot_2021-03-19_Options_aidas-gsi(4).png
Attachment 3: Screenshot_2021-03-19_Options_aidas-gsi(1).png
Screenshot_2021-03-19_Options_aidas-gsi(1).png
Attachment 4: Screenshot_2021-03-19_Options_aidas-gsi(5).png
Screenshot_2021-03-19_Options_aidas-gsi(5).png
Attachment 5: Screenshot_2021-03-19_Options_aidas-gsi(2).png
Screenshot_2021-03-19_Options_aidas-gsi(2).png
Attachment 6: Screenshot_2021-03-19_Options_aidas-gsi(6).png
Screenshot_2021-03-19_Options_aidas-gsi(6).png
Attachment 7: Screenshot_2021-03-19_Options_aidas-gsi(3).png
Screenshot_2021-03-19_Options_aidas-gsi(3).png
Attachment 8: Screenshot_2021-03-19_Options_aidas-gsi(7).png
Screenshot_2021-03-19_Options_aidas-gsi(7).png
  580   Fri Apr 19 13:52:34 2024 JB, CC, TDFriday 19 April contd.
12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3

      During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
      aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )

      AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
       *except* aida02 ASIC temp - known fault

      System wide checks OK *except* WR/FPGA errors - attachments 3-4

      WR timestamps OK - attachment 5

      ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
       ADC data items 10/16 < 20k, max 143k

      per FEE64 Rate spectra - attachment 11

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-13


15.10 Synchronise ASIC clocks
      Re-calibrate ALL ADCs 

      ASIC settings 2024Mar27-11.25.32

      Changes ( https://elog.ph.ed.ac.uk/CARME/499 )

       IBias LF feedback from 0xf to 0x8
       Diode link threshold from 0xbf to 0xca

      New ASIC settings saved as 2024Apr19-15.22.49

      All FEE64 slow comparators -> 0x14

      Data file S100_alpha/R12

      Pulser ealkthrough (test +)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity +
      Tail pulse
      tau_d 1ms

16.35 DAQ ends OK file S100_alpha/R12_24

      per p+n FEE64 1.8.L spectra - attachment 14
       aida09 pulser peak width 54 ch FWHM 

17.38 Data file S100_alpha/R13

      Pulser walkthrough (test -)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity -
      Tail pulse
      tau_d 1ms

18.43 DAQ ends OK file S100_alpha/R13_20

      per p+n FEE64 1.8.L spectra - attachment 15

18.55 Current status

      FEE64 power ON
      DSSSD bias ON

      All FEE64 slow comparator 0x14

      DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )


To Do

- S4 currently closed/controlled access for FRS startup

If we have access to S4 tomorrow

- switch test - to test + ( not critical )
- test AIDA interlock 
- further alpha background?



22.47 DSSSD bias & leakage current OK - attachments 16-17

      FEE64 temps OK - attachment 18
       *except* aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Data link, Tape Server & Merger - attachments 21-23

       
Attachment 1: Screenshot_from_2024-04-19_14-56-51.png
Screenshot_from_2024-04-19_14-56-51.png
Attachment 2: Screenshot_from_2024-04-19_14-57-04.png
Screenshot_from_2024-04-19_14-57-04.png
Attachment 3: Screenshot_from_2024-04-19_14-58-27.png
Screenshot_from_2024-04-19_14-58-27.png
Attachment 4: Screenshot_from_2024-04-19_14-58-23.png
Screenshot_from_2024-04-19_14-58-23.png
Attachment 5: Screenshot_from_2024-04-19_14-59-02.png
Screenshot_from_2024-04-19_14-59-02.png
Attachment 6: Screenshot_from_2024-04-19_15-01-31.png
Screenshot_from_2024-04-19_15-01-31.png
Attachment 7: Screenshot_from_2024-04-19_15-01-17.png
Screenshot_from_2024-04-19_15-01-17.png
Attachment 8: Screenshot_from_2024-04-19_15-01-02.png
Screenshot_from_2024-04-19_15-01-02.png
Attachment 9: Screenshot_from_2024-04-19_15-00-39.png
Screenshot_from_2024-04-19_15-00-39.png
Attachment 10: Screenshot_from_2024-04-19_14-59-51.png
Screenshot_from_2024-04-19_14-59-51.png
Attachment 11: Screenshot_from_2024-04-19_15-02-09.png
Screenshot_from_2024-04-19_15-02-09.png
Attachment 12: Screenshot_from_2024-04-19_15-04-36.png
Screenshot_from_2024-04-19_15-04-36.png
Attachment 13: Screenshot_from_2024-04-19_15-03-48.png
Screenshot_from_2024-04-19_15-03-48.png
Attachment 14: Screenshot_from_2024-04-19_16-56-36.png
Screenshot_from_2024-04-19_16-56-36.png
Attachment 15: Screenshot_from_2024-04-19_18-45-05.png
Screenshot_from_2024-04-19_18-45-05.png
Attachment 16: Screenshot_from_2024-04-19_22-47-43.png
Screenshot_from_2024-04-19_22-47-43.png
Attachment 17: Screenshot_from_2024-04-19_22-48-52.png
Screenshot_from_2024-04-19_22-48-52.png
Attachment 18: Screenshot_from_2024-04-19_22-49-31.png
Screenshot_from_2024-04-19_22-49-31.png
Attachment 19: Screenshot_from_2024-04-19_22-50-15.png
Screenshot_from_2024-04-19_22-50-15.png
Attachment 20: Screenshot_from_2024-04-19_22-51-23.png
Screenshot_from_2024-04-19_22-51-23.png
Attachment 21: Screenshot_from_2024-04-19_22-52-39.png
Screenshot_from_2024-04-19_22-52-39.png
Attachment 22: Screenshot_from_2024-04-19_22-52-14.png
Screenshot_from_2024-04-19_22-52-14.png
Attachment 23: Screenshot_from_2024-04-19_22-51-55.png
Screenshot_from_2024-04-19_22-51-55.png
  579   Fri Apr 19 02:35:14 2024 TDFriday 19 April
03.30


09.10



09.25 data file S100_alpha/R11
   
      BNC PB-5 pulser data

      amplitude 10.0V
      attenuation x10
      tau_d = 1ms
      polarity - ( initially + )
      frequency 25Hz

      ASIC settings2024Mar27-11.25.32
       slow comparator 0x64 all FEE64s


      aida02 pulser peak width 129 ch FWHM

      polsrity +

      aida09 pulser peak width 57 ch FWHM


09.40 switch to no storage

      p+n FEE64 slow comparators 0x64 -> 0xa
      n+n FEE64 slow comparators 0x64 -> 0xf

      radioactive source 152Eu 370kBq installed on the bottom side of the snout, centred tranversely, located beneath bplas-DSSSD-DSSSD-bplas position.

      ADC data item stats - attachment 35
      9/16 < 20k, max 108k

      per FEE64 Rate spectra - attachment 36

10.44 152Eu source removed

      ADC data item stats - attachment 37
      9/16 < 20k, max 102k

      per FEE64 Rate spectra - attachment 38
Attachment 1: Screenshot_from_2024-04-19_03-31-31.png
Screenshot_from_2024-04-19_03-31-31.png
Attachment 2: Screenshot_from_2024-04-19_03-30-58.png
Screenshot_from_2024-04-19_03-30-58.png
Attachment 3: Screenshot_from_2024-04-19_03-30-45.png
Screenshot_from_2024-04-19_03-30-45.png
Attachment 4: Screenshot_from_2024-04-19_03-27-45.png
Screenshot_from_2024-04-19_03-27-45.png
Attachment 5: Screenshot_from_2024-04-19_03-26-26.png
Screenshot_from_2024-04-19_03-26-26.png
Attachment 6: Screenshot_from_2024-04-19_03-25-53.png
Screenshot_from_2024-04-19_03-25-53.png
Attachment 7: Screenshot_from_2024-04-19_03-24-58.png
Screenshot_from_2024-04-19_03-24-58.png
Attachment 8: Screenshot_from_2024-04-19_09-08-46.png
Screenshot_from_2024-04-19_09-08-46.png
Attachment 9: Screenshot_from_2024-04-19_09-09-19.png
Screenshot_from_2024-04-19_09-09-19.png
Attachment 10: Screenshot_from_2024-04-19_09-10-19.png
Screenshot_from_2024-04-19_09-10-19.png
Attachment 11: Screenshot_from_2024-04-19_09-10-10.png
Screenshot_from_2024-04-19_09-10-10.png
Attachment 12: Screenshot_from_2024-04-19_09-10-03.png
Screenshot_from_2024-04-19_09-10-03.png
Attachment 13: Screenshot_from_2024-04-19_09-10-51.png
Screenshot_from_2024-04-19_09-10-51.png
Attachment 14: Screenshot_from_2024-04-19_09-11-08.png
Screenshot_from_2024-04-19_09-11-08.png
Attachment 15: Screenshot_from_2024-04-19_09-10-59.png
Screenshot_from_2024-04-19_09-10-59.png
Attachment 16: Screenshot_from_2024-04-19_09-11-20.png
Screenshot_from_2024-04-19_09-11-20.png
Attachment 17: Screenshot_from_2024-04-19_09-11-46.png
Screenshot_from_2024-04-19_09-11-46.png
Attachment 18: Screenshot_from_2024-04-19_09-11-31.png
Screenshot_from_2024-04-19_09-11-31.png
Attachment 19: Screenshot_from_2024-04-19_09-17-48.png
Screenshot_from_2024-04-19_09-17-48.png
Attachment 20: Screenshot_from_2024-04-19_09-17-22.png
Screenshot_from_2024-04-19_09-17-22.png
Attachment 21: Screenshot_from_2024-04-19_09-14-14.png
Screenshot_from_2024-04-19_09-14-14.png
Attachment 22: Screenshot_from_2024-04-19_09-13-23.png
Screenshot_from_2024-04-19_09-13-23.png
Attachment 23: Screenshot_from_2024-04-19_09-12-08.png
Screenshot_from_2024-04-19_09-12-08.png
Attachment 24: Screenshot_from_2024-04-19_09-12-01.png
Screenshot_from_2024-04-19_09-12-01.png
Attachment 25: Screenshot_from_2024-04-19_09-11-54.png
Screenshot_from_2024-04-19_09-11-54.png
Attachment 26: Screenshot_from_2024-04-19_09-18-40.png
Screenshot_from_2024-04-19_09-18-40.png
Attachment 27: Screenshot_from_2024-04-19_09-18-22.png
Screenshot_from_2024-04-19_09-18-22.png
Attachment 28: Screenshot_from_2024-04-19_09-18-01.png
Screenshot_from_2024-04-19_09-18-01.png
Attachment 29: Screenshot_from_2024-04-19_09-29-33.png
Screenshot_from_2024-04-19_09-29-33.png
Attachment 30: Screenshot_from_2024-04-19_09-27-40.png
Screenshot_from_2024-04-19_09-27-40.png
Attachment 31: Screenshot_from_2024-04-19_09-42-01.png
Screenshot_from_2024-04-19_09-42-01.png
Attachment 32: Screenshot_from_2024-04-19_09-41-31.png
Screenshot_from_2024-04-19_09-41-31.png
Attachment 33: Screenshot_from_2024-04-19_09-40-02.png
Screenshot_from_2024-04-19_09-40-02.png
Attachment 34: Screenshot_from_2024-04-19_09-37-03.png
Screenshot_from_2024-04-19_09-37-03.png
Attachment 35: Screenshot_from_2024-04-19_09-46-35.png
Screenshot_from_2024-04-19_09-46-35.png
Attachment 36: Screenshot_from_2024-04-19_09-47-08.png
Screenshot_from_2024-04-19_09-47-08.png
Attachment 37: Screenshot_from_2024-04-19_10-44-19.png
Screenshot_from_2024-04-19_10-44-19.png
Attachment 38: Screenshot_from_2024-04-19_10-43-58.png
Screenshot_from_2024-04-19_10-43-58.png
  482   Sat Jun 18 09:46:57 2022 OH, NHFriday 17th June
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23
Attachment 1: 220617_1322_stats.png
220617_1322_stats.png
Attachment 2: 220617_1420_stats.png
220617_1420_stats.png
Attachment 3: 220617_11428_stats.png
220617_11428_stats.png
Attachment 4: 220617_1428_layout1.png
220617_1428_layout1.png
Attachment 5: 220617_1440_stats_waveform_on.png
220617_1440_stats_waveform_on.png
Attachment 6: 220617_1444_stats_no_waveform.png
220617_1444_stats_no_waveform.png
Attachment 7: 220617_1459_stats_waveform_on.png
220617_1459_stats_waveform_on.png
Attachment 8: 220617_1517_stats_no_waveform.png
220617_1517_stats_no_waveform.png
Attachment 9: 220617_1624_stats_waveform_on.png
220617_1624_stats_waveform_on.png
Attachment 10: 220617_1626_stats_waveform_off.png
220617_1626_stats_waveform_off.png
Attachment 11: 220617_1649_stats_waveform_on.png
220617_1649_stats_waveform_on.png
Attachment 12: 220617_1652_no_waveform.png
220617_1652_no_waveform.png
Attachment 13: 220617_1709_waveform_on.png
220617_1709_waveform_on.png
Attachment 14: 220617_1724_stats_no_waveform.png
220617_1724_stats_no_waveform.png
Attachment 15: 220617_1729_stats_waveform_on.png
220617_1729_stats_waveform_on.png
Attachment 16: 220617_1733_stats_no_waveform.png
220617_1733_stats_no_waveform.png
Attachment 17: 220617_1736_0x21.png
220617_1736_0x21.png
Attachment 18: 220617_1738_0x14.png
220617_1738_0x14.png
Attachment 19: 220617_1738_0x12.png
220617_1738_0x12.png
Attachment 20: 220617_1807_stats_waveform_on.png
220617_1807_stats_waveform_on.png
Attachment 21: 220617_1832_stats_no_waveform.png
220617_1832_stats_no_waveform.png
Attachment 22: 220617_1844_0xf.png
220617_1844_0xf.png
Attachment 23: 220617_1845_stats_waveform_on.png
220617_1845_stats_waveform_on.png
  229   Fri Apr 16 07:19:00 2021 OH, MAFriday 16th April 08:00-16:00
08:19 System wide checks all ok
      Baselines reset on WR and FPGA checks for the start of the day

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3

09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
      No corruptions have been noted since

10:10 System check, clcock and ADC ok

	 Base 		Current 	Difference
aida07 fault 	 0xfb3a : 	 0xfb3d : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA and memory check ok

      Statistics ok - attachment 4
      Temperatures ok - attachment 5
      Bias and leakage currents ok - attachment 6

10:39 Fission fragments were punching through AIDA
      This was even with the maxmimum degraders in place
      They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.

11:38 They have found the issue with the degrader and have corrected it.
      We now see next to no implants in AIDA as expect.
      Histograms reset at around 11:35
      Layout 2 shows very few implants in 5 minutes - attachment 7

12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
      This was a large amount of dose going into the detectors which we should try to avoid

13:25 system check, clock and ADC ok

	 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics ok - attachment 9
      Temperatures ok - attachment 10
      Bias and leakage currents ok - attachment 11
FPGA and memory check ok


16:10 system check, clock and ADC ok

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


      Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14
FPGA and memory check ok
Attachment 1: 210416_0820_Stats.png
210416_0820_Stats.png
Attachment 2: 210416_0821_Temp.png
210416_0821_Temp.png
Attachment 3: 210416_0824_Bias.png
210416_0824_Bias.png
Attachment 4: 210416_1003Stats.png
210416_1003Stats.png
Attachment 5: 210416_1007Temp.png
210416_1007Temp.png
Attachment 6: 210416_1019_voltages.png
210416_1019_voltages.png
Attachment 7: 210416_1139_layout2.png
210416_1139_layout2.png
Attachment 8: 210416_1207_Leakage.png
210416_1207_Leakage.png
Attachment 9: 210416_1311_Stats.png
210416_1311_Stats.png
Attachment 10: 210416_1312_Temp.png
210416_1312_Temp.png
Attachment 11: 210416_1315_Voltages.png
210416_1315_Voltages.png
Attachment 12: 210416_1412_Voltages.png
210416_1412_Voltages.png
Attachment 13: 210416_1548_Stats.png
210416_1548_Stats.png
Attachment 14: 210416_1553_Temp.png
210416_1553_Temp.png
  319   Fri May 14 19:01:09 2021 JSFriday 14th May 20:00-00:00
20:00 Taking over from CB

20:33 Doing full checks.
Temps ok elog:319/8  
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

FPGA Timestamp errors -
		 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     34     30      8      1      4      1      2      2      3      6   : 35772
aida02 :     37     30     28      5      3      1      2      4      2      3      6   : 36644
aida03 :     41     29     30     16      4      3      1      5      1      3      6   : 36588
aida04 :     13      6     23     12      4      4      4      4      2      2      6   : 35412
aida05 :     20     32     23     10      2      2      2      2      1      4      6   : 36736
aida06 :     14      5     10      5      1      3      2      3      2      3      6   : 35680
aida07 :     44     23     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     39     30     28      9      0      3      2      3      3      2      6   : 35308
aida09 :     41     31     28     10      2      5      1      3      1      3      6   : 35484
aida10 :     43     23     27      7      3      1      1      3      3      3      6   : 36916
aida11 :     39     35     17     10      3      3      2      2      2      3      6   : 35908
aida12 :     37     36     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     47     32     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     23     33     14     10      3      4      3      4      3      2      6   : 36164
aida15 :     44     28     24     12      3      2      4      3      3      2      6   : 35920
aida16 :     22     17     24     17      3      4      2      3      1      3      6   : 35648

Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27 
Resume info #3 - elog:319/3 same as elog:317/26 
Pause info #2 - elog:319/4 same as elog:317/32 
AIDA disk info #6 - elog:319/5 same as elog:317/31 
AIDA ADC data items - elog:319/6 same as elog:317/30 
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see 
clearly once uploaded the screenshots] 

Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%

21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay

21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     39     34     29      8      1      4      1      2      2      3      6   : 35772
aida02 :     46     29     28      5      3      2      2      3      2      3      6   : 36288
aida03 :     39     25     29     16      4      2      2      5      1      3      6   : 36660
aida04 :     14      6     19     12      4      4      3      4      2      2      6   : 35096
aida05 :     20     30     24     10      2      2      2      2      1      4      6   : 36736
aida06 :     26      4     11      5      2      4      3      2      2      3      6   : 35672
aida07 :     46     22     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     28     32     28      9      0      2      2      4      3      2      6   : 35664
aida09 :     32     30     29     10      3      5      2      3      1      3      6   : 35776
aida10 :     39     23     25      7      3      1      1      3      3      3      6   : 36868
aida11 :     40     31     21     10      3      3      1      4      1      3      6   : 35688
aida12 :     17     31     12     10      2      2      3      4      3      2      6   : 35772
aida13 :     35     28     22      7      4      2      3      4      3      2      6   : 36012
aida14 :     30     33     14     10      3      4      3      4      3      2      6   : 36192
aida15 :     34     26     24     12      3      2      4      3      3      2      6   : 35864
aida16 :     38     24     27     17      3      4      2      3      1      3      6   : 35816

Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12 
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14 
Aida Correlation Info #8 - elog:319/15  - rates showed zero, checked again and normal elog:319/17

Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%

22:00
Temps ok
Stats ok 
ucesb ok

22:35 
Temps ok
Stats ok 
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up 
again elog:319/17

22:54 They are closing the file, we are on R5_155

23:00 
Attachment 1: bias1.png
bias1.png
Attachment 2: Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Attachment 3: Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Attachment 4: Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Attachment 5: Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Attachment 6: Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Attachment 7: Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Attachment 8: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: R5_108_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256861958 ( 1503007.9 Hz)
        Other data format:    5058042 (   29596.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        150 (       0.9 Hz)
                              SYNC100:      32636 (     191.0 Hz)
                              WR48-63:      32636 (     191.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4992471 (   29213.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     207149 (    1212.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      170.899 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                4.459               72.353
  2                1.217               86.117
  3                3.357               28.177
  4                0.000              160.970
  5                0.495               76.386
  6                0.216               93.617
  7                1.418               40.259
  8                3.253               44.571
  9                0.000              168.837
 10                5.250                9.058
 11                1.956               49.014
 12                0.182               82.483
 13                0.000              160.585
 14                0.143               88.740
 15                3.053               35.456
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8464368       3303          0          0          2          2       1138       1138          0       1023          0      11424
  1   16399642       4233          0          0         17         18       2099       2099          0          0          0      22505
  2   13045073    1576129          0          0          6          6       1793       1793          0    1572531          0      18179
  3   26563169    1558422          0          0         14         14       3487       3487          0    1551420          0      26353
  4    2213090     510296          0          0          0          0        328        328          0     509640          0       5100
  5   15498485     314714          0          0          4          4       1986       1986          0     310734          0      15610
  6   12553775     609520          0          0          7          7       1573       1573          0     606360          0      10360
  7   23384876     446841          0          0         10         10       3029       3029          0     440763          0      18175
  8   21832531       5542          0          0         15         15       2756       2756          0          0          0      22662
  9    1045474        278          0          0          0          0        139        139          0          0          0       1866
 10   42357744      10552          0          0         43         43       5233       5233          0          0          0      12639
 11   20606319       4998          0          0         11         11       2488       2488          0          0          0       4351
 12   14311836       3576          0          0          6          6       1782       1782          0          0          0      20348
 13    3016503        780          0          0          0          0        390        390          0          0          0       1508
 14   13047482       3248          0          0          4          4       1620       1620          0          0          0      12826
 15   22521591       5610          0          0         10         10       2795       2795          0          0          0       3243
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:74697.445s (    0.428 blocks/s,   0.027 Mb/s)
Attachment 10: Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Attachment 11: Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Attachment 12: Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Attachment 16: R5_128_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256841478 ( 1482693.9 Hz)
        Other data format:    5078522 (   29317.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        167 (       1.0 Hz)
                               RESUME:        166 (       1.0 Hz)
                              SYNC100:      32644 (     188.4 Hz)
                              WR48-63:      32644 (     188.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5012901 (   28938.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     205165 (    1184.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      173.226 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.621               76.080
  2                1.556               81.876
  3                3.719               27.628
  4                0.000              162.954
  5                0.710               80.266
  6                0.170               93.769
  7                2.714               44.740
  8                3.483               45.565
  9                0.000              167.005
 10                8.522               10.948
 11                2.181               50.562
 12                0.202               85.035
 13                0.000              161.510
 14                0.146               88.959
 15                1.812               38.597
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8468806       3273          0          0          7          7       1112       1112          0       1035          0      11260
  1   16364560       4360          0          0         13         13       2167       2167          0          0          0      23505
  2   13265609    1594590          0          0          6          6       1881       1881          0    1590816          0      18254
  3   26054686    1576337          0          0         14         14       3422       3422          0    1569465          0      26152
  4    2223261     507850          0          0          0          0        362        362          0     507126          0       4952
  5   15152054     312213          0          0          8          7       1938       1938          0     308322          0      15336
  6   12659379     606961          0          0          4          4       1612       1612          0     603729          0      10099
  7   22669666     438118          0          0         14         14       2841       2841          0     432408          0      17594
  8   22337729       5564          0          0         15         15       2767       2767          0          0          0      22524
  9    1020352        240          0          0          0          0        120        120          0          0          0       1807
 10   42596491      10676          0          0         59         59       5279       5279          0          0          0      12076
 11   20672887       5060          0          0         13         13       2517       2517          0          0          0       4379
 12   14335521       3608          0          0          5          5       1799       1799          0          0          0      20105
 13    2812573        690          0          0          0          0        345        345          0          0          0       1405
 14   13325633       3260          0          0          4          4       1626       1626          0          0          0      12554
 15   22882271       5722          0          0          5          5       2856       2856          0          0          0       3163
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:78055.344s (    0.410 blocks/s,   0.026 Mb/s)
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Attachment 18: ucesb.png
ucesb.png
Attachment 19: _~.png
_~.png
Attachment 20: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 21: Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Attachment 22: Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Attachment 23: Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Attachment 24: Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Attachment 25: Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Attachment 26: Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Attachment 27: R5_157_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256650793 ( 1485751.4 Hz)
        Other data format:    5269207 (   30503.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        130 (       0.8 Hz)
                               RESUME:        129 (       0.7 Hz)
                              SYNC100:      32643 (     189.0 Hz)
                              WR48-63:      32643 (     189.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5203662 (   30124.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     223116 (    1291.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      172.741 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.094               74.418
  2                0.927               76.576
  3                3.655               26.911
  4                0.000              161.752
  5                0.023               80.653
  6                0.215               91.583
  7                1.871               44.002
  8                1.038               39.676
  9                0.000              166.718
 10                8.591               10.747
 11                3.917               45.280
 12                0.214               88.240
 13                0.000              163.408
 14                0.280               88.718
 15                1.416               44.122
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8604841       3308          0          0          0          0       1138       1138          0       1032          0      12424
  1   16359820       4228          0          0         13         13       2101       2101          0          0          0      24974
  2   13547296    1610585          0          0          6          6       1930       1930          0    1606713          0      19730
  3   25981004    1589167          0          0         14         13       3491       3491          0    1582158          0      28914
  4    2316430     549243          0          0          0          0        351        351          0     548541          0       5271
  5   15208801     341870          0          0          1          1       1974       1974          0     337920          0      16457
  6   12807457     657341          0          0          5          5       1667       1667          0     653997          0      10925
  7   22697274     479019          0          0         10         10       2849       2849          0     473301          0      19398
  8   22587880       5594          0          0          5          5       2792       2792          0          0          0      24704
  9    1078064        266          0          0          0          0        133        133          0          0          0       1963
 10   42543713      10740          0          0         43         43       5327       5327          0          0          0      13313
 11   20362862       5182          0          0         14         14       2577       2577          0          0          0       4606
 12   13994455       3436          0          0          7          7       1711       1711          0          0          0      21818
 13    2731986        666          0          0          0          0        333        333          0          0          0       1621
 14   13145005       3222          0          0          8          8       1603       1603          0          0          0      13539
 15   22683905       5340          0          0          4          4       2666       2666          0          0          0       3459
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:83137.008s (    0.385 blocks/s,   0.024 Mb/s)
  317   Fri May 14 14:57:32 2021 CB + OH, TD, NHFriday 14th May 16:00-20:00
16:00 Shift taken over from CA

16:00 
Temperatures OK. Attach 1. 
Upon reloading aida11 first returns "No respose" for all values. 
Upon refreshing, aida11 responds but aida 9 returns "No response". 
Upon refreshing again, attach 1.

Bias OK. Attach 2.

Statistics OK. Attach 3-7

System-wide checks
Clock OK
ADC - all fail
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     32     28     25      7      1      4      1      3      2      3      6   : 36112
aida02 :     43     28     29      5      4      2      3      4      1      3      6   : 36092
aida03 :     39     26     28      5      4      3      2      5      1      3      6   : 36428
aida04 :      9      5     21      8      5      4      4      4      2      2      6   : 35292
aida05 :     46     32     26      7      2      2      2      2      2      3      6   : 35768
aida06 :      4      5     16      1      3      4      2      4      1      3      6   : 35352
aida07 :     47     21     25      5      4      3      1      4      1      3      6   : 35604
aida08 :     62     33     32      6      0      2      3      3      3      2      6   : 35520
aida09 :     33     25     28      7      3      5      2      4      1      3      6   : 36140
aida10 :     47     29     22      6      3      2      2      2      2      3      6   : 35716
aida11 :     39     30     29      6      2      3      1      3      2      3      6   : 36124
aida12 :     38     36     20      6      3      2      3      3      3      2      6   : 35448
aida13 :     42     29     24      3      4      2      3      3      3      2      6   : 35440
aida14 :     31     28     18      7      3      3      4      4      2      2      6   : 35228
aida15 :     48     25     30      5      3      2      4      4      2      2      6   : 35272
aida16 :     37     16     28      6      3      3      2      2      2      3      6   : 35796

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021



16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time. 
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
      Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20



17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17

System-wide checks
Clock - all pass
ADC - all pass
WR
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     56     35     27      8      1      4      1      2      2      3      6   : 35816
aida02 :     43     31     27      5      3      1      3      3      2      3      6   : 36404
aida03 :     45     28     28     16      4      2      2      4      1      3      6   : 36180
aida04 :      2      2     18     12      4      3      3      4      2      2      6   : 34872
aida05 :     28     28     24     10      2      2      2      2      1      4      6   : 36752
aida06 :     28     11      9      5      2      3      3      3      1      3      6   : 35064
aida07 :     52     24     23      9      4      3      1      2      2      3      6   : 35744
aida08 :     21     26     27      8      0      2      3      3      3      2      6   : 35284
aida09 :     40     32     26     10      3      5      1      3      1      3      6   : 35520
aida10 :     46     23     27      7      3      1      1      3      3      3      6   : 36928
aida11 :     49     31     20     10      2      2      1      2      2      3      6   : 35516
aida12 :     39     35     17      9      2      2      3      4      3      2      6   : 35940
aida13 :     40     32     20      7      4      2      3      4      3      2      6   : 36032
aida14 :     33     33     16     10      3      4      3      4      3      2      6   : 36236
aida15 :     53     27     25     12      3      2      4      4      2      2      6   : 35452
aida16 :     35     22     32     15      3      4      3      3      1      3      6   : 36060

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24

System-wide checks
Clock - all pass
ADC - all pass
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     31     31      8      1      4      1      2      2      3      6   : 35788
aida02 :     42     31     29      5      3      2      4      4      1      3      6   : 36304
aida03 :     29     30     29     16      4      3      2      4      1      3      6   : 36276
aida04 :     28      4     20     12      4      3      4      4      2      2      6   : 35280
aida05 :     25     28     23     10      2      2      2      2      1      4      6   : 36724
aida06 :      5      3     10      5      2      4      2      2      2      3      6   : 35308
aida07 :     38     26     22      9      4      3      1      2      2      3      6   : 35688
aida08 :     52     30     28      8      0      2      2      3      3      2      6   : 35200
aida09 :     22     26     21     10      3      4      1      4      1      3      6   : 35704
aida10 :     37     28     25      7      3      1      1      3      3      3      6   : 36900
aida11 :     53     30     20     10      2      3      2      2      2      3      6   : 35908
aida12 :     37     34     17     10      2      2      3      4      3      2      6   : 35956
aida13 :     47     34     16      7      4      2      3      4      3      2      6   : 36012
aida14 :     37     29     17     10      3      4      3      4      3      2      6   : 36236
aida15 :     44     28     25     12      3      2      4      4      2      2      6   : 35424
aida16 :     40     21     33     15      3      4      1      3      1      3      6   : 35576

18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%


19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32

System-wide checks

Clock, ADC - all pass
WR

	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc4 : 	 12  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     33     30      8      1      4      1      2      2      3      6   : 35788
aida02 :     51     30     30      5      3      2      3      4      1      3      6   : 36092
aida03 :     52     31     28     16      4      2      2      5      1      3      6   : 36744
aida04 :      6      8     20     12      4      4      3      4      2      2      6   : 35096
aida05 :     30     31     22     10      2      2      2      2      1      4      6   : 36752
aida06 :     12      6     10      5      2      3      3      2      2      3      6   : 35488
aida07 :     53     24     21      9      4      3      1      3      2      3      6   : 36228
aida08 :     36     28     28      9      0      2      3      3      3      2      6   : 35408
aida09 :     27     28     25     10      3      4      2      3      1      3      6   : 35548
aida10 :     50     23     26      7      3      1      1      3      3      3      6   : 36928
aida11 :     42     28     22     10      3      3      2      2      2      3      6   : 35944
aida12 :     39     35     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     43     34     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     42     32     16     10      3      4      3      4      3      2      6   : 36264
aida15 :     36     30     25     12      3      2      4      3      3      2      6   : 35920
aida16 :     40     21     25     15      3      4      2      3      1      3      6   : 35704

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19:40 Analysed R5_86. Attached. Dead time as before.

19:57 Shift handed over to James S
Attachment 1: Temp1.png
Temp1.png
Attachment 2: Bias1.png
Bias1.png
Attachment 3: Stats1-1.png
Stats1-1.png
Attachment 4: Stats1-2.png
Stats1-2.png
Attachment 5: Stats1-3.png
Stats1-3.png
Attachment 6: Stats1-4.png
Stats1-4.png
Attachment 7: Stats1-5.png
Stats1-5.png
Attachment 8: Stats1-6.png
Stats1-6.png
Attachment 9: R5_45_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
Attachment 10: Temps2.png
Temps2.png
Attachment 11: Bias2.png
Bias2.png
Attachment 12: Stats2-1.png
Stats2-1.png
Attachment 13: Stats2-2.png
Stats2-2.png
Attachment 14: Stats2-3.png
Stats2-3.png
Attachment 15: Stats2-4.png
Stats2-4.png
Attachment 16: Stats2-5.png
Stats2-5.png
Attachment 17: Stats2-6.png
Stats2-6.png
Attachment 18: Temp3.png
Temp3.png
Attachment 19: Bias3.png
Bias3.png
Attachment 20: Stats3-1.png
Stats3-1.png
Attachment 21: Stats3-2.png
Stats3-2.png
Attachment 22: Stats3-3.png
Stats3-3.png
Attachment 23: Stats3-4.png
Stats3-4.png
Attachment 24: Stats3-5.png
Stats3-5.png
Attachment 25: R5_67_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255938817 ( 1305497.2 Hz)
        Other data format:    5981183 (   30508.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.6 Hz)
                               RESUME:        113 (       0.6 Hz)
                              SYNC100:      32730 (     166.9 Hz)
                              WR48-63:      32730 (     166.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5915496 (   30173.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     257806 (    1315.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      196.047 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.111              102.399
  2                0.513              105.076
  3                3.587               39.451
  4                0.000              182.105
  5                0.792              103.780
  6                0.167              115.991
  7                1.244               72.680
  8                0.805               56.828
  9                0.000              190.127
 10                2.629                6.967
 11                0.295               76.751
 12                0.222              114.996
 13                0.000              188.225
 14                0.031              124.690
 15                2.491               63.947
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8223770       3364          0          0          2          2       1092       1092          0       1176          0      13825
  1   16628095       4338          0          0         12         12       2157       2157          0          0          0      28636
  2   13173045    1831289          0          0          3          3       1864       1864          0    1827555          0      22667
  3   27251797    1807380          0          0         13         13       3668       3668          0    1800018          0      33190
  4    2564061     622927          0          0          0          0        404        404          0     622119          0       6108
  5   14810679     385418          0          0          6          6       1903       1903          0     381600          0      19053
  6   12067625     742516          0          0          6          6       1616       1616          0     739272          0      12600
  7   21266971     549224          0          0         11         11       2723       2723          0     543756          0      22659
  8   23290358       5786          0          0          5          5       2888       2888          0          0          0      28199
  9    1157906        324          0          0          0          0        162        162          0          0          0       2443
 10   45608185      11489          0          0         27         26       5718       5718          0          0          0      16139
 11   20146427       4876          0          0          6          6       2432       2432          0          0          0       5579
 12   13411417       3300          0          0          7          7       1643       1643          0          0          0      24922
 13    2681820        654          0          0          0          0        327        327          0          0          0       1925
 14   11884997       2904          0          0          1          1       1451       1451          0          0          0      15892
 15   21771664       5394          0          0         15         15       2682       2682          0          0          0       3969
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:67311.180s (    0.475 blocks/s,   0.030 Mb/s)
Attachment 26: Stats_RESUME.png
Stats_RESUME.png
Attachment 27: Stats-CORRELATION.png
Stats-CORRELATION.png
Attachment 28: Bias4.png
Bias4.png
Attachment 29: Stats-good.png
Stats-good.png
Attachment 30: Stats-AIDAADC.png
Stats-AIDAADC.png
Attachment 31: Stats_DISC.png
Stats_DISC.png
Attachment 32: Stats_PAUSE.png
Stats_PAUSE.png
Attachment 33: R5_86_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256871525 ( 1549964.4 Hz)
        Other data format:    5048475 (   30462.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        133 (       0.8 Hz)
                               RESUME:        132 (       0.8 Hz)
                              SYNC100:      32617 (     196.8 Hz)
                              WR48-63:      32617 (     196.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4982976 (   30067.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     217412 (    1311.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      165.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.819               66.425
  2                0.365               78.785
  3                3.792               29.131
  4                0.000              149.995
  5                0.868               74.751
  6                0.080               85.583
  7                2.138               33.988
  8                2.899               39.410
  9                0.000              164.566
 10                6.499               11.006
 11                1.139               43.564
 12                0.944               80.601
 13                0.000              152.925
 14                0.102               87.597
 15                2.546               36.680
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8622288       3316          0          0          4          3       1158       1158          0        993          0      11812
  1   17129008       4510          0          0         12         12       2243       2243          0          0          0      24429
  2   13474006    1553973          0          0          6          6       1869       1869          0    1550223          0      19028
  3   26129234    1524704          0          0         20         20       3374       3374          0    1517916          0      27835
  4    2168475     521045          0          0          0          0        352        352          0     520341          0       5247
  5   15823578     326080          0          0          4          4       1963       1963          0     322146          0      16074
  6   12515224     622754          0          0          2          2       1625       1625          0     619500          0      10516
  7   23990039     458079          0          0         12         12       3099       3099          0     451857          0      18924
  8   22108276       5660          0          0         13         13       2817       2817          0          0          0      24206
  9    1062355        236          0          0          0          0        118        118          0          0          0       1949
 10   41223894      10244          0          0         40         40       5082       5082          0          0          0      13136
 11   20572413       4974          0          0          7          7       2480       2480          0          0          0       4577
 12   14036482       3446          0          0          3          3       1720       1720          0          0          0      21370
 13    3191210        818          0          0          0          0        409        409          0          0          0       1573
 14   12949121       3198          0          0          3          3       1596       1596          0          0          0      13450
 15   21875922       5438          0          0          7          7       2712       2712          0          0          0       3286
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:70812.570s (    0.452 blocks/s,   0.028 Mb/s)
  316   Fri May 14 07:05:08 2021 CAFriday 14th May 08:00 - 16:00 shift
08:00 CA takes over

08:28 Problem at UNILAC - no beam

09:30 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveforms)

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA timestamp errors:

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

09:36 FEE64 Temperatures ok - attachment 1

      Detector bias / leakage currents ok - attachment 2

      Statistics - attachments 3-8

09:50 beam is back - writing to file R4_265

09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA

10:10 FRS increase degrader thickness

10.28 all histograms, stats and merger stats zero'd

10.50 all system wide checks ok *except*
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      2      2      2      2      2      2      4      3      3      6   : 36860
aida02 :      7      3      5      1      2      5      2      4      3      3      6   : 37284
aida03 :      7      5      3      1      4      4      2      3      3      3      6   : 36756
aida04 :      3      1     20      4      2      3      3      5      1      3      6   : 36052
aida05 :     19     25     26      6      2      2      3      2      2      3      6   : 35828
aida06 :      1      3     16      1      3      4      1      5      1      3      6   : 35580
aida07 :      6      4      1      2      3      3      2      3      3      3      6   : 36552
aida08 :      8      5      2      3      2      3      2      2      2      4      6   : 37064
aida09 :      9      7      1      2      2      4      2      3      3      3      6   : 36652
aida10 :      2      5      3      1      1      4      2      3      3      3      6   : 36544
aida11 :     16      4      4      2      1      2      2      3      3      3      6   : 36384
aida12 :      2      1      1      3      1      2      2      3      3      3      6   : 36288
aida13 :      0      1      1      2      0      4      3      2      3      3      6   : 36184
aida14 :      6      3      2      0      2      3      2      3      3      3      6   : 36432
aida15 :     25      8      2      1      2      3      1      2      2      4      6   : 36836
aida16 :      3      5      1      2      2      2      3      2      3      3      6   : 36100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.50 DAQ continues S496/R4_304

11:18 FEE64 Temperatures ok - attachment 9

      Detector bias & leakage currents ok - attachment 10

      statistics - attachments 11-16

15:00 FEE64           avg. CPU usage (%)

      1               52
      2               55
      3               55
      4               60
      5               93
      6               50
      7               53
      8               55
      9               55
      10              51
      11              50
      12              67
      13              56
      14              55
      15              55
      16              58

15:08 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveform)

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA errors:

 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:10 Temperatures ok - attachment 17

      Detector bias / leakage current ok - attachment 18

      Statistics - attachments 18 -24
Attachment 1: Screenshot_from_2021-05-14_08-32-16.png
Screenshot_from_2021-05-14_08-32-16.png
Attachment 2: Screenshot_from_2021-05-14_08-32-45.png
Screenshot_from_2021-05-14_08-32-45.png
Attachment 3: Screenshot_from_2021-05-14_08-33-11.png
Screenshot_from_2021-05-14_08-33-11.png
Attachment 4: Screenshot_from_2021-05-14_08-33-33.png
Screenshot_from_2021-05-14_08-33-33.png
Attachment 5: Screenshot_from_2021-05-14_08-34-12.png
Screenshot_from_2021-05-14_08-34-12.png
Attachment 6: Screenshot_from_2021-05-14_08-34-44.png
Screenshot_from_2021-05-14_08-34-44.png
Attachment 7: Screenshot_from_2021-05-14_08-35-17.png
Screenshot_from_2021-05-14_08-35-17.png
Attachment 8: Screenshot_from_2021-05-14_08-35-55.png
Screenshot_from_2021-05-14_08-35-55.png
Attachment 9: Screenshot_from_2021-05-14_10-17-25.png
Screenshot_from_2021-05-14_10-17-25.png
Attachment 10: Screenshot_from_2021-05-14_10-17-43.png
Screenshot_from_2021-05-14_10-17-43.png
Attachment 11: Screenshot_from_2021-05-14_10-22-39.png
Screenshot_from_2021-05-14_10-22-39.png
Attachment 12: Screenshot_from_2021-05-14_10-23-13.png
Screenshot_from_2021-05-14_10-23-13.png
Attachment 13: Screenshot_from_2021-05-14_10-23-41.png
Screenshot_from_2021-05-14_10-23-41.png
Attachment 14: Screenshot_from_2021-05-14_10-24-12.png
Screenshot_from_2021-05-14_10-24-12.png
Attachment 15: Screenshot_from_2021-05-14_10-24-37.png
Screenshot_from_2021-05-14_10-24-37.png
Attachment 16: Screenshot_from_2021-05-14_10-25-20.png
Screenshot_from_2021-05-14_10-25-20.png
Attachment 17: Screenshot_from_2021-05-14_14-07-48.png
Screenshot_from_2021-05-14_14-07-48.png
Attachment 18: Screenshot_from_2021-05-14_14-11-23.png
Screenshot_from_2021-05-14_14-11-23.png
Attachment 19: Screenshot_from_2021-05-14_14-13-21.png
Screenshot_from_2021-05-14_14-13-21.png
Attachment 20: Screenshot_from_2021-05-14_14-13-38.png
Screenshot_from_2021-05-14_14-13-38.png
Attachment 21: Screenshot_from_2021-05-14_14-14-12.png
Screenshot_from_2021-05-14_14-14-12.png
Attachment 22: Screenshot_from_2021-05-14_14-14-39.png
Screenshot_from_2021-05-14_14-14-39.png
Attachment 23: Screenshot_from_2021-05-14_14-15-06.png
Screenshot_from_2021-05-14_14-15-06.png
Attachment 24: Screenshot_from_2021-05-14_14-15-26.png
Screenshot_from_2021-05-14_14-15-26.png
ELOG V3.1.3-7933898