Starting a new night shift - All good

1am full check:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

2:30am: AIDA02 FEE stopped working. DESPEC main DAQ was alerted and remobe AIDA from timestitcher while I was restarting AIDA DAQ.

2:45am: AIDA02  FEE is back and AIDA has been added to the timestitcher again in the main DAQ.

The AIDA runs at the time of the crash to look at are R7_348, 349.

All spectra were zeroed at this time and checked.

4:00 am full check:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - attachement # 11
Tape service - attachement # 12
Spectra - attachement # 13
ucesb - attachement # 14

6:00am full check:
DSSSD bias & leakage current  ok - attachment # 15-16
FEE64 temperatures  ok - attachment # 17
ADC data item stats - attachment # 18
Merger ok - attachement # 19
Tape service - attachement # 20
Spectra - attachement # 21
ucesb - attachement # 22

6:30 the nearline/offline analysis team reported possible noise issue in DSSD#2 and asked if the threshold were changed at any point. I can't find any mention of changes in the elog. 
I've not done a setup of the all DAQ when AIDA02 fee crashes earlier this morning. I just did a setup of the merger. Thus I don't think the threshold have changed when I stopeed and restarted the DAQ.
I can see that currently slow comparator threshod are set to 0xa for most of the DSSD2 fees except for AIDA06 which is set to 0xf. I'm not sure this is the thershold we would need to increase. Something to check during the next shift.

In the UCESB window I can see that the number of decay in DSSD2 is double the number of decay in DSSD1 while the number of implant is similar. Looking at other shift it looks that this was alsready the case in prvious shifts.

07.06 analysis of data file S181/R7_457 - attachment 23

 max deadtime 5.8% aida08, all other FEE64s < 1.4%

LEC data rate 1.788M, HEC data rate 4.3k

 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257532265 ( 1788132.4 Hz)
        Other data format:    4387735 (   30465.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1032 (       7.2 Hz)
                               RESUME:       1032 (       7.2 Hz)
                              SYNC100:      32537 (     225.9 Hz)
                              WR48-63:      32537 (     225.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4320597 (   29999.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     613796 (    4261.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      144.023 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.236                0.000
  1                0.214                0.000
  2                0.096                0.000
  3                0.448                0.000
  4                0.012                0.000
  5                0.887                0.000
  6                0.003                0.000
  7                8.383                0.000
  8                0.002                0.000
  9                3.230                0.000
 10                1.587                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.540                0.000
 14                0.000                0.000
 15                1.978                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13645957       7863          0          0         38         38       1738       1738          0       4311          0      14717
  1    9668775       2474          0          0         22         22       1215       1215          0          0          0      34051
  2   13971257    1386114          0          0         14         14       1900       1900          0    1382286          0      19760
  3   18385419       4726          0          0         51         51       2312       2312          0          0          0      26170
  4    5575749     526316          0          0          3          3        775        775          0     524760          0       8695
  5   17895997       4760          0          0        110        110       2270       2270          0          0          0      31206
  6    5682061     408056          0          0          1          1        783        783          0     406488          0      19551
  7   46828757      11780          0          0        204        204       5686       5686          0          0          0     333264
  8    6455085       1532          0          0          1          1        765        765          0          0          0      22949
  9   38428621    1359707          0          0        141        141       4975       4975          0    1349475          0      21660
 10   18781575       5148          0          0        247        247       2327       2327          0          0          0      13075
 11    4229184     654561          0          0          1          1        641        641          0     653277          0      20787
 12    4432128       1092          0          0          0          0        546        546          0          0          0       5445
 13   14014813       3450          0          0         86         86       1639       1639          0          0          0      19095
 14    5156679       1244          0          0          0          0        622        622          0          0          0      13849
 15   34380208       8912          0          0        113        113       4343       4343          0          0          0       9522
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   52.717s (  607.017 blocks/s,  37.939 Mb/s)