Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C

0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.

After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.

I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.

I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.

We are now back in business with AIDA back in main DAQ at  01:40

01:45 full checks:

           Bias: AIDA-Graphana - Attachment 1

               Leakage current ramping up again ! S4 temperaturenot really going down either: 25C

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5

      ucesb - attachment 6

03:00 full checks

          No change to be highlighted. All pretty much the same as during previous full checks.

05:00 full checks

Expt still running very smoothly.

           Bias: AIDA-Graphana - Attachment 7-8

               Leakage current going down slightly

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
      per FEE64 Rate spectra - attachment 11

      Merger etc - attachment 12

      ucesb - attachment 13

07:15 all running smoothly.

07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.

07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.

It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!

0.15 Checks:
      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage current dropped a bit since the evening shift ~22:00.

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

      ucesb - attachment 7

2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12

3:30 Still running smoothly. 

5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.

        DAQ restarted OK. Main DAQ informed.

00.09 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      per p+n FEE64 1.8.L spectra - attachments 5-6
       aida09 pulser peak width 53 ch FWHM 

      per FEE64 1.8.H spectra - attachments 7-8

      per n+n FEE64 1.8.W spectra - 20us FSR - attachment 9
       p+n 1.8.W not running currently

      Merger, TapeServer etc - attachments 10-11

      analysis data file S181/R4_427 - attachment 12
       max FEE64 deadtime 1.7%
       HEC data rate 3.1kHz

00.28 all histograms zero'd


02.39 DAQ continues data file S181/R4_460
      Beam off since 02.08

      DSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data items stats OK - attachment 15
       9x < 20k, max 239k aida08
   
      per FEE64 Rate spectra - attachment 16


07.15 DAQ continues data file S181/R4_517
      Beam off since 02.08
      HKR report c 3-4 h until UNILAC fixed

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18

      ADC data items stats OK - attachment 19
       8x < 20k, max 238k aida08
   
      per FEE64 Rate spectra - attachment 20

23.36 Analysis of data file R17_34 - attachment 1
      max deadtime c. 1% (aida02)
      no timewarps

23.38 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 3
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

01.03 DSSSD bias & leakage current OK - attachment 7

      FEE64 temperatures OK - attachment 8
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 9

      per FEE64 Rate spectra - attachment 10

      Merger etc - attachment 11

03.06 DSSSD bias & leakage current OK - attachment 12

      FEE64 temperatures OK - attachment 13
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 14

      per FEE64 Rate spectra - attachment 15

      Merger etc - attachment 16

05.04 DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Merger etc - attachment 21

07.29 DSSSD bias & leakage current OK - attachment 22

      FEE64 temperatures OK - attachment 23
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 24

      per FEE64 Rate spectra - attachment 25

      Merger etc - attachment 26

0.00 checks - all looks ok

Voltage/currents - attachment 1

Rates attachment 2 + 3

Temps - attachment 4

Merger - attachment 5

ucesb - attachment 6

Grafana - attachment 7

1.30 -aida1 in merger Links, aida02 in rate spectra, has stopped - resetting DAQ.

Datafile R7_961 being written when stopped

1.40 - seems ok

1.50 checks - seems ok

Voltage/currents - attachment 8

Rates attachment 9 + 10

Temps - attachment 11

Merger - attachment 12

ucesb - attachment 13

Grafana - attachment 14

3.30 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1010 being written when stopped

3.38 - seems ok

3.40 checks - all looks ok

Voltage/currents - attachment 15

Rates attachment 16 + 17

Temps - attachment 18

Merger - attachment 19

ucesb - attachment 20

Grafana - attachment 21

Zeroed spectra - attachments 22-28

5.39 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1062 being written when stopped

AIDA out of the main data

Could not get it to restart folowing the notes on the elog - SOAP errors as shown in attachment 29. called Tom who''s logged in 

6.10 - seems to be working again. Not sure what happened. Tom thinks multiple restarts fixed it

Started data file R8

AIDA back in the main data

FRS shifters report a problem with the AIDA white rabbit time stamp - attachment 30

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  17D8C4ED 8F46CF7B , WR/10=>   2627A17C18714BF, Readout Time =>   2627A17C2A74000
aida02 : White Rabbit=>  17D8C4ED A0A1B623 , WR/10=>   2627A17C3435F03, Readout Time =>   2627A17C4410000
aida03 : White Rabbit=>  17D8C4ED AFDACE68 , WR/10=>   2627A17C4C914A4, Readout Time =>   2627A17C5D3C000
aida04 : White Rabbit=>  17D8C4ED C0516FC2 , WR/10=>   2627A17C66E8B2D, Readout Time =>   2627A17C7AC4000
aida05 : White Rabbit=>  17D8C4ED D3B93DAB , WR/10=>   2627A17C85F52F7, Readout Time =>   2627A17C9AEC000
aida06 : White Rabbit=>  17D8C4ED E8C9B243 , WR/10=>   2627A17CA7A91D3, Readout Time =>   2627A17CB704000
aida07 : White Rabbit=>  17D8C4ED FFE475FD , WR/10=>   2627A17CCCA0BCC, Readout Time =>   2627A17CDD94000
aida08 : White Rabbit=>  17D8C4EE 103232A6 , WR/10=>   2627A17CE6B6B77, Readout Time =>   2627A17CF7BC000
aida09 : White Rabbit=>  17D8C4EE 20A1F612 , WR/10=>   2627A17D0103235, Readout Time =>   2627A17D112C000
aida10 : White Rabbit=>  17D8C4EE 307D8FE7 , WR/10=>   2627A17D1A627FD, Readout Time =>   2627A17D2958000
aida11 : White Rabbit=>  17D8C4EE 4402829D , WR/10=>   2627A17D399D9DC, Readout Time =>   2627A17D4570000
aida12 : White Rabbit=>  17D8C4EE 6288B04D , WR/10=>   2627A17D6A744D4, Readout Time =>   2627A17D7D94000
aida13 : White Rabbit=>  17D8C4EE 751E1306 , WR/10=>   2627A17D88301E7, Readout Time =>   2627A17D9C34000
aida14 : White Rabbit=>  17D8C4EE 894E8806 , WR/10=>   2627A17DA87DA67, Readout Time =>   2627A17DBFF4000
aida15 : White Rabbit=>  17D8C4EE 9EEEB997 , WR/10=>   2627A17DCB178F5, Readout Time =>   2627A17DDC18000
aida16 : White Rabbit=>  17D8C4EE AF0BC923 , WR/10=>   2627A17DE4DFA83, Readout Time =>   2627A17DF5C8000

Tom thinks everything appears ok with AIDA

Have converted WR timestamp to data/time - see attachment 40

At the ~second timescale AIDA WR timestamp looks OK, i.e. no gross errors. Will need to check correlations with other detector sub-systems to confirm WR on the ~us time timescale.. 

However, ~10us offset reported by DESPEC online crew is correct (AIDA timestamps data at a later point in the signal processing cycle than other detector sub-systems). Have suggested online crew contact Nic or Calum to check what the AIDA WR timediff spectra should look like.

7.00 checks - all looks ok

Voltage/currents - attachment 31

Rates attachment 32 + 33

Temps - attachment 36

Merger - attachment 37

ucesb - attachment 39

Grafana - attachment 38

Handed over to Tom

The following messages are in the system log very often:

Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk)

Overnight alpha data

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Feb 20 17:46:47 2020
Last event : Fri Feb 21 10:07:57 2020
Duration   : 58869.9 seconds
--------------------------------------
MBS Events   (M):     1248648 (21 Hz)
AIDA Words   (W): 10222680133 (27 kHz)
ADC Words    (A):  3407431088 (-15076 Hz)
-- Decays    (D):  3407017078 (-15083 Hz)
-- Implants  (I):      414010 (7 Hz)
Info Words   (F):  6815249045 (-30146 Hz)
-- SYNC48    (S):  3407559635 (-15074 Hz)
-- SYNC63    (s):  3407559635 (-15074 Hz)
-- PAUSE     (P):         614 (0 Hz)
-- RESUME    (R):         614 (0 Hz)
-- Discrim   (D):      128547 (2 Hz)
-- Unknown   (U):           0 (0 Hz)
Timewarps    (T):          79 (0 Hz)
Dead Time    (X): 26.8658 s (0.04563588%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |           W           A           D           I           F           S           s     P     R       D     U     T  X
   1 |        5025        1246        1093         153        3779        1675        1675     0     0     429     0     0  0
   2 |        6180         991         851         140        5189        2060        2060     0     0    1069     0     0  0
   3 |  7692461404  2564028610  2564028286         324  5128432794  2564153392  2564153392   614   614 	124782     0    58 26.8658
   4 |        2277         642         581          61        1635         759         759     0     0     117     0     0  0
   5 |        6174        1249        1010         239        4925        2058        2058     0     0     809     0     0  0
   6 |     2467062      822244      822191          53     1644818      822354      822354     0     0     110     0     0  0
   7 |        2484         740         701          39        1744         828         828     0     0      88     0     0  0
   8 |   358445004   119481621   119069026      412595   238963383   119481668   119481668     0     0      47     0     1  0
   9 |        7134        1530        1243         287        5604        2378        2378     0     0     848     0     0  0
  10 |  2168031885   722677295   722677295           0  1445354590   722677295   722677295     0     0       0     0    20  0
  11 |        3981        1252        1190          62        2729        1327        1327     0     0      75     0     0  0
  12 |     1241523      413668      413611          57      827855      413841      413841     0     0     173     0     0  0
**** DESPECAnalysis: PostLoop

A few FEEs running "hot", maybe check. 3 had some alignment issues may need rechecking.
Timewarps somehow in aida03 and aida10 but may be code mistake - must check.
Only aida03 ran fast enough to have dead time.

aida10 shows decay data now - DSSD seems ok

Fig 1: DSSD 1D hit pattern 

Fig 2: DSSD Energy

Fig 3: Single channel in aida03 that is hot

Fig 4: Channels 48-64 in aida08 are hot (wiring?)

Fig 5: Channels 17, 19 (crosstalk?) and 27, 33, 38, 39

All otehr channels quiet and OK. 

Everything appears to be running smoothly. Attachments 1-4 were taken around 16:00

DSSSD Bias & Leakage Current - Check: OK (see attachment 1)
FEE64 Temperature - Check: OK (see attachment 2)
ADC Data Item Stats and Spectrum Browser - Check: OK (see attachments 3 and 4)

Another set of screenshots was taken around 18:00, and everything looks good.

DSSSD bias & leakage current  ok - attachment # 5-6
FEE64 temperatures  ok - attachment # 7
ADC data item stats - attachment # 8
Merger ok - Attachement #9
Tape service - attachement # 10
ucesb - attachment #11

At 19:17, AIDA03 temperatures gave no response (see attachment 12), but after two minutes, they returned to normal.

I noted that the  bplse in ucesb has no data (see attachment 13). It may need to change the disc thresholds again
 it does not affect AIDA

At 19:30 most aida temperature are high and I spoke to people on site via zoom one of them she is working with AIDA so she was able to help. She called Nick and it is found that there is a problem in the cooling system so she stopped AIDA

At 22 the cooling system fixed and Nic and the other person on site restarted AIDA