Starting a new night shift - All good

1am full check:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

2:30am: AIDA02 FEE stopped working. DESPEC main DAQ was alerted and remobe AIDA from timestitcher while I was restarting AIDA DAQ.

2:45am: AIDA02  FEE is back and AIDA has been added to the timestitcher again in the main DAQ.

The AIDA runs at the time of the crash to look at are R7_348, 349.

All spectra were zeroed at this time and checked.

4:00 am full check:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - attachement # 11
Tape service - attachement # 12
Spectra - attachement # 13
ucesb - attachement # 14

6:00am full check:
DSSSD bias & leakage current  ok - attachment # 15-16
FEE64 temperatures  ok - attachment # 17
ADC data item stats - attachment # 18
Merger ok - attachement # 19
Tape service - attachement # 20
Spectra - attachement # 21
ucesb - attachement # 22

6:30 the nearline/offline analysis team reported possible noise issue in DSSD#2 and asked if the threshold were changed at any point. I can't find any mention of changes in the elog. 
I've not done a setup of the all DAQ when AIDA02 fee crashes earlier this morning. I just did a setup of the merger. Thus I don't think the threshold have changed when I stopeed and restarted the DAQ.
I can see that currently slow comparator threshod are set to 0xa for most of the DSSD2 fees except for AIDA06 which is set to 0xf. I'm not sure this is the thershold we would need to increase. Something to check during the next shift.

In the UCESB window I can see that the number of decay in DSSD2 is double the number of decay in DSSD1 while the number of implant is similar. Looking at other shift it looks that this was alsready the case in prvious shifts.

07.06 analysis of data file S181/R7_457 - attachment 23

 max deadtime 5.8% aida08, all other FEE64s < 1.4%

LEC data rate 1.788M, HEC data rate 4.3k

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13

9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20

11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27

All seems normal.

Screenshots attached.

07:10 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      Merger, TapeServer etc - attachment 5

      per p+n FEE64 1.8.L spectra - attachment 6
       aida09 pulser peak width 68 ch FWHM 
07.13  peak area 25253 total area 27548 in 221mins => 1.9-2.1Hz => deadtime < 5% or less which is consistent with data file analysis below

      analysis data file S181/R6_274 - attachment 
       max FEE64 deadtime 24% aida08
       HEC data rate 4.5kHz, LEC data rate 2.284M


14.00 DAQ crashed by accident
      FEE64s power cycled 
      DAQ restarted
      data file S181/R7

      DSSSD bias & leakage current OK - attachments 8-9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11

      WR timestamps OK - attachment 12

      All system wide checks *except* as reported attachments 13-15

      per FEE64 Rate spectra - attachment 16

      per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 66 ch FWHM

      per FEE64 1.8.H spectra - attachments 18-19

      per FEE64 1.8.W spectra - 20us FSR - attachment 20-21

      Merger, TapeServer etc - attachments 22-23
       data file S181/R7_16

      analysis data file S181/R7_16
       max deadtime 7.3% aida08, all other FEE64s < 2%
       LEC data rate 1.877M HEC data rate 4.7k

00.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 3

      ADC data items stats OK - attachment 2
   
      per FEE64 Rate spectra - attachment 7

      per p+n FEE64 1.8.L spectra - attachments 6
       aida09 pulser peak width 63 ch FWHM 

      Merger, TapeServer etc - attachments 4-5

      analysis data file S181/R6_274 - attachment 8
       max FEE64 deadtime 19% aida08
       HEC data rate 3.8kHz, LEC data rate 2.258M

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

03.12 Merger, TapeServer etc - attachments 11-12
       before & after Merger restart

      per FEE64 Rate spectra - attachment 13

      ADC data items stats OK - attachment 14
       aida02 zero rate

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage current OK - attachment 16

      analysis data file S181/R6_372 - attachment 17
       max FEE64 deadtime 22% aida08
       HEC data rate 4.4kHz, LEC data rate 2.295M

03.32 all histograms zero'd

03.57 per p+n FEE64 1.8.L spectra - attachment 18
       25 mins = 1500 @ 2Hz = 3000 cf. peak area c. 3000 => deadtime low

Per NH

"From ELOG (assuming times accurateish)

10:00 Source in (AIDA removed from DESPEC DAQ)
11:30: Source out (?)
11:45: Background Runs
13:00: bPlast thresholds changed
15:45: Beam Returned"
-
                  ADC data item rate
EOF time  file    LEC        HEC
09.02    R4_538   0.906M     1.7k
10.02      _551   0.927M     1.7k
11.00      _564   0.875M     1.7k
11.19      _568   0.872M     1.7k
12.18    R6_7     1.121M     1.9k
13.03     6_19    1.134M     1.9k
14.02     6_35    1.197M     2.0k
15.00     6_52    1.232M     2.0k
15.10     6_54    1.225M     2.0k
15.36     6_56    1.467M     2.0k
16.00     6_65    1.630M     3.0k
17.01     6_89    1.778M     3.7k
21.43     6_207   1.759M     2.7k
12.6.24
00.09     6_274   2.258M     3.8k
03.12     6_372   2.295M     4.4k
07.13     6_506   2.284M     4.5k
11.12     6_611   1.853M     3.6k
14.55     7_16    1.877M     4.7k
13.6.24
07.01     7_457   1.788M     4.3k
12.34     7_610   2.000M     4.7k
14.00     7_649   1.756M     2.5k
14.12     7_654   1.817M     2.5k
15.33     7_693   1.907M     3.5k
14.6.24
07.23     8_35    1.677M     3.7k
10.54     8_122   1.880M     4.7k
14.38     8_213   1.935M     3.6k
15.6.24
01.25     9_27    0.951k     1.6k  (beam off from c. 18.15 14.6.24)
08.56     9_152   1.215M     2.1k  gamma source in use?
15.27     9_275   1.520M     2.3k  370kBq 152Eu source in use
16.6.24
15.06     9_747   1.667M     2.3k

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 

All looks good.

Sceenshots attached.

All seems OK

Screenshots attached.

17.33 per FEE64 1.8.W spectra - attachments 8-9
      per p+n FEE64 1.8.L spectra - attachment 10
       aida09 pulser peak width 66 ch FWHM
       cf 
       c. 11:00 today 57 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/650
       cf
       c. 00:10 today 53 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/649

17.48 ADC data item stats - attachment 11
      All histograms zero'd

17.51 Analysis data file S181/R6_96 - attachment 12
      max deadtime 5% (aida08)
      HEC Data items 3.7kHz, LEC data items 1.8MHz
      cf.
      c. 12.29 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/650
      c. 00.09 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/649

10.57 DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2

      ADC, MBS correlation scaler data item stats - attachments 3-4
       ADC 9x < 20k, max 243k aida08

      ASIC settings aida01 & aida02 - attachments 5-6

      ALL system wide checks OK *except* attachments 7-8

      WR timestamps OK - attachment 9

      BNC PB-5 settings - attachment 10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L spectra - attachment 12
       aida09 pulser peak width 57 ch FWHM

      per FEE64 1.8.H spectra - attachments 13-14

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16
       no p+n FEE64 wavefroms - reason unknown

      Merger, TapeServer etc - attachments 17-18

11.25 Lost control of aida-gsi window system 
      All command terminals lost, could not start new command terminals

      aida-gsi restart

      FEE64 power cycle
      DSSSD bias cycle

      Note file manager, calculator apps now working as expected

12.11 DAQ starts 
      data file S181/R6

      ASIC settings 2024Apr19-15.22.49

      BNC PB-5 Pulser
      Amplitude 10.0V
      Attenuation x10
      Fall time 1ms
      Frequency 2Hz
      Polarity + (connected to test + signal daisy chain to all p+n FEE64s)

      Sync ASIC clocks OK
      All system wide checks OK *except* aida02 FPGA/WR decoder - attachments 19-20

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-22

12.29 analysis data file S181/R6_9 - attachment 23
      max dead time c. 2% (aida08)

15:34 For some reason the AIDA MBS wasn't connecting to the time sorter (or a check stream) [NH]
      Try to reset the AIDA MBS 
      Due to confusion the AIDA Merger was rebooted too, while the DAQ was running, this broke all the FEEs -> Powecycled
      Synced ASIC clocks, fixed all waveforms
      Masked all discriminators
      All looks ok -> Sending back to MBS and back in DESPEC DAQ

00.09 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      per p+n FEE64 1.8.L spectra - attachments 5-6
       aida09 pulser peak width 53 ch FWHM 

      per FEE64 1.8.H spectra - attachments 7-8

      per n+n FEE64 1.8.W spectra - 20us FSR - attachment 9
       p+n 1.8.W not running currently

      Merger, TapeServer etc - attachments 10-11

      analysis data file S181/R4_427 - attachment 12
       max FEE64 deadtime 1.7%
       HEC data rate 3.1kHz

00.28 all histograms zero'd


02.39 DAQ continues data file S181/R4_460
      Beam off since 02.08

      DSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data items stats OK - attachment 15
       9x < 20k, max 239k aida08
   
      per FEE64 Rate spectra - attachment 16


07.15 DAQ continues data file S181/R4_517
      Beam off since 02.08
      HKR report c 3-4 h until UNILAC fixed

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18

      ADC data items stats OK - attachment 19
       8x < 20k, max 238k aida08
   
      per FEE64 Rate spectra - attachment 20

Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

8am checks

temps ok - attachment 2

rates ok - attachment 8

voltage ok - attachment 5

merger ok - attachment  6

9am checks

temps ok - attachment 10

rates ok - attachment 9

voltage ok - attachment 12

merger ok - attachment  11

(Ignore  Grafana plots 13-43 inc - not refreshed)

9.16 beam stopped to optimise beam intensity

10.00 checks - still no beam

temps ok - attachment 17

rates ok - attachment 15+16

voltage ok - attachment 14

merger ok - attachment  18

11.00 checks - still no beam

temps ok - attachment 23

rates ok - attachment 21+22

voltage ok - attachment 20

merger ok - attachment  24

12pm checks

temps ok - attachment 28

rates ok - attachment 26+27

voltage ok - attachment 29

merger ok - attachment  30

12.45 - beam back

temps ok - attachment 34

rates ok - attachment 32+33

voltage ok - attachment 35

merger ok - attachment  36

2pm checks

temps ok - attachment 40

rates ok - attachment 38+39

voltage ok - attachment 41

merger ok - attachment  42

/tmp/R4_276 added - attachment 44

max deadtime 10.5% (aida08)

n+n FEE64s deadtime 

3pm checks

temps ok - attachment 48

rates ok - attachment 46+47

voltage ok - attachment 45+50

merger ok - attachment  49

23.47 DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 4-5 & 9

      WR timestamp - attachment 7

      all system wide checks OK *except* attachments 6 & 10

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-13

      Merger, TapeServer etc - attachments 14-15
       data file S181/R4, TapeServer No Storage mode

00.00 Beam ON 3s spill on, 2s spill off
      99%+ fission fragments
      ADC data item stats - attachment 16


02.03 Data storage enabled
 
      per FEE64 1.8.H spectra - attachments 17-18

02.35 analysis data file S181/R4_28 - attachment 19
      data file S181/R4
      max deadtime aida08 18%
      n+n FEE64s 12-18%, all p+n FEE64s < 2%

02.41 Noted that there is disc data written to disk - all disc data now disabled for all FEE64s

04.08 DSSSD bias & leakage current OK - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Rate spectra - attachment 23

04.15 aida02 no data
      restarted merger per https://elog.ph.ed.ac.uk/DESPEC/644

06.00 DSSSD bias & leakage current OK - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27

AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.

17:00

DSSSD bias & leakage current - attachment 2

FEE64 temperatures OK - attachment 3

Statistics  attachment 4

17:39

Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.

23:19 Flange removed. Starting to take beam.

1) Select 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode please ignore this step

2) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

3) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

4) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

5) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

6) Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

8) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

9) Unselect 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode at the start of this procedure please ignore this step


   If you continue to have problems, please request expert assistance.

23.58 8.6.24

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

03.43 slow comparator 0x64

      DSSSD bias & leakage current OK - attachments 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 1.8.H spectra - attachments 8-9

      per FEE64 Rate spectra - attachment 10


09.11 DSSSD bias & leakage current OK - attachments 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats - attachment 13

      per FEE64 Rate spectra - attachment 14


15.05 online spectra
      per DSSSD HEC E_p versus E_n and x versus y
      observe flat field of c. 500MeV at per spull rate c. 5Hz - fission fragments?

14.09 aida02 WR stall
      DAQ STOP
      restart Merger, Mrger SETUP, Merger GO
      DAQ GO

      all histograms & stats zero'd

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 4

      WR timestamp before/after Merger restart - attachments 3 & 5

      all system wide checks OK *except* attachments 6-7

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 8-10
       ADC data items rates high - source installed?

      per FEE64 Rate spectra - attachments 11-13

      per FEE64 1.8.W spectra - 20us FSR - attachments 14-15

      per p+n FEE64 1.8.L spectra - attachment 16
       aida09 pulser peak width 60 ch FWHM

      Merger, TapeServer etc - attachments 17-19
       data file S181/R2_249 switch TapeServer to No Storage mode

      per p+n FEE64 1.8.W spectra - 20us FSR - attachment 20

20.33 DSSSD bias & leakage current OK - attachments 21

      FEE64 temperatures OK - attachment 22

      ADC data item stats - attachment 23

      per FEE64 Rate spectra - attachment 24

05.01 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      9x < 20k, max 228k aida08

      per FEE64 Rate spectra - attachment 4


08.18 DSSSD bias & leakage current - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 8


14.37 DSSSD bias & leakage current - attachment 9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 12

      Writing to disk file S181/R1 (continuing alpha background file)

14.38 End run S181/R1 and start new run file S181/R2

17.09 08.18 DSSSD bias & leakage current - attachment 13

      FEE64 temperatures OK - attachment 14

      MBS correlation scaler data - attachment 15
       data to aida01, aida02, aida03 and aida10

      ADC data item stats - attachment 16
      9x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 17

      Merger/TapeServer etc - attachments 18-19 

12.09 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      8x < 20k, max 247k aida08

      per FEE64 Rate spectra - attachment 4

12.16 all histograms & stats zero'd


21.08 DSSSD bias & leakage current - attachments 5-6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8
      8x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 9


      

We are trying to bias each system on its own and in sequence biasing each other detector system to see the interdependence.
 

HV OK. TEMP OK. Attachment 1 &2.

9.43 bPlast unbiased, BB7 unbiased. Attachments 3-6 is the noise situation with only AIDA biased (bPlast was biased but turned off after AIDA was biased.)

10/16 FEEs < 20k noise. Max noise is 250k. Downstream AIDA appears to be seeing additional noise since S100.

11:21 Returning after an hour the noise has increased significantly in many channels. Scale changed to 10000 in histograms, it appears that the problem is from a couple hot channels.

11:41 'Power' cycled AIDA turned off. bPlast turned on, on its own, then BB7. bPlast doesn't seem to notice anything.

12:21 bPlast and BB7 on. AIDA ON. Rates and histograms given by attachment 7-8. More FEEs noisy, but same level as 11:21. There appears to be the same noisy channel in all of the FEEs (?) Some grounding not properly accounted for, or a wire touching somewhere (?)

13:55h: AIDA & BB7 on. Rates and histograms given in attachments 9-10. Same level as bplast & AIDA on. Same noisy channels in FEEs. According to Nic, these correspond to the end of the ribbon cables & is normal.

13:59 AIDA, bPlast & BB7 on before lunch. Same levels of noise as in previous combinations (attachments 11-12).

So it seems we can run with the three sub-systems biased simultaneously.