AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 18 of 37  ELOG logo
  ID Date Author Subjectdown
  391   Fri Oct 29 10:49:01 2021 OHNoise tests - 28th October -29th October
Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24
Attachment 1: 20211028-1300-rates-groundedfees.png
20211028-1300-rates-groundedfees.png
Attachment 2: 20211028-1300-ratespectra.png
20211028-1300-ratespectra.png
Attachment 3: 20211028-1300-ratespectraymax150.png
20211028-1300-ratespectraymax150.png
Attachment 4: 20211028-1300-aida11wave.png
20211028-1300-aida11wave.png
Attachment 5: 20211028-1300-aida02wave.png
20211028-1300-aida02wave.png
Attachment 6: 20211018-1326-rates-psuchange.png
20211018-1326-rates-psuchange.png
Attachment 7: 20211028-1326-ratesspectra-newpsu.png
20211028-1326-ratesspectra-newpsu.png
Attachment 8: 20211028-1326-rates-linymax10-newpsu.png
20211028-1326-rates-linymax10-newpsu.png
Attachment 9: 20211028-1326-wavesaida12-pnpulser-newpsu.png
20211028-1326-wavesaida12-pnpulser-newpsu.png
Attachment 10: 20211028-1326-wavesaida02-newpsupng
Attachment 11: 20211028-1410-ratespulserconnoff.png
20211028-1410-ratespulserconnoff.png
Attachment 12: 20211028-1412-ratespectra-normal.png
20211028-1412-ratespectra-normal.png
Attachment 13: 20211028-1412-ratesspectra-ymax10.png
20211028-1412-ratesspectra-ymax10.png
Attachment 14: 20211028-1413-ratesymax10-pulserONpng
Attachment 15: 20211028-1414-ratesspectra-pulseron.png
20211028-1414-ratesspectra-pulseron.png
Attachment 16: 20211028-1424-pulserpeakspn.png
20211028-1424-pulserpeakspn.png
Attachment 17: 20211028_1449_nn_18L.png
20211028_1449_nn_18L.png
Attachment 18: 20211028_1451_pn_18l.png
20211028_1451_pn_18l.png
Attachment 19: PXL_20211028_121721708.jpg
PXL_20211028_121721708.jpg
Attachment 20: PXL_20211028_121738665.jpg
PXL_20211028_121738665.jpg
Attachment 21: PXL_20211028_121728173.jpg
PXL_20211028_121728173.jpg
Attachment 22: 20211029_1218_stats.png
20211029_1218_stats.png
Attachment 23: 20211029-1400-pnlk1234nnlk124.png
20211029-1400-pnlk1234nnlk124.png
Attachment 24: 20211029-1436-nolk3bar1and3.png
20211029-1436-nolk3bar1and3.png
  390   Wed Oct 27 15:12:50 2021 OH, NHNoise tests - 27th October
 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6
Attachment 1: 271021-waveaida10-pndssd-filter.png
271021-waveaida10-pndssd-filter.png
Attachment 2: 271021-dssds-fixed12.png
271021-dssds-fixed12.png
Attachment 3: 271021-waves-dssd-nn.png
271021-waves-dssd-nn.png
Attachment 4: 271021-waveaida10-pndssd.png
271021-waveaida10-pndssd.png
Attachment 5: 271021-waves-aida02-dssdnn-filter.png
271021-waves-aida02-dssdnn-filter.png
Attachment 6: 271021-waveaida10-pndssd-filter.png
271021-waveaida10-pndssd-filter.png
  387   Thu Oct 21 09:33:08 2021 OHNoise tests - 21st Oct 2021
Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.
Attachment 1: 211021_0951_fee8_x8L.png
211021_0951_fee8_x8L.png
Attachment 2: 211021_0953_fee14waveform_zoom.png
211021_0953_fee14waveform_zoom.png
Attachment 3: 211021_0956_fee14_waveform.png
211021_0956_fee14_waveform.png
Attachment 4: 211021_0957_fee8_x8L.png
211021_0957_fee8_x8L.png
Attachment 5: 211021_0959_fee8_waveform_zoom.png
211021_0959_fee8_waveform_zoom.png
  574   Mon Apr 15 09:23:43 2024 CC, JB, HMA, TDNoise in AIDA when connecting BB7

9:40 started up AIDA. TEMP OK. Noise condition same as Saturday. Attachments 1-3.

9:56 BB7 premaps connected but not powered. Attahcment 4.

10:20 BB7 Preamps powered. Attachment 5.

Noise increased marginally, but still 10/16 <20kHz, max 130k.

 

 

15.15  per FEE64 Rate spectra - attachment 6

             ADC data item stats - attachment 7

           increased noise observed - 6/16 < 20k, max 300k 

 

17.10  per FEE64 1.8.W spectra - 20us FSR - attachments 8-9

            per FEE64 Rate spectra - attachment 10

             ADC data item stats - attachment 11

           increased noise observed - 7/16 < 20k, max 180k 

20.30  per FEE64 Rate spectra - attachment 12

             ADC data item stats - attachment 13

           status quo ante - 10/16 < 20k, max 130k 

 

          FEE64 power OFF

          DSSSD bias OFF

 

 

Attachment 1: Screenshot_from_2024-04-15_09-55-15.png
Screenshot_from_2024-04-15_09-55-15.png
Attachment 2: Screenshot_from_2024-04-15_09-54-58.png
Screenshot_from_2024-04-15_09-54-58.png
Attachment 3: Screenshot_from_2024-04-15_09-54-17.png
Screenshot_from_2024-04-15_09-54-17.png
Attachment 4: Screenshot_from_2024-04-15_10-13-20.png
Screenshot_from_2024-04-15_10-13-20.png
Attachment 5: Screenshot_from_2024-04-15_10-24-39.png
Screenshot_from_2024-04-15_10-24-39.png
Attachment 6: Screenshot_from_2024-04-15_15-24-34.png
Screenshot_from_2024-04-15_15-24-34.png
Attachment 7: Screenshot_from_2024-04-15_15-23-41.png
Screenshot_from_2024-04-15_15-23-41.png
Attachment 8: Screenshot_from_2024-04-15_17-16-19.png
Screenshot_from_2024-04-15_17-16-19.png
Attachment 9: Screenshot_from_2024-04-15_17-15-27.png
Screenshot_from_2024-04-15_17-15-27.png
Attachment 10: Screenshot_from_2024-04-15_17-14-32.png
Screenshot_from_2024-04-15_17-14-32.png
Attachment 11: Screenshot_from_2024-04-15_17-13-37.png
Screenshot_from_2024-04-15_17-13-37.png
Attachment 12: Screenshot_from_2024-04-15_20-29-18.png
Screenshot_from_2024-04-15_20-29-18.png
Attachment 13: Screenshot_from_2024-04-15_20-28-48.png
Screenshot_from_2024-04-15_20-28-48.png
  462   Mon May 16 17:23:27 2022 TDNewMerger stats
Attachment 1: Screenshot_from_2022-05-16_18-17-10.png
Screenshot_from_2022-05-16_18-17-10.png
Attachment 2: Screenshot_from_2022-05-16_18-17-50.png
Screenshot_from_2022-05-16_18-17-50.png
Attachment 3: Screenshot_from_2022-05-16_18-18-24.png
Screenshot_from_2022-05-16_18-18-24.png
Attachment 4: Screenshot_from_2022-05-16_18-20-02.png
Screenshot_from_2022-05-16_18-20-02.png
  51   Mon May 20 13:33:06 2019 NHNew Merger/MBS Test Runs

New merger has been worked on by VP which fixes the timewarp issues and MBS integration.

Currently no WAVE capture is supported, VP will re-add WAVE histogramming for startup testing.

10.05.2019 - Pulser walkthrough
-> Pulser rate 50 Hz
-> 1 minute per 0.1 V (0.1 to 1 V)
-> extra time on 1 V

With this 95% of channels calibrate easily (good statistics)
Issue: ~16 channels (most of ASIC3) in FEE10 are horrificly noisy. Not calibrated

Fig 1. Pulser of FEE1 (no DSSD)
Fig 2. Pulser of FEE9 (DSSD)
Fig 3&4. Pulser of FEE10 noisy channels

Future TODO: Will investigate WAVE data and look if issue can be found.

20.05.2019 - Alpha walkthrough
-> Pulser off
-> FEE10ASIC3 has readout off & discriminator masked
-> Slow threshold 0x64
-> Fast threshold 0xff
-> HEC threshold 0x2

File is nyx/gamma/nhubbard/AIDA/alpha_bg_200519_

Rate is approx. 1MB/s still of WR timestamp data.

(NB, MIDAS Alpha walkthrough before beamtime I see no alpha events in!)

Fig 5: Temps
Fig 6: Stats
Fig 7: Merger Stats
Fig 8: MBS Stats

Attachment 1: 20190522-PulserFEE1.png
20190522-PulserFEE1.png
Attachment 2: 20190522-PulserFEE9.png
20190522-PulserFEE9.png
Attachment 3: 20190522-PulserFEE10-Bad.png
20190522-PulserFEE10-Bad.png
Attachment 4: 20190522-PulserFEE10-VBad.png
20190522-PulserFEE10-VBad.png
Attachment 5: 20190522-Temps.png
20190522-Temps.png
Attachment 6: 20190522-Stats.png
20190522-Stats.png
Attachment 7: 20190522-StatsMerger.png
20190522-StatsMerger.png
Attachment 8: 20190522-MBSRate.png
20190522-MBSRate.png
  61   Tue Aug 13 13:55:45 2019 NHNew Merger/MBS Test Runs

AIDA Finally working again so can test VP's added WAVE histogramming mode.

Fig 1. Waveforms for FEE12 ASIC 1. Sample rate = 10, Threshold = 10500
n.b. lots of "blank" wave channels for unknown reason, all low-energy channels show pulser so it's not a DSSD signal problem?

Fig 2. Pulser spectrum for FEE12 ASIC 1. (10 peaks 0.1 -> 1 V)

Fig 3. Waveforms for FEE10 ASIC 3.

n.b. WAVE channels frequently stopped updating, needed start/stop DAQ or restart WAVE readout option. Unsure if it's related to high rate/odd signals.
Might be related to issue in FEE12 too. WAVE good events statistics lower than empty FEEs.
All rates quite high, looks quite noisy at the moment.

Fig 4. Pulser of FEE10 ASIC 3.

Note that FEE10.3 has poor peaks, in WAVE we see:
Some normal looking pulses which are merely noiser than other channels.
Some pulses which look very odd (baseline really low, etc).

Will recheck ground for noise issues, but the huge peaks are confusing to me. (3.5.W, etc)

Quote:

New merger has been worked on by VP which fixes the timewarp issues and MBS integration.

Currently no WAVE capture is supported, VP will re-add WAVE histogramming for startup testing.

10.05.2019 - Pulser walkthrough
-> Pulser rate 50 Hz
-> 1 minute per 0.1 V (0.1 to 1 V)
-> extra time on 1 V

With this 95% of channels calibrate easily (good statistics)
Issue: ~16 channels (most of ASIC3) in FEE10 are horrificly noisy. Not calibrated

Fig 1. Pulser of FEE1 (no DSSD)
Fig 2. Pulser of FEE9 (DSSD)
Fig 3&4. Pulser of FEE10 noisy channels

Future TODO: Will investigate WAVE data and look if issue can be found.

20.05.2019 - Alpha walkthrough
-> Pulser off
-> FEE10ASIC3 has readout off & discriminator masked
-> Slow threshold 0x64
-> Fast threshold 0xff
-> HEC threshold 0x2

File is nyx/gamma/nhubbard/AIDA/alpha_bg_200519_

Rate is approx. 1MB/s still of WR timestamp data.

(NB, MIDAS Alpha walkthrough before beamtime I see no alpha events in!)

Fig 5: Temps
Fig 6: Stats
Fig 7: Merger Stats
Fig 8: MBS Stats

 

Attachment 1: AIDA_FEE122-ASIC1.png
AIDA_FEE122-ASIC1.png
Attachment 2: Pulser-FEE12-ASIC1.png
Pulser-FEE12-ASIC1.png
Attachment 3: AIDA-FEE10-ASIC3.png
AIDA-FEE10-ASIC3.png
Attachment 4: Pulser-FEE10-ASIC3.png
Pulser-FEE10-ASIC3.png
Attachment 5: Rates-WAVE.png
Rates-WAVE.png
Attachment 6: Rates-good.png
Rates-good.png
  157   Wed Mar 18 18:02:37 2020 PJCSNew Firmware loaded in all FEE64s

Loaded new frimware into all 12 FEE64s in the system. IWR_Dec19_4.bin 

Booted ok after power-cycle.  

RESET/SETUP/GO all fine.

system wide checks ( SWC ) clocks, white rabbit ( WR ) OK, ADC calibrated : aida10 failed. Couldn't get them to calibrate.

Checked aida01 waveforms for ASIC1. set threshold for LED to 5000 to stop triggering in the noise.

Set White Rabbit register SYNC Rollover Target ( 14 ) to 0x4000 instead of 0xe to reduce the data rate to better understand the waveform activity.

All FEE64s are set to enable all channels of ASIC1 with low thresholds so a lot of data moving. Checked activity using statistics window.

Checked waveforms working for ASIC1 of aida01. Noisy but capturing the pulser pulse.

 Enabled all the waveforms on aida01 to see if they all function ok. After a few seconds waveform readout stopped. Not sure of the reason.

STOP, change all four thresholds to 5000, Check ASIC controls : all ok, GO. No activity.

RESET/SETUP ok.

SWC : clock failed, ADC failed , WR ok ????

Power-cycle...... concludes ok, RESET/SETUP ok, SWC : clocks all failed, ADC all failed, WR ok.

Checked WR status browser page for aida01. WR timestamp 0 ! status : 0xA should be 0 .

Checked tcl for SWC WR check and it doesn't check the status. Changed at DL and at GSI and now it does.

so SWC WR check no all fails.

Contacted NH and he found Fatima VME crate with WR source in had been switched off. It was powered on again.

RESET/SETUP , SWC: clocks all ok, ADC all ok, WR all ok.

set aida01 and all odd number  LED thresholds to 5000 and aida02 all even number thresholds to 10000 and activity of statistic Wave Good Events is similar across the system. 

Checked each of the Waveform capture controls pages ( only shows the status registers for ASIC1 ) and all are fine, nothing sticking.

enabled all the ASIC waveforms on aida01 again and it sticks, clear out three ASICs and operate the waveform reset in the System Functions and it restarts and runs.

So a qualified success. Don't enable too many channels at once ?

left running at 18:00 UK time and checked at 21:30 UK time , still operating ok.

 

 

  102   Wed Nov 27 16:33:50 2019 NHNew Bias & Waveforms

HV core (-160 V) now attached to bottom 3 FEEs (with grounded kapton)

DSSDs bias OK, leakages unchanged

No appreciable change in waveforms

Some estimates

aida01/aida05/aida09 VHF noise: 5 channels frequency = 10 MHz ?

All FEEs:
Oscillation at ~25-35 channels = 1.4 to 2 MHz ?
Varies a bit with each FEE.

No obvious issue with upper FEEs unsure why 10 MHz is present - fee09 has been replaced but showed it before as well.

Lower frequency noise on every FEE may be ground noise from e.g. switching PSU?

No more access to S4 to check thoroughly again.

 

 

 

 

Attachment 1: waves_new_odd.png
waves_new_odd.png
Attachment 2: waves_new_even.png
waves_new_even.png
  520   Thu Aug 31 15:24:56 2023 NHNew AIDA MBS PC
The AIDA MBS FDR will be x86l-119 from now on, not x86l-94

the MBS relay and startup scripts will be changed for this
  675   Wed Dec 11 14:46:21 2024 TD, JB, CC, MPMounting of AIDA, bPlast and BB7 for test

Snout assembled for detector position and timing test.

The planned test is to use a BGO with a 22-Na source and the implantation stack to characterise the timing and position of back-to-back 511 keV events (proxy for mock beta decay event).

The implantation stack is assembled as follows:

Upstream ----> AIDA test frame (as a spacer) ----> AIDA DSSSD (3208-3/3208-21/3208-22, Upstream AIDA used in S100 & S181, April 2024) ----> bPlast (Downstream bPlast) ----> BB7 (mounted in test frame connected to AIDA adaptor) ----> Downstream

The AIDA DSSSD condition seems to be the same as when it was dismounted after S181 and it, apart from some specks of dust no bond wires appear to be broken to the naked eye.

BGO signals appear on the scope with a 22-Na source but the response is quite messy.

 

 

17:36

MSL type BB18(DS)-1000 24cm  x 8cm DSSSD upstream position within snout using 8x upstream (23") ribbon cables 

MSL type BB(DS)-1000 positioned downstream of downstream bPlast within snout using 1x downstream (29") ribbon cable. For BB7 'isolated' Right Coupler Kapton PCB replaced by unmodified Right Coupler Kapton PCB.

 

 

 

Attachment 1: 20241211_161837.jpg
20241211_161837.jpg
Attachment 2: 20241211_161833.jpg
20241211_161833.jpg
Attachment 3: 20241211_161830.jpg
20241211_161830.jpg
Attachment 4: 20241211_161820.jpg
20241211_161820.jpg
  537   Thu Mar 14 13:00:23 2024 JB, NH, MA, AM, GAMounting and biasing DSSD 2

new Downstream DSSD2: 3208-2/3208-5/3208-8

Covered with black cloth.

Voltage (V) Current (uA)
10 3.1
20 3.8
30  
40  
50 4.515
60  
70 4.2
80 4.545
90 5.5
100 7.940
110  
120  

Voltage (V)

Current (uA)
10  
20  
30  
40  
50  
60  
70  
100 7.6
110 9.5
120 Breakdown
   
   

Wafer 3 (beam left) shows breakdown issues at 90V... curious

We replace with november beam triple: 3208-3/3208-21/3208-22

We must speak to Micron about why so many 3208 wafers seem to have issues at >90V
It seems unlikely to be other issues

  536   Thu Mar 14 12:13:17 2024 JB, NH, MAMounting and biasing DSSD 1

Upstream bPlast mounted

new Upstream DSSD: 3208-2/3208-5/3208-8

Covered with black cloth.

Voltage (V) Current (uA)
10 3.345
20 3.6
30  
40  
50 4.5ish
60  
70 5.4
80  
90  
100 5.6
110  
120 5.7
   

V-I behaviour found to be nominal.

  38   Mon Apr 1 08:50:35 2019 CA, TDMonday April 1st 2019 - BEAM START
09.28 System wide checks - aida06 global clock failure (check clock status)

                         - aida02 , aida04, aida06 fail calibration (ADC calibration)

                         - SYNC error counter ok

                         - SYNC pulse check ok

                         - 'collect memory information from FEE64 Linux' still returns scripting error

                         - WR decoder test passes

09.33 good event statistics ok, except aida09, aida10 running hot. (attachment 1)

      FEE temperatures ok (attachment 2)

      detector bias/leakage current ok (attachment 3)

09.38 screenshots of NewMerger tab, Merger/Tape HTTPd and NewMerger terminal (attachments 4, 5, 6)

10.05 1.8.L pulser peak spectra (attachment 7)

10.13 1.8.W waveform spectra (attachment 8)

21.13 Beam on

      AIDA DAQ running - writing to file R9 in directory 290319

      System wide checks - same as above

      good event statistics ok - although rates on 9,10,11,12 have dropped (attachment 10)

      ASIC check load performed

      FEE temperatures ok (attachment 9)

      detector bias leakage currents ok (attachment 11)

      Scintillator 41 -> approx 1000 ions/s, 11 second super cycle (beam on 10s, beam off 1s)
      c. 130-150MeV/u 107Ag ions incident on plastic scintillator + AIDA DSSSD

21.36 High energy spectra for aida09 - (attachment 12)

      High energy spectra for aida11 - (attachment 13)

      Low energy spectra - (attachment 14)

      1D hit pattern - (attachment 15)

21.53 ASIC control settings - see for slow comparator threshold, fast comparator threshold (HEC) and fast
      comparator threshold (LEC) (attachment 16)

      NewMerger screenshot - working (attachment 17)

22.00 ASIC options - including directory path to saved ASIC settings (attachment 18)

22.09 beam stop
      TapeServer stopped, restarted not writing to storage
      one complete file produced R9_0 (2.0 GB)
      plus R9_1 (154 MB)

*NOTE FROM 02/04/19 - DEGRADER in front of detector, silver beam not observed, just unidentified light ions.
Attachment 1: 010419_0930_stats.png
010419_0930_stats.png
Attachment 2: 010419_0933_temp.png
010419_0933_temp.png
Attachment 3: 010419_0933_bias.png
010419_0933_bias.png
Attachment 4: 010419_0936_mergertab.png
010419_0936_mergertab.png
Attachment 5: 010419_0937_mergerterm.png
010419_0937_mergerterm.png
Attachment 6: 010419_0938_newmergerterm.png
010419_0938_newmergerterm.png
Attachment 7: 010419_1005_18L.png
010419_1005_18L.png
Attachment 8: 101419_1013_18W.png
101419_1013_18W.png
Attachment 9: 010419_2108_temp.png
010419_2108_temp.png
Attachment 10: 010419_2109_stats.png
010419_2109_stats.png
Attachment 11: 010419_2111_bias.png
010419_2111_bias.png
Attachment 12: 010419_2136_115Haida09.png
010419_2136_115Haida09.png
Attachment 13: 010419_2139_1H_aida11.png
010419_2139_1H_aida11.png
Attachment 14: 010419_lowenergyspectra.png
010419_lowenergyspectra.png
Attachment 15: 010419_1Dhitpatternstat.png
010419_1Dhitpatternstat.png
Attachment 16: 010419_AsicSettings.png
010419_AsicSettings.png
Attachment 17: 010419_newmerger_2150.png
010419_newmerger_2150.png
Attachment 18: 010419_asicsettings.png
010419_asicsettings.png
  426   Mon May 9 08:36:23 2022 TD, NHMonday 9 May
09.32 Re-start AIDA

Attachments 1 & 2 1.8.W spectra 20us FSR

Attachments 3 & 4 1.8.L spectra
 aida10 98ch FWHM
 aida06 384 ch FWHM

Attachment 5 per FEE64 rate spectra

Attachments 6-13 system wide checks - all OK

Attachment 14 ADC data item stats
 n+n FEE64s c. 250k -> c. 370k
 aida01, 7 & 12 c. 200-400k 

Attachment 15 FEE64 temps OK

Attachment 16 DSSD bias & leakage current OK

Attachment 17 per FEE64 sys console status summary



10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
      ASIC check load

Attachments 18 ADC data item stats
 no change to n+n FEE64s, very high p+n FEE64s now at lower rates

Attachments 19 & 20 1.8.W spectra 20us FSR


18.27 all ADC data rates very high
      aida05, asic #1 is again producing HEC data
      aida14, asic #2 producing very little data

Attachments 23 & 24 1.8.W spectra 20us FSR

Attachments 25 & 26 1.8.L spectra
 aida10 165ch FWHM

Attachment 27 per FEE64 rate spectra

Attachment 28 ADC data item stats

Attachment 29 FEE64 temps OK

Attachment 30 DSSD bias & leakage current OK

Attachments 21 & 22 1.8.L spectra
 aida10 65ch FWHM
 aida06 394ch FWHM

Attachment 31 Rate with threesholds at 0x64

14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change. 


14.30 Replace FEE64 aida05
      old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
      old dhcpd.conf file backed up
Attachment 1: Screenshot_from_2022-05-09_09-58-10.png
Screenshot_from_2022-05-09_09-58-10.png
Attachment 2: Screenshot_from_2022-05-09_09-56-53.png
Screenshot_from_2022-05-09_09-56-53.png
Attachment 3: Screenshot_from_2022-05-09_09-54-56.png
Screenshot_from_2022-05-09_09-54-56.png
Attachment 4: Screenshot_from_2022-05-09_09-53-43.png
Screenshot_from_2022-05-09_09-53-43.png
Attachment 5: Screenshot_from_2022-05-09_09-51-41.png
Screenshot_from_2022-05-09_09-51-41.png
Attachment 6: Screenshot_from_2022-05-09_09-50-11.png
Screenshot_from_2022-05-09_09-50-11.png
Attachment 7: Screenshot_from_2022-05-09_09-50-07.png
Screenshot_from_2022-05-09_09-50-07.png
Attachment 8: Screenshot_from_2022-05-09_09-50-02.png
Screenshot_from_2022-05-09_09-50-02.png
Attachment 9: Screenshot_from_2022-05-09_09-48-13.png
Screenshot_from_2022-05-09_09-48-13.png
Attachment 10: Screenshot_from_2022-05-09_09-48-06.png
Screenshot_from_2022-05-09_09-48-06.png
Attachment 11: Screenshot_from_2022-05-09_09-48-00.png
Screenshot_from_2022-05-09_09-48-00.png
Attachment 12: Screenshot_from_2022-05-09_09-47-50.png
Screenshot_from_2022-05-09_09-47-50.png
Attachment 13: Screenshot_from_2022-05-09_09-47-27.png
Screenshot_from_2022-05-09_09-47-27.png
Attachment 14: Screenshot_from_2022-05-09_09-47-21.png
Screenshot_from_2022-05-09_09-47-21.png
Attachment 15: Screenshot_from_2022-05-09_09-47-12.png
Screenshot_from_2022-05-09_09-47-12.png
Attachment 16: Screenshot_from_2022-05-09_09-47-00.png
Screenshot_from_2022-05-09_09-47-00.png
Attachment 17: Screenshot_from_2022-05-09_09-36-29.png
Screenshot_from_2022-05-09_09-36-29.png
Attachment 18: Screenshot_from_2022-05-09_10-16-16.png
Screenshot_from_2022-05-09_10-16-16.png
Attachment 19: Screenshot_from_2022-05-09_10-17-40.png
Screenshot_from_2022-05-09_10-17-40.png
Attachment 20: Screenshot_from_2022-05-09_10-17-09.png
Screenshot_from_2022-05-09_10-17-09.png
Attachment 21: Screenshot_from_2022-05-09_10-32-52.png
Screenshot_from_2022-05-09_10-32-52.png
Attachment 22: Screenshot_from_2022-05-09_10-31-48.png
Screenshot_from_2022-05-09_10-31-48.png
Attachment 23: Screenshot_from_2022-05-09_18-16-39.png
Screenshot_from_2022-05-09_18-16-39.png
Attachment 24: Screenshot_from_2022-05-09_18-16-01.png
Screenshot_from_2022-05-09_18-16-01.png
Attachment 25: Screenshot_from_2022-05-09_18-15-32.png
Screenshot_from_2022-05-09_18-15-32.png
Attachment 26: Screenshot_from_2022-05-09_18-14-56.png
Screenshot_from_2022-05-09_18-14-56.png
Attachment 27: Screenshot_from_2022-05-09_18-13-43.png
Screenshot_from_2022-05-09_18-13-43.png
Attachment 28: Screenshot_from_2022-05-09_18-13-26.png
Screenshot_from_2022-05-09_18-13-26.png
Attachment 29: Screenshot_from_2022-05-09_18-13-16.png
Screenshot_from_2022-05-09_18-13-16.png
Attachment 30: Screenshot_from_2022-05-09_18-13-03.png
Screenshot_from_2022-05-09_18-13-03.png
Attachment 31: AnyDeskMSI_2022-05-09_21-45-12.png
AnyDeskMSI_2022-05-09_21-45-12.png
  181   Mon Mar 8 17:02:38 2021 CAMonday 8th March 18:00 - 00:00
18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

18:24 Rate spectra - attachment 4

18:30 Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

19:28 beam back after a brief outage - AIDA on file R9_736

      FRS having DAQ issues

19:57 FRS DAQ/Scalars back

20:12 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

      Merger ok - 4.5M data items/s

      TapeServer ok - 46MB/s

      Data forwarding to MBS ok

20:33 No beam - AIDA on file R9_821

21:08 rates spectra - attachment 8

22:35 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 9
      FEE Temperatures - attachment 10
      Bias and leakage currents ok - attachment 11

      Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

      temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload.

23:07 media/ThirdDrive at 65% capacity

      writing to file R9_1021

      file compression currently up to R1_686

23:36 beam stable ~1e9 pps

      rate spectra - attachment 12




      
Attachment 1: Screenshot_from_2021-03-08_17-05-44.png
Screenshot_from_2021-03-08_17-05-44.png
Attachment 2: Screenshot_from_2021-03-08_17-07-04.png
Screenshot_from_2021-03-08_17-07-04.png
Attachment 3: Screenshot_from_2021-03-08_17-08-51.png
Screenshot_from_2021-03-08_17-08-51.png
Attachment 4: Screenshot_from_2021-03-08_17-23-58.png
Screenshot_from_2021-03-08_17-23-58.png
Attachment 5: Screenshot_from_2021-03-08_19-10-56.png
Screenshot_from_2021-03-08_19-10-56.png
Attachment 6: Screenshot_from_2021-03-08_19-11-33.png
Screenshot_from_2021-03-08_19-11-33.png
Attachment 7: Screenshot_from_2021-03-08_19-11-53.png
Screenshot_from_2021-03-08_19-11-53.png
Attachment 8: Screenshot_from_2021-03-08_20-08-12.png
Screenshot_from_2021-03-08_20-08-12.png
Attachment 9: Screenshot_from_2021-03-08_21-28-29.png
Screenshot_from_2021-03-08_21-28-29.png
Attachment 10: Screenshot_from_2021-03-08_21-29-08.png
Screenshot_from_2021-03-08_21-29-08.png
Attachment 11: Screenshot_from_2021-03-08_21-32-35.png
Screenshot_from_2021-03-08_21-32-35.png
Attachment 12: Screenshot_from_2021-03-08_22-38-41.png
Screenshot_from_2021-03-08_22-38-41.png
  179   Mon Mar 8 07:02:17 2021 CA, LSMonday 8th March 08:00 - 17:00
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude 1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

08:20 Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

      Writing to file R6_1148

09:00 Beam off for optimisation - writing to file R6_1212

      Expected back at 09:30

09:30 Beam will be off for a bit longer - an hour or two

09:34 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
      Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds 
to finish
      Something confusing, will look at data and report to VP
      Reverted to original merger and back... 45 MB/s
      R9

12.00 (LS)
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment7)
      FEE temps (attatchment8)
      Rate spectra (attachment9), beam still down
      Leakage Currents written to sheets (attachment10)
      Merger ~45M items/s
      Tapeserver ~43MB/s

14.00 Still no beam but is expected to return soon
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment11)
      FEE temps (attatchment12)
      Leakage Currents written to sheets (attachment13)
      Merger ~45M items/s
      Tapeserver ~44MB/s

14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)

15.00 Beam is back (attachment15)
      Still writing to R9 (roughly around R9_386)

15.28 No beam (attachment16)

15.41 Beam back (attachment17)
      Roughly R9 (roughly around R9_439)

16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
 
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment19)
      Spectra Rate (attachment20)
      FEE temps (attatchment21)
      Leakage Currents written to sheets (attachment22), 
      still rising but looking on Grafana 7 day period should start to lower around 17.00
      Merger ~45M items/s
      Tapeserver ~43MB/s

16.55 Beam down (attachment23), back a couple minutes later

15.51 beam down (attachment24)
      
Attachment 1: Screenshot_from_2021-03-08_07-04-21.png
Screenshot_from_2021-03-08_07-04-21.png
Attachment 2: Screenshot_from_2021-03-08_07-07-24.png
Screenshot_from_2021-03-08_07-07-24.png
Attachment 3: Screenshot_from_2021-03-08_07-07-49.png
Screenshot_from_2021-03-08_07-07-49.png
Attachment 4: Screenshot_from_2021-03-08_08-31-37.png
Screenshot_from_2021-03-08_08-31-37.png
Attachment 5: Screenshot_from_2021-03-08_08-32-58.png
Screenshot_from_2021-03-08_08-32-58.png
Attachment 6: Screenshot_from_2021-03-08_08-34-25.png
Screenshot_from_2021-03-08_08-34-25.png
Attachment 7: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 8: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 11: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 12: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 13: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 14: 1500_Grafana_LeakageCurrents.PNG
1500_Grafana_LeakageCurrents.PNG
Attachment 15: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 16: 1528_SpectraRate.PNG
1528_SpectraRate.PNG
Attachment 17: 1541_SpectraRate.PNG
1541_SpectraRate.PNG
Attachment 18: 1600_R9_469_hit_rates.PNG
1600_R9_469_hit_rates.PNG
Attachment 19: 1620_Statistics.PNG
1620_Statistics.PNG
Attachment 20: 1620_SpectraRate.PNG
1620_SpectraRate.PNG
Attachment 21: 1620_FEETemps.PNG
1620_FEETemps.PNG
Attachment 22: 1620_LeakageCurrents.PNG
1620_LeakageCurrents.PNG
Attachment 23: 1655_SpectraRate.PNG
1655_SpectraRate.PNG
Attachment 24: 1749_SpectraRate.PNG
1749_SpectraRate.PNG
  178   Sun Mar 7 23:13:50 2021 OHMonday 8th March
00:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


00:14 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4

02:15 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped

04:00 Noticed more bad merge events
      System wide checks has also noted an FPGA error.
      All other system wide checks same results as previous

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A3F29 BA0DF4D2 , WR/10=>   23DD31DC5CE3215, Readout Time =>   23DD31DC6F34000 
aida02 : White Rabbit=>  166A3F29 CC49ECD7 , WR/10=>   23DD31DC7A0FE15, Readout Time =>   23DD31DC8428000 
aida03 : White Rabbit=>  166A3F29 DF55B7E8 , WR/10=>   23DD31DC9889264, Readout Time =>   23DD31DCA85C000 
aida04 : White Rabbit=>  166A3F29 EECF82B7 , WR/10=>   23DD31DCB14C045, Readout Time =>   23DD31DCC2B8000 
aida05 : White Rabbit=>  166A3F29 FED2EFD2 , WR/10=>   23DD31DCCAEB195, Readout Time =>   23DD31DCDC14000 
aida06 : White Rabbit=>  166A3F2A 0EDC023D , WR/10=>   23DD31DCE49336C, Readout Time =>   23DD31DCF204000 
aida07 : White Rabbit=>  166A3F2A 1E8170D3 , WR/10=>   23DD31DCFD9BE7B, Readout Time =>   23DD31DD10A0000 
aida08 : White Rabbit=>  166A3F2A 30448F76 , WR/10=>   23DD31DD1A074BF, Readout Time =>   23DD31DD2A34000 
aida09 : White Rabbit=>  166A3F2A 3FE36E61 , WR/10=>   23DD31DD33057D6, Readout Time =>   23DD31DD3B94000 
aida10 : White Rabbit=>  166A3F2A 4E83A1F7 , WR/10=>   23DD31DD4A6C365, Readout Time =>   23DD31DD46A8000 
aida11 : White Rabbit=>  166A3F2A 5FD19086 , WR/10=>   23DD31DD661C1A7, Readout Time =>   23DD31DD7664000 
aida12 : White Rabbit=>  166A3F2A 6FE06F3D , WR/10=>   23DD31DD7FCD7EC, Readout Time =>   23DD31DD9684000 

04:07 Beam is back AIDA on file R6_815
      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10
      ASIC check ok

      Multiple merge data error events observed

06:04 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 11
      FEE Temperatures - attachment 12
      Bias and leakage currents ok - attachment 13

07:22 Beam noticed dropped but back quickly
Attachment 1: 210308_0014_Stats.png
210308_0014_Stats.png
Attachment 2: 2303108_0015_Temp.png
2303108_0015_Temp.png
Attachment 3: 210308_0016_Bias.png
210308_0016_Bias.png
Attachment 4: 210308_0120_System.png
210308_0120_System.png
Attachment 5: 210308_0213_Stats.png
210308_0213_Stats.png
Attachment 6: 210308_0213_Temp.png
210308_0213_Temp.png
Attachment 7: 210308_0215_Bias.png
210308_0215_Bias.png
Attachment 8: 210308_0403_Stats.png
210308_0403_Stats.png
Attachment 9: 210308_0404_Temp.png
210308_0404_Temp.png
Attachment 10: 210308_04011_Bias.png
210308_04011_Bias.png
Attachment 11: 210308_0602_Stats.png
210308_0602_Stats.png
Attachment 12: 210308_0603_Temp.png
210308_0603_Temp.png
Attachment 13: 210308_0604_Bias.png
210308_0604_Bias.png
  566   Mon Apr 8 16:39:00 2024 JB, CC, TDMonday 8 April
17.32 Power and detector bias cycle

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2
       *except* aida02 ASIC temp u/s

      ASIC settings 2024Mar27-11.25.32
       slow comparator p+n FEE64s 0xa (100keV), n+n FEE64s 0xf (150keV)

      BNC PB-5 pulser
      amplitude 10.0V
      attenuation x10
      frequency 25Hz
      polarity -
      tau_d 1ms
      tail pulse

      test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

      test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

      All system wide checks OK *except* aida02 & aida03 WR decoder status errors - attachment 10

      WR timestamps OK - attachment 11

      ADC data item stats - attachment 3
       12 of 16 < 20k, all < 100k

      per FEE64 Rate spectra - attachment 4

      per n+n FEE64 1.8.L spectra - attachment 5

      1.8.W spectra - 20us FSR - attachments 6-9
       preamplifier output noise generally very good

*Current ground configuration*

CAEN N1419ET LK fitted (non floating outputs)

ground cable from aida04-aida12-aida02-aida09-aida01-aida05
HV#0 aida12-aida03-aida15

ground cable from aida08-aida16-aida06-aida10-aida14-aida13
HV#1 aida16-aida07-aida11

LK3 fitted aida03, aida07

LK1 fitted aida02, aida04, aid06, aida08

test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

all DSSSD ribbon cable drain wires grounded to their resepctive AIDA adaptor PCBs

JB observed open circuit between AIDA Al snout and AIDA support frame ( as expected )

AIDA PSU cabling as reported https://elog.ph.ed.ac.uk/DESPEC/560

Snout configuration:

- all bPlast cables are disconnected and floating. 
- CC taped up all the cables and exit points in snout.
- CC, JB covered the exit of the snout with two additional layers of aluminium foil which were also taped shut.


To Do:

- repower detector, see if similar conditions persist.
- disconnect test - pulser, again see if conditions persist.

- aida04 asic#1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?




       
Attachment 1: Screenshot_from_2024-04-08_17-32-49.png
Screenshot_from_2024-04-08_17-32-49.png
Attachment 2: Screenshot_from_2024-04-08_17-33-10.png
Screenshot_from_2024-04-08_17-33-10.png
Attachment 3: Screenshot_from_2024-04-08_17-33-14.png
Screenshot_from_2024-04-08_17-33-14.png
Attachment 4: Screenshot_from_2024-04-08_17-33-33.png
Screenshot_from_2024-04-08_17-33-33.png
Attachment 5: Screenshot_from_2024-04-08_17-34-40.png
Screenshot_from_2024-04-08_17-34-40.png
Attachment 6: Screenshot_from_2024-04-08_17-35-00.png
Screenshot_from_2024-04-08_17-35-00.png
Attachment 7: Screenshot_from_2024-04-08_17-35-19.png
Screenshot_from_2024-04-08_17-35-19.png
Attachment 8: Screenshot_from_2024-04-08_17-37-28.png
Screenshot_from_2024-04-08_17-37-28.png
Attachment 9: Screenshot_from_2024-04-08_17-37-56.png
Screenshot_from_2024-04-08_17-37-56.png
Attachment 10: Screenshot_from_2024-04-08_17-48-23.png
Screenshot_from_2024-04-08_17-48-23.png
Attachment 11: Screenshot_from_2024-04-08_17-48-49.png
Screenshot_from_2024-04-08_17-48-49.png
  359   Mon Jun 7 10:33:48 2021 TDMonday 7 June
11.30 DAQ continues OK - file S496/R32_144
      alpha background

      All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      5     11      6      3      4      2      2      1      4      7   : 40572
aida02 :      1      3      6      7      2      3      3      2      1      4      7   : 40540
aida03 :     21      7      7      8      2      4      1      3      2      3      7   : 39804
aida04 :     13      7      5      6      1      3      2      3      2      3      7   : 39740
aida05 :      2      8      6      5      1      2      2      3      1      4      7   : 40584
aida06 :     10      4      5      5      4      3      1      3      1      4      7   : 40632
aida07 :      9      3      4      7      3      3      3      2      1      4      7   : 40604
aida08 :     11      2      6      4      4      3      3      2      1      4      7   : 40604
aida09 :     10      2      5      4      2      2      2      3      1      4      7   : 40584
aida10 :     14      5      8      6      2      1      1      3      1      4      7   : 40352
aida11 :      8      6      9      5      4      4      3      3      2      3      7   : 40320
aida12 :     10      5      3      7      3      3      2      2      1      4      7   : 40352
aida13 :      7      5      4      6      5      4      3      3      2      3      7   : 40324
aida14 :      7      5      4      8      4      4      3      3      2      3      7   : 40324
aida15 :      8      4      7      4      2      2      2      4      1      4      7   : 41136
aida16 :     13      5      7      6      3      4      3      3      2      3      7   : 40268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

11.35 analysis of file S496/R32_143 - attachment 9
       max deadtime 0.29% aida03
Attachment 1: Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-07_Statistics_aidas-gsi.png
Screenshot_2021-06-07_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 986.png
986.png
Attachment 8: 987.png
987.png
Attachment 9: R32_143
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   57707988 (   21988.9 Hz)
        Other data format:  204212013 (   77812.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.0 Hz)
                               RESUME:         85 (       0.0 Hz)
                              SYNC100:      41772 (      15.9 Hz)
                              WR48-63:      41772 (      15.9 Hz)
                           FEE64 disc:    1109863 (     422.9 Hz)
                             MBS info:  203018436 (   77357.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2624.418 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2432.488
  2                0.665                0.000
  3                7.525                0.000
  4                0.000                0.000
  5                0.000             1464.970
  6                0.000                0.000
  7                0.000             1288.051
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        112          0          0          0          0          0          0          0          0          0          0          0
  1    1940572        788          0          0          0          0        394        394          0          0          0          0
  2        442  101676817          0          0         30         30      17294      17294          0  101642169          0          0
  3   40857770  102528388          0          0         55         55      21074      21074    1109863  101376267          0          0
  4        129          0          0          0          0          0          0          0          0          0          0          0
  5    7006930       2878          0          0          0          0       1439       1439          0          0          0          0
  6        548          0          0          0          0          0          0          0          0          0          0          0
  7    7898329       3142          0          0          0          0       1571       1571          0          0          0          0
  8        285          0          0          0          0          0          0          0          0          0          0          0
  9         71          0          0          0          0          0          0          0          0          0          0          0
 10       1135          0          0          0          0          0          0          0          0          0          0          0
 11        122          0          0          0          0          0          0          0          0          0          0          0
 12        679          0          0          0          0          0          0          0          0          0          0          0
 13         92          0          0          0          0          0          0          0          0          0          0          0
 14        666          0          0          0          0          0          0          0          0          0          0          0
 15        106          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.551s ( 1082.882 blocks/s,  67.680 Mb/s)
ELOG V3.1.3-7933898