Initial results: copy from DESPEC elog.

We have managed to obtain the time difference between the bPlast White rabbit and the AIDA Fast time. We had to gate out the zero fast-time events and then also condition that we only take the data from aida10. We see this in 684/1. Then if we look at the time difference between the fast time and the bPlast WRT we see a sharp peak at zero, this makes sense as the AIDA fast time discriminator is being triggered by a pulser which is then being used as a trigger for bPlasts DAQ so these events should be virtually arriving without delay. The delay we do see is infact around 750 ns.

The data collected overnight was also analysed as shown in 684/3 this is the data accumulated with the 22Na source close to the snout and a global threshold in the fast discriminator of 0x0f. The centre peak was roughly fitted with a gaussian:    

  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  Constant     1.21277e+03   2.74526e+01   5.66915e-01  -5.61074e-07
   2  Mean        -1.78513e+02   8.50699e+00   1.62947e-01   1.58554e-06
   3  Sigma        3.48571e+02   4.03993e+00   6.26343e-05  -1.86863e-03

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

Yesterday we spent time essentially performing a dry run to get AIDA DAQ and bPlast into the time sorter, there were some issues with the AIDA mbs PC x86l-119 which was related to some boot issue, it was booting to a newer version of debian while the Relay for AIDA to MBS is on an older version (scratch) (the machine is quite old). We spent a bit of time getting the thresholds correct for each of the ASICs. In the end we just elected to have a blanket level of 0x0f for all of the ASICs when we removed the pulser.

Last night we left the setup with the following thresholds on all of the FEEs to collect data. 681/1. The pulser was also turned off and the 22Na source was moved close to the snout.

This morning c. 9:25 we returned and checked the temperatures, HV and statistics and everything seems to be ok. The only problem was that the timesorter has crashed this morning at 8 am.

AIDA DSSSDs and FEE64-DSSSD cabling are electrically isolated (by design) from snout, mounting rods and support assembly.
 FEE64-DSSSD ribbon cables screened by 3M 1245 Cu foil, drain wire to FEE64 adaptor PCB

AIDA FEE64s are electrically isolated (by design) from the AIDA support frame and each other.

BNC PB-5 test signal 
 daisy chained to FEE64 adaptor PCBs via screened RG174 cable/Lemo 00-250 

CAEN N1419ET DSSSD bias
 daisy chained to 1x p+n junction FEE64 adaptor PCB/Si wafer via screened RG174 cable/Lemo 00-250
 CAEN N1419ET outputs floating, locally grounded by 1x n+n Ohmic FEE64 adaptor PCB link/DSSSD

AIDA FEE64s are connected to the DESPEC 19" rack
 power (AIDA FEE64 PSUs),
 timestamp (AIDA NIM MACB modules),
 system console (via serial-USB cables to USB hubs),
 RJ45 network cables (24 port Gbit network switch)

AIDA PSU PowerStax MS1U-6C-222233-01 - see attachments 1-2
 PSU outputs V+/V- and common 
 no direct ground connection

AIDA MACB
 HDMI cabling MACB-FEE64 is screened and shield is grounded

system console
 ?

RJ45 network cables are coupled by ferrite beads to FEE64 PCB and network switch
 no direct ground connection

The AIDA downstream window ('endcap' of snout) consists of 2x 50um aluminized mylar sheets

Attachment 1 - top aida06

Attachment 2 - top aida01

Attachment 3 - top aida-gsi (AIDA server)

Attachment 4 - TapeServer

Attachment 5 - NewMerger stats block 6 (aida06)

Attachment 6 - NewMerger stats block 1 (aida01)

Attachment 7 - NewMerger stats block 0 (all FEE64s)

Attachment 8 - NewMerger options

Attachment 9 - NewMerger link stats

Attachment 10 - NewMerger

Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats

Attacgment 15 - Network traffic

Adaptor PCB link settings

n+n FEE64s

aida02, 4, 6 & 8 - LK1 fitted, HV Lemo-00 not connected - left/right wafer n+n side bias ring -> gnd

p+n FEE64s

   top - aida09, 1, 10 & aida13, 5, 14 - LK2 & LK4 fitted - front/back FP -> gnd
bottom - aida11, 3, 12 & aida15, 7, 16 - LK2 & LK4 fitted - front/back FP -> gnd

   top - aida01, 5 - LK3 fitted - middle wafer n+n side bias ring -> gnd
bottom - aida03, 7 - LK3 fitted - DSSSD PCB gnd -> gnd

AIDA PSU

   top 1-2, 3-4, 9-10, 11-12
bottom 5-6, 7-8, 13-14, 15-16


ASIC settings 2021Apr29-13-16-00
     slow comparator 0x64 -> 0xa

BNC PB-5 settings
    amplitude 1.000V
    attenuation x1
    tail pulse
    tau_d 1ms
    polarity +
    frequency 25Hz

AIDA analysis and presentation by J. Bormans for AIDA made in the HISPEC-DESPEC collaboration meeting.

https://docs.google.com/presentation/d/1hlZ30r294UqbVKGy3jg-wompFpc7TSfD/edit?usp=sharing&ouid=102131181856760114019&rtpof=true&sd=true

Noise seemed less strong today, unsure of cause. I had re-taped snout as tape was not sticking well.

Figures 1-4: Waves for all 4 FEEs connected to the DSSD
Figures 5-8: Zoomed in on region 400-600
Figures 9-10: Rates in DSSD

Note in particular fee09 has a clear noise pickup somewhere. Also appears when no DSSD voltage.
Peak distance = 6 channels = 16.6 MHz.

Fee09 is connected to HV bias core (-160 V)
However unplugging the HV cable from it shows no difference.

Must be related to something else.

Update 16:29 CEST:
Distance is more like 5 channels (20 MHz)
Fee09 picks it by far most strongly but all fees you can make it out
Additionally can see a slower waveform especially on other fees... approximately 40 channels = 2.5 MHz

Checked with multimeter that low resistance path between FEE cooling plate and ground LEMO connector
also low resistance between ribbon cable copper shield and ground LEMO connector.

Noise seems to appear even if pulser cable unplugged at pulser end or when attentuation is increased weaking pulse.

The AIDA water system has been repaired and not leaking.

The interlock did not light immediately, but power cable reconnected to box and now it works.

The snout endcap has been sealed with new copper tape over the red tape

Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive



-

the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s

-

Preliminary alpha background analysis (over weekend, 90 hours statistics)

Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Apr 28 19:50:20 2022
Last event : Mon May  2 14:04:03 2022
Duration   : 324823 seconds
--------------------------------------
MBS Events   (M):     912899 (2 Hz)
AIDA Words   (W): 7472078760 (-3441 Hz)
ADC Words    (A): 2742871865 (-4778 Hz)
-- Decays    (D): 2728717223 (-4821 Hz)
-- Implants  (I):   14154642 (43 Hz)
Info Words   (F): 4729206895 (1 kHz)
-- SYNC48    (S):    2093328 (6 Hz)
-- SYNC63    (s):    2093328 (6 Hz)
-- PAUSE     (P):    3823161 (11 Hz)
-- RESUME    (R):    3823161 (11 Hz)
-- Discrim   (D): 1390520565 (4 kHz)
-- Scaler    (C): 3326853352 (3 kHz)
-- Unknown   (U):          0 (0 Hz)
Timewarps    (T):         20 (0 Hz)
Dead Time    (X): 5024.43 s (1.54682%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
   1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
   2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
   3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
   4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
   5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
   6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
   7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
   8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
   9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
  10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
  11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
  12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
  13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
  14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
  15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
  16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9

PAUSE/RESUME stats are dominated by 2x FEE64s - aida07 and aida16

These FEE64s also have the highest rates of discriminator data?

> The snout endcap has been sealed with new copper tape over the red tape
> 
> Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
> It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive
> 
> 
> 
> -
> 
> the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
> When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s
> 
> -
> 
> Preliminary alpha background analysis (over weekend, 90 hours statistics)
> 
> Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04
> 
> ======================================
> AIDA Unpacker Analysis
> --------------------------------------
> First event: Thu Apr 28 19:50:20 2022
> Last event : Mon May  2 14:04:03 2022
> Duration   : 324823 seconds
> --------------------------------------
> MBS Events   (M):     912899 (2 Hz)
> AIDA Words   (W): 7472078760 (-3441 Hz)
> ADC Words    (A): 2742871865 (-4778 Hz)
> -- Decays    (D): 2728717223 (-4821 Hz)
> -- Implants  (I):   14154642 (43 Hz)
> Info Words   (F): 4729206895 (1 kHz)
> -- SYNC48    (S):    2093328 (6 Hz)
> -- SYNC63    (s):    2093328 (6 Hz)
> -- PAUSE     (P):    3823161 (11 Hz)
> -- RESUME    (R):    3823161 (11 Hz)
> -- Discrim   (D): 1390520565 (4 kHz)
> -- Scaler    (C): 3326853352 (3 kHz)
> -- Unknown   (U):          0 (0 Hz)
> Timewarps    (T):         20 (0 Hz)
> Dead Time    (X): 5024.43 s (1.54682%)
> ======================================
> ======================================
> Individual FEE Overview
> ======================================
>  FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
>    1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
>    2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
>    3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
>    4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
>    5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
>    6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
>    7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
>    8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
>    9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
>   10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
>   11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
>   12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
>   13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
>   14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
>   15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
>   16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9

The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible

01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.

For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)

The system has been powered up and biased successfully. To be monitored over the next few days.

A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.

A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...

The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.

Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
> 
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
> 
> The system has been powered up and biased successfully. To be monitored over the next few days.
> 
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
> 
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
> 
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
> 
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.

The hose has arrived and the water line will be prepared in my office mid next week.

All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)

A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.

The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.

There will be a DESPEC Dry Run next week Mon-Tue.

FEEs have been on this morning, temperatures attached

Leakage current attached

the DSSD is attached to the following AIDA modules:
Beam Left: 10
Beam Right: 12
Bottom: 11
Top: 9

MIDAS Servers on, will be looking at resolution (pulser)

AIDA and the Snout have been remounted in S4

I had a check of the AIDA connectors and all looked good, the kapton PCB connectors were sitting comfortably above the interstage and I saw no stressed/loose connectors.
Multimeter showed no continuity between snout or inter-stage and AIDA ground

We put a single layer of wide black tape inside the end-cap to improve the light seal and minimise reflections

After mounting the snout I managed to bias both DSSDs to 100 V with no issues. I had to raise the DSSD#1 (Ch0) limit to 40 uA because it rises quite high during ramp up (then settles).

It is very hot in S4 (28 C) and the roof is closed for high-radiation experiments, so I expect it to get hotter. DSSD leakage currents were similar to before (20 and 15 uA)

There is also the new Temp and Humidity monitor, which you can view on AnyDesk at 192.168.11.119 (I think), there is a firefox window open already 

The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system

Currently

- 1 DSSD and Plastic is in the AIDA snout, which is not mounted

- The FEEs are offline due to a water supply issue, this will be repaired early 21/03/2019

- The pis & aida-3 are online

The water interlock also didn't clear, which will be investigated

Ideally installation can be completed tomorrow but this depends on water

The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.

(All data below taken with thresholds 0xA for all)

Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"

Leakage currents high but S4 very warm at the moment as roof is still closed.

Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.

Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done

Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)

aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3