AIDA snout length 37cm (from end of AIDA snout support assembly)

downstream scintillator (assembly depth 1cm)   36cm

downstream dsssd (#1)                          35.2cm

                 (#2)                          34.3cm

                 (#3)                          33.4cm

upstream dsssd   (#4)                          32.5cm

upstream scintillator (assembly depth 1cm)     31.7cm

Cable lengths required are therefore

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 22.5 = 54.35cm 21.4"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 16.5 = 47.45cm 18.7"

                 (#3)           33.4 - 10.5 + 2.15 + 5 + 10.5 = 40.55cm 16.0"

upstream dsssd   (#4)           32.5 - 10.5 + 2.15 + 5 + 4.5 = 33.65cm 13.2"

length of RH Kapton PCB coupler = 10.5cm (Carlo can you check please?)

turning the corner at the base of the AIDA snout support assembly 2.15+5=7.15cm
(see Peter's drawing)

distances to FEE64s 4.5, 10.5, 16.5 and 22.5cm
(see Peter's drawing)

f we use 3x DSSSDs then the cable lengths required will be as follows:

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 16.5 = 48.35cm 19.0"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 10.5 = 41.45cm 16.3"

upstream dsssd   (#3)           33.4 - 10.5 + 2.15 + 5 + 4.5 = 34.55cm 13.6"

https://docs.google.com/spreadsheets/d/1fZBybHodSvacUdKpDM_VKznqTrh49CKnqKVQVMEDWGs/edit?usp=sharing

A spreadsheet of ambient conditions and water temperature in S4 for AIDA.
Will show the distance to danger points (interlock shutdown and condensation on the pipes)

Included waves for all 12 FEEs

FEES 1, 5, 9 all show this HF noise on it, and are all located at the top of the AIDA mount.
Connections: HV from DSSD (-ve) is in FEE9
Pulser input is in FEE9 (non-inverted)

Resistance between FEE cooling plate and LEMO connector for all 3 FEEs is a few Ohms, as is resistance between copper braid on DSSD cable and the LEMO connector.

All FEEs show some noise but mainly those 3 and all four with DSSD. Will try to check DSSD connections in case something is messed up.

FEE layout for reference: (Beam's perspective)

                              FEE9
                              FEE5
                              FEE1

FEE10 FEE6 FEE2                FEE4 FEE8 FEE12

                              FEE3
                              FEE7
                              FEE11

Noise seemed less strong today, unsure of cause. I had re-taped snout as tape was not sticking well.

Figures 1-4: Waves for all 4 FEEs connected to the DSSD
Figures 5-8: Zoomed in on region 400-600
Figures 9-10: Rates in DSSD

Note in particular fee09 has a clear noise pickup somewhere. Also appears when no DSSD voltage.
Peak distance = 6 channels = 16.6 MHz.

Fee09 is connected to HV bias core (-160 V)
However unplugging the HV cable from it shows no difference.

Must be related to something else.

Update 16:29 CEST:
Distance is more like 5 channels (20 MHz)
Fee09 picks it by far most strongly but all fees you can make it out
Additionally can see a slower waveform especially on other fees... approximately 40 channels = 2.5 MHz

Checked with multimeter that low resistance path between FEE cooling plate and ground LEMO connector
also low resistance between ribbon cable copper shield and ground LEMO connector.

Noise seems to appear even if pulser cable unplugged at pulser end or when attentuation is increased weaking pulse.

Figures of intense high frequency noise pickup in AIDA.

Will investigate source in S4.

Edit: Period is approx 40 cycles / approx 2.5 MHz (?)

AIDA Finally working again so can test VP's added WAVE histogramming mode.

Fig 1. Waveforms for FEE12 ASIC 1. Sample rate = 10, Threshold = 10500
n.b. lots of "blank" wave channels for unknown reason, all low-energy channels show pulser so it's not a DSSD signal problem?

Fig 2. Pulser spectrum for FEE12 ASIC 1. (10 peaks 0.1 -> 1 V)

Fig 3. Waveforms for FEE10 ASIC 3.

n.b. WAVE channels frequently stopped updating, needed start/stop DAQ or restart WAVE readout option. Unsure if it's related to high rate/odd signals.
Might be related to issue in FEE12 too. WAVE good events statistics lower than empty FEEs.
All rates quite high, looks quite noisy at the moment.

Fig 4. Pulser of FEE10 ASIC 3.

Note that FEE10.3 has poor peaks, in WAVE we see:
Some normal looking pulses which are merely noiser than other channels.
Some pulses which look very odd (baseline really low, etc).

Will recheck ground for noise issues, but the huge peaks are confusing to me. (3.5.W, etc)

The other 6 FEEs were powered on one at a time to check no issues. None found.
All 12 FEEs plugged back into the PSU.

Note: The FEE temperatures reported on Friday were before a Reset/Setup of the DAQ, which means the ASICs run very fast and the hence run hotter.
Reset/Setup shows more sustainable temps (attached)

These temps are similar to those observed in Sep 2018 but warmer by ~4 C than those observed Dec-Jan and April.

The water issue has been resolved - merely incorrect temperature guages (replaced). Water is 20 C as required.

S4 conditions at 13:12 CEST

26.5 C / 47.2 % / Td = 14.3 C

Based on email by TD doing the following procedure to  ensure FEEs are OK after water issue

1) disconnect the power cables of *all* FEE64s at the FEE64 PSUs
2) re-connect power cable, power-up and check/test the FEE64s
  *one at a time* until stable operatinmg temperature is achieved
... say 30mins?

13:34 CEST: All be FEE1 unplugged. Powering on!

Vertex temp read slightly warm (67 C) so powered off. Waiting for a while for water to circulate since off for a while...

15:01 CEST: FEE1 Powered on again, Vertex reading 67 C again but it's stable and slowly going down. Watching and seeing but I think it's OK.
15:35 CEST: FEE1 has been stable around 67 C all the time. Other temps fine. Presuming this isn't a major concern. (Perhaps related to reasonably warm/humid conditions?)

15:47 CEST: FEE2 Powered on.
16:02 CEST: Temperature stable, not exceeded 60 C.

16:07 CEST: FEE3 Powered On.
16:17 CEST: Temperature stable, peaked at 58.44 C

16:21 CEST: FEE4 Powered On.
16:40 CEST: Both FPGA and ASIC reading warm: 65.5 C and 58 C. Seeming to not be rising anymore.

16:41 CEST: FEE5 Powered On.
16:53 CEST: Temperature stable, at 64 C

16:56 CEST: FEE6 Powered On.
17:12 CEST: FPGA temp is ok but ASIC warm. Tmax ASIC = 56.88, FPGA = 55.38, PSU = 27.44

17:15 Turning FEEs and Water off for the weekend
Will test 7-12 on Monday.
Might be cooler in S4.

Cooling seems to be working but temps hotter than before.

Temp: 26 C
Humidity: 50%
Dew Point: ~15 C

There are two temperature dials outside S4, one shows exactly 20 C and one shows nearer 16 C.

Trying to clarify this discrepancy as 16 C is too cold at the moment.

In contact with developer of cooling system to confirm apparent 16 C after the heat exchanger

Today's conditions:

Temp: 26.9 C
Humidity: 48.7%
Dewpoint 15.2 C

S4 Conditions:

25.2 C, 35.3% RH, Td = 8.8 C

Water temperature: 20 C

-

Rotating the dewpoint sensor on the pipe seems to have made it switch over to solid red = humidity OK.
Waiting confirmation to see if we should check a FEE (as per PJCS suggestion) before water is turned back on.
Will check the outside copper after lunch.

13:04 : Copper seems warm and dry, humidity sensor still solid red.

It occurs to to wonder if there is now water inside the modules. If the metal of the cooling plates is damp on the outside then it is safe to assume there is water on the inside too . Can I suggest that one  of the modules is dismounted, disassembled and checked?

Patrick

Water is still off.

Checked dewpoint sensor still flashing red light (indicating an error). Pipes warmed. Not what I expected. Possible fault?

I checked S4's ambient temp/humidity with a home-brought sensor: Reports 27 C/50% RH (Dew Point = 16 C)
Water temperature is set to exactly 20 C (there's a second dial showing ~18 C but I am unsure if that's the same water supply, I will ask)

Checking the dewpoint PSU with a multimeter shows a voltage of 16 V.
The PSU itself is rated for 12 V output, and the humidity sensor expects 24 V.
Regardless of which voltage 16 V seems incorrect. Might be causing issues.

Humidity is a bit lower now (4pm) the sensor has been in all day, 28 C/35% RH (Dew Point = 11 C) which isn't an issue.

Water is still off until I am sure what to do, or until the dewpoint sensor stops complaining.

The humidity sensor hasn't reported humdities under 90% yet. In order to try and help dry the pipes and turn AIDA the water has been turned off the weekend.
Sunday temperatures might reach 40 C which means it may struggle anyway.

Will consider trying to get a temperature/humidity monitor for S4 to check if the environment is fine too.

Currently AIDA's interlock is off (no lights) and hence AIDA cannot be powered on. Under investigation - power supply has been replugged but seems ok.

Update: It seems the interlock is correctly stopping AIDA from being powered on due to high humidity on the cooling lines. Am currently trying to verify the temperature with GSI technical staff

The latest version of MIDAS has a new page to check the WR timestamp of each FEE.

In AIDA Hardware control click: GSI White Rabbit Control
In expert options click: Collect all Timestamps
Verify every FEE has a good timestamp.

New merger has been worked on by VP which fixes the timewarp issues and MBS integration.

Currently no WAVE capture is supported, VP will re-add WAVE histogramming for startup testing.

10.05.2019 - Pulser walkthrough
-> Pulser rate 50 Hz
-> 1 minute per 0.1 V (0.1 to 1 V)
-> extra time on 1 V

With this 95% of channels calibrate easily (good statistics)
Issue: ~16 channels (most of ASIC3) in FEE10 are horrificly noisy. Not calibrated

Fig 1. Pulser of FEE1 (no DSSD)
Fig 2. Pulser of FEE9 (DSSD)
Fig 3&4. Pulser of FEE10 noisy channels

Future TODO: Will investigate WAVE data and look if issue can be found.

20.05.2019 - Alpha walkthrough
-> Pulser off
-> FEE10ASIC3 has readout off & discriminator masked
-> Slow threshold 0x64
-> Fast threshold 0xff
-> HEC threshold 0x2

File is nyx/gamma/nhubbard/AIDA/alpha_bg_200519_

Rate is approx. 1MB/s still of WR timestamp data.

(NB, MIDAS Alpha walkthrough before beamtime I see no alpha events in!)

Fig 5: Temps
Fig 6: Stats
Fig 7: Merger Stats
Fig 8: MBS Stats

Update on issue #48 - the "confusing state" is that the FEE has restarted and hence is undefined again.

An attached TTY log from the pi shows that the module is kernel panicking.
I have seen a couple of FEEs panic with the same error now.

(NB. The Day/time of the logs is wrong as the pi does not have the correct time - pis dont have a RTC or Internet access so the time isn't corrected)

Temperature of the modules is fine.

Aida is currently powered off (and I am away from GSI)

X projection of high energy channel ADC hits from R13 (attachment 1)

XY hit pattern from R13 (attachment 2)
    -discontinuity observed
    -possible timing issues or issues with Event Clustering

X projection of high energy channel ADC hits from R16 (attachment 3)

XY hit pattern from R16 (attachment 4)
    -Beam was being diffused at this point, and tpcs were being calibrated.
    -Gap between Y strips ~ 23 -> 30

High Energy Ey vs Ex plots from R13 and R16 (attachments 5 and 6 respectively)

it seems a FEE somtimes enters a confusing state and stops sending data
the current merger requires all FEEs to be active and so this stops the entire system from proceeding.

On MIDAS the page reports the module is "undefined"

On the TTY console (PUTTY) it returns: do_GetState returned z=0 and 8

Resetting the DAQ in question via MIDAS works (Putty logs of the stages shown) and then the merger resumes without trouble.

All AIDA modules have been updated to a new firmware 0xd330701 which should fix the merger issues.
A new GSI WR timestamp page has been added to MIDAS - the SYNC rollover is suggested to be 0xe for now.
(VP is working on lowering the rate of WR SYNC infowords)

Tested yesterday and Merger no longer reported major timestamp issues
n.b. aida11 and aida12 didn't update properly until today, this may have caused a few.

MBS relay is currently not functioning (data is confusing) but this is under investigation by VP ASAP.