1) Login to Raspberry Pi 

ssh pi@nnrpi2

2) Kill any instances of the TclHttpd server

ps -ef | grep TclHttpd
kill -9 <pid>

3) Start server

/MIDAS/TclHttpd/linux-arm/TclHttpd-server &

10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
      Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%

10.15 DSSSD stack as follows

      DSSSD  serial  thickness  depletion 
      #              (um)       voltage (V)
      1      3208-7  1019       125              upstream
      2      3208-13 1020       120
      3      3208-14 1020       130              downstream

11.30 AIDA power relays sequenced on
      
      DAQ start

      system wide checks ok

11.33 attachment 2 - fee temperatures - ok

      attachment 3 - good event statistics with slow comparator threshold = 100

      attachment 4 - good event statistics with slow comparator threshold = 50

      attachment 5 - good event statistics with slow comparator threshold = 20

      attachment 6 - good event statistics with slow comparator threshold = 10
     
13.51 attachment 7 - Rates
                   - aida09 1 ASIC not working (no data)

      attachment 8 - good event statistics

      attachments 9/10 - 1.8.L spectra
                       - no peak observed in aida09

      attachments 11/12 - 1.8.W spectra
                        - waveform only observed in naida 2,4,7,8,12

      pulser peak widths -   FEE         width(ch)
                              1             71
                              2             96
                              3             66
                              4             79
                              5             74
                              6             96
                              7             89
                              8             83
                              9           no peak
                              10            82
                              11            153
                              12            120

17.26 FATIMA moved upstream around AIDA

      attachment 13 - good event statistics

      ASIC check-load performed

      attachment 14 - good event statistics following checkload

12.51 - detector bias and leakage currents - detector 3 has no current
                                           - other detectors ok

13.13 - tried changing from HV3 cables to HV4 - still open circuit 

      - tried different permutations of moving HV-4 braid and core cables (high voltage to ground connections)

       (dssd3)

      (all directions defined as if looking upstream)

      - core -> bottom, braid -> left     open - circuit

      - core -> top, braid -> right       trips at 8V

      - core -> bottom, braid -> right     open - circuit

      - core -> top, braid-> left          trips at 8V

      - removed jumpers - still trips

13.36 - testing continuity of bond wires on dsssd 3208-14 & 3208-15

      - observe 20 ohm resistance on p+p and p+n bond wires 

19:41 - dsssd 1 and 2 not biasing, 3 ok.

      - changed bias configuration -> HV core 1 & 2 now on top, braid on right

      - ribbon cables for DSSSD #1 & #2 p+n junction strips down were misaligned

      - all detectors biased succesfully

HV cable configuration (looking upstream): HV-1 Core -> top-inner (FEE 1)
                        
                                           HV-2 Core -> top-middle (FEE 5)

                                           HV-3 Core -> bottom-outer (FEE 11)

                                           HV-1 Braid -> right-inner (FEE 2)

                                           HV-2 Braid -> right-middle (FEE 6)

                                           HV-3 Braid -> left-outer (FEE 12)

Attachment 3 - rough sketch of HV cable configuration, looking upstream.
       

Ribbon cables fitted with kapton PCB couplers -> 3 isolated + 1 non-isolated for each cable length/DSSSD

Ribbon cable lengths:

without kapton PCB coupler (cm) - 40, 47, 51.5

with kapton PCB coupler (cm) - 50.5, 57.5, 62

gap between plastic and dsssd - 15 mm

length between top of snout and dsssd - 5cm

AIDA snout length 37cm (from end of AIDA snout support assembly)

downstream scintillator (assembly depth 1cm)   36cm

downstream dsssd (#1)                          35.2cm

                 (#2)                          34.3cm

                 (#3)                          33.4cm

upstream dsssd   (#4)                          32.5cm

upstream scintillator (assembly depth 1cm)     31.7cm

Cable lengths required are therefore

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 22.5 = 54.35cm 21.4"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 16.5 = 47.45cm 18.7"

                 (#3)           33.4 - 10.5 + 2.15 + 5 + 10.5 = 40.55cm 16.0"

upstream dsssd   (#4)           32.5 - 10.5 + 2.15 + 5 + 4.5 = 33.65cm 13.2"

length of RH Kapton PCB coupler = 10.5cm (Carlo can you check please?)

turning the corner at the base of the AIDA snout support assembly 2.15+5=7.15cm
(see Peter's drawing)

distances to FEE64s 4.5, 10.5, 16.5 and 22.5cm
(see Peter's drawing)

f we use 3x DSSSDs then the cable lengths required will be as follows:

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 16.5 = 48.35cm 19.0"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 10.5 = 41.45cm 16.3"

upstream dsssd   (#3)           33.4 - 10.5 + 2.15 + 5 + 4.5 = 34.55cm 13.6"

https://docs.google.com/spreadsheets/d/1fZBybHodSvacUdKpDM_VKznqTrh49CKnqKVQVMEDWGs/edit?usp=sharing

A spreadsheet of ambient conditions and water temperature in S4 for AIDA.
Will show the distance to danger points (interlock shutdown and condensation on the pipes)

Included waves for all 12 FEEs

FEES 1, 5, 9 all show this HF noise on it, and are all located at the top of the AIDA mount.
Connections: HV from DSSD (-ve) is in FEE9
Pulser input is in FEE9 (non-inverted)

Resistance between FEE cooling plate and LEMO connector for all 3 FEEs is a few Ohms, as is resistance between copper braid on DSSD cable and the LEMO connector.

All FEEs show some noise but mainly those 3 and all four with DSSD. Will try to check DSSD connections in case something is messed up.

FEE layout for reference: (Beam's perspective)

                              FEE9
                              FEE5
                              FEE1

FEE10 FEE6 FEE2                FEE4 FEE8 FEE12

                              FEE3
                              FEE7
                              FEE11

Noise seemed less strong today, unsure of cause. I had re-taped snout as tape was not sticking well.

Figures 1-4: Waves for all 4 FEEs connected to the DSSD
Figures 5-8: Zoomed in on region 400-600
Figures 9-10: Rates in DSSD

Note in particular fee09 has a clear noise pickup somewhere. Also appears when no DSSD voltage.
Peak distance = 6 channels = 16.6 MHz.

Fee09 is connected to HV bias core (-160 V)
However unplugging the HV cable from it shows no difference.

Must be related to something else.

Update 16:29 CEST:
Distance is more like 5 channels (20 MHz)
Fee09 picks it by far most strongly but all fees you can make it out
Additionally can see a slower waveform especially on other fees... approximately 40 channels = 2.5 MHz

Checked with multimeter that low resistance path between FEE cooling plate and ground LEMO connector
also low resistance between ribbon cable copper shield and ground LEMO connector.

Noise seems to appear even if pulser cable unplugged at pulser end or when attentuation is increased weaking pulse.

Figures of intense high frequency noise pickup in AIDA.

Will investigate source in S4.

Edit: Period is approx 40 cycles / approx 2.5 MHz (?)

AIDA Finally working again so can test VP's added WAVE histogramming mode.

Fig 1. Waveforms for FEE12 ASIC 1. Sample rate = 10, Threshold = 10500
n.b. lots of "blank" wave channels for unknown reason, all low-energy channels show pulser so it's not a DSSD signal problem?

Fig 2. Pulser spectrum for FEE12 ASIC 1. (10 peaks 0.1 -> 1 V)

Fig 3. Waveforms for FEE10 ASIC 3.

n.b. WAVE channels frequently stopped updating, needed start/stop DAQ or restart WAVE readout option. Unsure if it's related to high rate/odd signals.
Might be related to issue in FEE12 too. WAVE good events statistics lower than empty FEEs.
All rates quite high, looks quite noisy at the moment.

Fig 4. Pulser of FEE10 ASIC 3.

Note that FEE10.3 has poor peaks, in WAVE we see:
Some normal looking pulses which are merely noiser than other channels.
Some pulses which look very odd (baseline really low, etc).

Will recheck ground for noise issues, but the huge peaks are confusing to me. (3.5.W, etc)

The other 6 FEEs were powered on one at a time to check no issues. None found.
All 12 FEEs plugged back into the PSU.

Note: The FEE temperatures reported on Friday were before a Reset/Setup of the DAQ, which means the ASICs run very fast and the hence run hotter.
Reset/Setup shows more sustainable temps (attached)

These temps are similar to those observed in Sep 2018 but warmer by ~4 C than those observed Dec-Jan and April.

The water issue has been resolved - merely incorrect temperature guages (replaced). Water is 20 C as required.

S4 conditions at 13:12 CEST

26.5 C / 47.2 % / Td = 14.3 C

Based on email by TD doing the following procedure to  ensure FEEs are OK after water issue

1) disconnect the power cables of *all* FEE64s at the FEE64 PSUs
2) re-connect power cable, power-up and check/test the FEE64s
  *one at a time* until stable operatinmg temperature is achieved
... say 30mins?

13:34 CEST: All be FEE1 unplugged. Powering on!

Vertex temp read slightly warm (67 C) so powered off. Waiting for a while for water to circulate since off for a while...

15:01 CEST: FEE1 Powered on again, Vertex reading 67 C again but it's stable and slowly going down. Watching and seeing but I think it's OK.
15:35 CEST: FEE1 has been stable around 67 C all the time. Other temps fine. Presuming this isn't a major concern. (Perhaps related to reasonably warm/humid conditions?)

15:47 CEST: FEE2 Powered on.
16:02 CEST: Temperature stable, not exceeded 60 C.

16:07 CEST: FEE3 Powered On.
16:17 CEST: Temperature stable, peaked at 58.44 C

16:21 CEST: FEE4 Powered On.
16:40 CEST: Both FPGA and ASIC reading warm: 65.5 C and 58 C. Seeming to not be rising anymore.

16:41 CEST: FEE5 Powered On.
16:53 CEST: Temperature stable, at 64 C

16:56 CEST: FEE6 Powered On.
17:12 CEST: FPGA temp is ok but ASIC warm. Tmax ASIC = 56.88, FPGA = 55.38, PSU = 27.44

17:15 Turning FEEs and Water off for the weekend
Will test 7-12 on Monday.
Might be cooler in S4.

Cooling seems to be working but temps hotter than before.

Temp: 26 C
Humidity: 50%
Dew Point: ~15 C

There are two temperature dials outside S4, one shows exactly 20 C and one shows nearer 16 C.

Trying to clarify this discrepancy as 16 C is too cold at the moment.

In contact with developer of cooling system to confirm apparent 16 C after the heat exchanger

Today's conditions:

Temp: 26.9 C
Humidity: 48.7%
Dewpoint 15.2 C

S4 Conditions:

25.2 C, 35.3% RH, Td = 8.8 C

Water temperature: 20 C

-

Rotating the dewpoint sensor on the pipe seems to have made it switch over to solid red = humidity OK.
Waiting confirmation to see if we should check a FEE (as per PJCS suggestion) before water is turned back on.
Will check the outside copper after lunch.

13:04 : Copper seems warm and dry, humidity sensor still solid red.

It occurs to to wonder if there is now water inside the modules. If the metal of the cooling plates is damp on the outside then it is safe to assume there is water on the inside too . Can I suggest that one  of the modules is dismounted, disassembled and checked?

Patrick

Water is still off.

Checked dewpoint sensor still flashing red light (indicating an error). Pipes warmed. Not what I expected. Possible fault?

I checked S4's ambient temp/humidity with a home-brought sensor: Reports 27 C/50% RH (Dew Point = 16 C)
Water temperature is set to exactly 20 C (there's a second dial showing ~18 C but I am unsure if that's the same water supply, I will ask)

Checking the dewpoint PSU with a multimeter shows a voltage of 16 V.
The PSU itself is rated for 12 V output, and the humidity sensor expects 24 V.
Regardless of which voltage 16 V seems incorrect. Might be causing issues.

Humidity is a bit lower now (4pm) the sensor has been in all day, 28 C/35% RH (Dew Point = 11 C) which isn't an issue.

Water is still off until I am sure what to do, or until the dewpoint sensor stops complaining.

The humidity sensor hasn't reported humdities under 90% yet. In order to try and help dry the pipes and turn AIDA the water has been turned off the weekend.
Sunday temperatures might reach 40 C which means it may struggle anyway.

Will consider trying to get a temperature/humidity monitor for S4 to check if the environment is fine too.

Currently AIDA's interlock is off (no lights) and hence AIDA cannot be powered on. Under investigation - power supply has been replugged but seems ok.

Update: It seems the interlock is correctly stopping AIDA from being powered on due to high humidity on the cooling lines. Am currently trying to verify the temperature with GSI technical staff

The latest version of MIDAS has a new page to check the WR timestamp of each FEE.

In AIDA Hardware control click: GSI White Rabbit Control
In expert options click: Collect all Timestamps
Verify every FEE has a good timestamp.