Investigate noise situation of AIDA (no DSSSDs connected)

Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
- aida16 very noisy ASICS 1&2, normal ASICS 3&4
- back-side aidas (02,04,06,08) worse resolution than front-side
- high-frequency noise still noticeable in waveforms

Figs 7-8: Remove pulser in (from NIM rack) only
- no noticable change

Figs 9-10: Remove all pulser cables from adapter boards
- looks improved

Figs 11-12: HV Braid->GND Jumpers removed

Fig 13-14: All HV cables removed
- aida06, aida14 isn't updating
- very noticable improvement

Fig 15-16: Intermediate HV cables re-added
- signal worse in FEE64s with HV cables
- aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good

Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.

Pulser widths now:
- aida14 (-ve amplitude) = 13.5 channels
- aida08 (+ve amplitude) = 40 channels (?)

DSSD stack will be as follows

1      3208-7  1019       125              upstream
2      3208-13 1020       120              downstream

First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance

3208-7: 
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

3208-13:
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

Conclusion: All bias bond wires are intact on both DSSDs

Visual inspection is both DSSDs look good with no damage or bond wire issues

Installed in snout: full stack

3 mm bPlas 1
17 mm Air
1 mm AIDA 1  (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2

It occurs to to wonder if there is now water inside the modules. If the metal of the cooling plates is damp on the outside then it is safe to assume there is water on the inside too . Can I suggest that one  of the modules is dismounted, disassembled and checked?

Patrick

> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
> 
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
> 
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>  
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> 
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5 
> 
> Attach 5: Timestamp shown by merger

Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of 
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a 
logical "rollover" of the lower 14 bits of this 10nS timestamp.


Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?


It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.

The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be 
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.

If this occurs again then the system wide check results would be interesting.

Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida*

Rates in all FEEs much higher than before - waveform shows very noisy in all systems. - attachment 1

Good event statistics - attachment 2

DISC in 7 and 3 above 300k

ASIC check load performed.

Good event statistics - attachment 3

Thesholds at 0xa for all FEE

Disc rate now 0 across all FEE


Pulser settings: attachment 4

pulser peak widths -   FEE         width(ch)
                              1             128
                              2             145
                              3             84
                              4             74
                              5             207
                              6             125
                              7             172
                              8             80
                              9             193
                              10            183
                              11            238
                              12            138

Attachments 5 & 6: pulser peaks for all FEE64

16.24: DAQ stop

       slow comparator threshold changed to 0x64

       ASIC check performed

       Alpha run/DAQ start (w/ data transfer enabled) - writing to file 31Oct19/R13

       Merger and TapeService ok - attachments 7 & 8

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 

Taken 4 FEE64s from CRYRING (the 4 easiest to access)
  41:d8:2b
  41:f6:5a
  41:ee:71
  41:d0:0e

In addition a FEE without rails marked only 20 was found in my old office for 5
- This probably came in a shipment to/for repairs

DSSSD#1 unbiased

DESPEC Platform moved out of beam and gamma platform moved from AIDA
JB works on grounding the original 8 DSSSD#1 FEEs

Connected grounding from ribbon cable drain to adapter PCB
Ground loop connected

AIDA rates unchanged bad still fig 1
Waves fig 2 (p+n 6000-9000)
Waves fig 3 (n+n 8000-10000)
Statistics fig 4

AIDA snout was reconnected and we saw originally a very good reduction in noise from S452 (fig 1)
Occasional bursts of increased rates were noticed, perhaps related to works in S4.

Tape rate at optimal conditions was around 6 MB/s
We still saw some extrinsic noise in waves, two frequencies observed : 1.5 MHz and 100 kHz or so

Figs 2, 3 are at 50 MS/s
Figs 4, 5 are at 5 MS/s

Today the situation has become a lot worse for reasons still unknown.
aida01 shows very strong noise, frequency 125 kHz or so it seems (still at 5 MS/s)
Amplitude 100 mV or so I think

Investigation still underway

Remount snout in S4 (photos to come) 
Alignment with DEGAS OK but a bit short...  DEGAS platform will be brought 31 mm closer to AIDA by rail adjustment

Can mount bPlas booster boards on old ITEM frames and also partly on AIDA snout (isolated)

-

Reconnect all adapter boards and grounds, bias AIDA...

HV OK. We still have a light leak at the snout endcap. "Fixed" with sheet for now

HV leakage currents figure 1

Start AIDA DAQ and check rates.. all seem OK. aida07 is very noisy other p+n good
Notice some issues in aida04, check and ribbon cable slightly loose on one side of adapter
No change to other FEEs.
Aida07 was inspected but nothing obviously wrong

Rates figure 2
Rate spectra figure 3
Wavesform figures 4 (p+n), 5 (n+n)
Pulser peak withs
 p+n figure 6, width ~  70 channels (50 keV)
 n+n figure 7, width ~ 570 channels (400 keV)

n.b. double peak in aida14 is an artifact from an uncleared histogram

DSSDs debiased to allow work on bPlas cabling without tripping DSSDs

BB7 connected to aida07, sitting in a cardboard box

Update Tcl/Tml files in ASIC, ASIC4 and sys.tml frm PJCS email to allow setting per-ASIC polarity on FEE64
Restart MIDAS@8015

Power on all FEEs

Note nnrpi is not responding to ssh... MP investigates if it ahs been unplugged
 No, but a powercycle worked and fixed it

See error: /MIDAS/FEE_ASIC/aida06/ASICs.txt  not found, create the folders

We see some data in the L spectra, and changing the settings from one ASIC seems to work properly

Stopped for CC/MP to determine the which ASICs correspond to p+n and n+n

Follow up: Alpha data overnight or source?

AIDA and the Snout have been remounted in S4

I had a check of the AIDA connectors and all looked good, the kapton PCB connectors were sitting comfortably above the interstage and I saw no stressed/loose connectors.
Multimeter showed no continuity between snout or inter-stage and AIDA ground

We put a single layer of wide black tape inside the end-cap to improve the light seal and minimise reflections

After mounting the snout I managed to bias both DSSDs to 100 V with no issues. I had to raise the DSSD#1 (Ch0) limit to 40 uA because it rises quite high during ramp up (then settles).

It is very hot in S4 (28 C) and the roof is closed for high-radiation experiments, so I expect it to get hotter. DSSD leakage currents were similar to before (20 and 15 uA)

There is also the new Temp and Humidity monitor, which you can view on AnyDesk at 192.168.11.119 (I think), there is a firefox window open already 

The AIDA snout (with single DSSD and plastic) has been mounted in S4 cave again. DSSD was connected to the outer 4 FEEs again.

-160 V Bias applied and DSSD holds voltage with leakage current of around 1.3 uA (was dropping as only biased briefly)

Bias off and FEEs offline again now as day nearly done. MBS testing will be performed shortly, as well as resolution checks.

Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference

> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.

All AIDA modules have been updated to a new firmware 0xd330701 which should fix the merger issues.
A new GSI WR timestamp page has been added to MIDAS - the SYNC rollover is suggested to be 0xe for now.
(VP is working on lowering the rate of WR SYNC infowords)

Tested yesterday and Merger no longer reported major timestamp issues
n.b. aida11 and aida12 didn't update properly until today, this may have caused a few.

MBS relay is currently not functioning (data is confusing) but this is under investigation by VP ASAP.