09:55 Alpha run stopped

10:45 Investigating the large noise in FEE28
      First test was swapping the PSU positions of 29 and 30 - attachment 1
      PSU order 25, 26, 27 ,28, 30, 29, 31, 32

      Little affect on the statistics. 30 remains low and 29 is still high

11:15 System back in normal configuration with 29 and 30 going to original psu units
      PSU order 25, 26, 27 ,28, 29, 30, 31, 32
      System does seem worse off in this order - attachment 2

11:29 Bias cables core and braid switched between detectors 1 and 2
      No noticeable difference - attachment 3

11:44 Switched around cables running to FEE so DSSD1 was running to 29 and DSSD2 to 30
      Noise moved with cable. Though DSSD1 seems worse in 29 than it did in 30. - attachment 4

12:00 Cables moved back in normal operation. Stats returned to as they were at 11:15 - attachment 5

13:28 Changed kapton for a new one. No change to stats - attachment 6

13:55 Wiggled rain wire (Technical term) Tripped DSSD 2 bias. Ramped up bias. Noise gone. Mystery not solved. -
attachment 7


14.09 DAQ start
      AIDA file June18/R7

      ASIC settings DB key 2018Jun22-11.28.18
      *all* slow comparators 0xa

      Pulser walkthrough

      BNC PB-4
      Amplitude 90,000-10,000 @ 10,000 step
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency range 250Hz
      delay MIN, tail pulse, INT ref

14.18 DAQ stop
      See attachment 8 & 9

14.19 DAQ start
      AIDA file June18/R8

      Pulser walkthrough

      BNC PB-4
      Amplitude 12,000-2,000 @ 2,000 step
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency range 250Hz
      delay MIN, tail pulse, INT ref

14.28 DAQ stop
      See attachment 10 & 11

      DAQ statistics OK - see attachment 12
      7x FEE64s <10k
      13x FEE64s 10-20k
      3x FEE64s 20-30k
      3x FEE64s 30-40k
      2x FEE64s 40-50k
      4x FEE64 >50k (nnaida28, 29, 31 & 32)
      *all* system wide checks OK *except* nnaida7 & 21 fail ADC calibration
      FEE64 temperatures OK - see attachment 13
      DSSSD biases and leakage currents OK - see attachments 14 & 15

14.43 DAQ start
      AIDA file June18/R9

      ASIC settings DB key 2018Jun22-11.28.18
      *all* slow comparators 0x64

      alpha background

      BNC PB-4 OFF

18.30 Julabo FL11006 recirculating chiller - water refill - 75%

      per FEE64 1.8.W spectra - range 2ms - see attachments 16 & 17
      per FEE64 1.8.W spectra - range 200us - see attachments 18 & 19
      per FEE64 1.8.W spectra - range 20us - see attachments 20 & 21

04:12 TD found the nnaida14 appeared to have become noisy at some point. Biases/;eakages/temps ok
      ASIC check load appears to have fixed problem
      Noise appears to affect R9_0 to R9_51

13:45 Biases and leakages all ok - attachment 1
      FEE temperatures ok
      System wide checks ok
      T = 25.1C H = 35.7% DP 8.4C
      Currently up to file R9_54

21:53 Biases and leakages all ok - attachment 2
      FEE Temps ok
      System wide checks ok
      Currently up to file R9_56

10:20 Bias and leakage currents ok - attachment 1
      FEE temperatures ok
      System wide checks all ok
      Currently on file R9_58

15:25 Bias and leakage currents ok - attachment 2
      FEE temperatures ok - attachment 3
      System wide checks ok
      Currently on file R9_59

10:18 Bias and leakage currents ok - attachment 1
      FEE Temperatures ok
      System wide checks all ok
      Currently on file R9_63
      Humidity 43.3%, Temperature 25.1C, Dew Point +11.8C
      Filler level ok

12:51 Bias and leakage currents ok - attachment 2
      FEE Temperatures ok
      System wide checks all ok
      Currently on file R9_64

19:20 Bias and leakage currents - attachment 3
      FEE temperatures ok
      System wide checks all ok
      Currently on file R9_65
      ASIC 2 is still missing from FEE29. ASIC checks earlier in the afternoon have failed to recover it. -
attachment 4
      This ASIC has been observed to be working since it was installed last week. Once the alpha run is over
further attempts
      will be made to test it is back.

R7_0 -> R7_1 High Energy pulser run
R8_0 -> R8_1 Low energy puler run
R9_52 -> R9_62 Alpha run

On analysing all three runs it appears an ASIC is missing in FEE 29 - attachments 1 - 3

This ASIC does not appear to be missing in the alpha run carried out on June 24th - attachment 4

I do not believe that this is a broken ASIC.

I have performed an ASIC check load on the FEE module to try and regain it. I will check the files created since
later to see if it has been regained.

Using the pulser runs two different sets of offsets were calculated. One using the low energy run and one using
the high energy.
To compare the two different sets of offsets calculated Ex vs Ey plots were created for the first detector.

Attachment 5 shows these plots on the left is the low energy offsets and on the right the high energy pulser offsets
It can be seen that the grouping is much better with the events being tighter to the y = x line on the run sorted
using the low energy pulser offsets.

10:27 Bias and leakage currents ok - attachment 1
      Fee temperatures ok - attachment 2
      Temperature 24.7C, Humdidity 44.5%, D.P +11.8C
      System wide checks all ok
      Currently on file R9_69

      Vandle guys are about to start building. They have been made aware to avoid the power cables for the
chiller and AIDA
      Also of the network cables and HDMI cables.

14:46 Bias and leakage currents ok - attachment 3
      System wide checks all ok
      FEE temperatures all ok
      Currently on R9_70

19:55 Bias and leakage currents ok - attachment 4
      Fee temperatures ok
      System wide checks ok
      

11:05 Bias and leakage currents ok - attachment 1
      FEE temperatures ok
      System wide checks ok
      Currently on file R9_74
      Temperature 25.3C, Humidity 42.9%, D.P +12.1C

13:27 Bias and leakage currents ok - attachment 2
      FEE Temperatures ok
      System wide checks ok

      Analysis of the pulser run R9_52 -> R9_74 has shown no major losses in terms of apparent broken bond wires.
      The large gap in FEE64 29 can be explained by an ASIC dropping out: See https://elog.ph.ed.ac.uk/AIDA/730
      Another odd trend appears in the channels relating to DSSD 8 on the x side. - attachment 5
      The pattern shows one strip on one strip off for the first 32 strips.
      nnaida18 is also the FEE which is running at a much higher rate during this run. - attachment 3
      I'm not sure that I believe that these channels are actually relate to broken bond wires.

      Here are the hit patterns for the 8 DSSD - attachment 4
      Attached is also the x and y histograms for DSSD 8 which shows a weird pattern - attachment 5

10:25 Bias and leakage currents ok - attachment 1
      FEE Temperatures ok
      System wide checks ok
      Alpha run R9 stopped on file R9_74

Attached is an updated inventory of the cables and adapter boards we have in RIKEN.

10:35 Bias and leakage currents ok - attachment 1
      FEE Temps ok - attachment 2
      System wide checks all ok
      

10:23 Bias and leakage currents ok - attachment 1
      Fee temperatures ok - attachment 2
      System wide checks ok

14:07 Bias and leakage currents ok - attachment 3
      FEE temperatures ok
      System wide checks ok

      Have been unable to recover ASIC 2 in nnaida29. Have checked removing the detector cabling and can still
not recover the ASIC
      Also noise in DSSD1 and 2 is still high - attachment 4
      However in DSSD3-8 it is very good

      Humidity 44.7%, Temperature 25.5C, D.P +13C

15:57 FEE modules turned off via rly16
      DSSD bias removed

      AIDA has been put in a safe state.

Attachment 1 - DHCP configuration file /etc/dhcpd.conf

Attachment 2 - DHCP startup messages

Attachment 3 - FEE64 startup messages from /var/log/messages

Note

1) dhcpd service is not listed by services tool which we have usually used to restart dhcp following
changes to the configuration. Currently we restart dhcpd as follows

kill -SIGTERM <pid dhcpd>
/usr/local/sbin/dhcpd

2) significant delays (~6 and ~9 minutes) starting aida07 and aida05, otherwise appears to be normal startup

> 
> Attachment 1 - DHCP configuration file /etc/dhcpd.conf
> 
> Attachment 2 - DHCP startup messages
> 
> Attachment 3 - FEE64 startup messages from /var/log/messages
> 
> Note
> 
> 1) dhcpd service is not listed by services tool which we have usually used to restart dhcp following
> changes to the configuration. Currently we restart dhcpd as follows
> 
> kill -SIGTERM <pid dhcpd>
> /usr/local/sbin/dhcpd
> 
> 2) significant delays (~6 and ~9 minutes) starting aida07 and aida05, otherwise appears to be normal startup

Regarding startup of 5 and 7. what is the report from the console log monitor?

Currently seems to be a problem with the memory check in the system wide checks that produces the message shown
- attachment 1

DAQ is being run with the following options - attachment 2 and 3

The PSU Temperature readout in aida07 appears to not be working. Shows a permanent value of 160 degC - attachment 4
Water inlet and outlet feel cold so believe it is an artiface
aida07 also appears to quite often take longer to mount than the others. Often coming in 5 minutes later than them.

Only tested aida01 - aida04 for noise as we lacked the required HDMI type c cables to set up a full MACB branch
structure.

We appear to have considerable noise issues in the FEE modules with adapter boards on but no DSSD cables
connected but test input loop and pulsar connected.
The pulsar appears to be the circuit of doom for this noise. With all test input ports connected to a pulsar
chain each FEE is seeing in the region of 800k rate of good events with discriminator thresholds set to 255.
Removing the pulsar cable from the end FEE but leaving the chain in place reduces this to around 500k.

Grounding an adapter board to the copper cooling plate also further reduces this noise somewhat but still leaves
rates of around 300k.

If an adapter board is isolated from the chain so that it has nothing connected to it the rate drops to around
300-400 which is just the rate of the sync pulses.

For the following screenshots aida01 and aida02 are isolated adapter boards.
aida03 and aida04 are connected via lemo cables via the test input to other FEE modules in their chains.
Waveforms for the 4 FEEs - attachment 5-8
Full statistics for the 4 FEEs - attachment 9-12
Good events comparison - attachment 13

It appears we need to provide a low resistance path between the FEE cooling plates like we do in RIKEN.
Currently the measured resistance between FEE 6 and 10 is around 2 ohm.

Also we may need to isolate the pulsar signals themselves

A potential difference of 5mV was also measured between the chassis of the NIM modules and the cooling plates.
Resistance was 0ohm when the power was on.

> Currently seems to be a problem with the memory check in the system wide checks that produces the message shown
> - attachment 1
> 
> DAQ is being run with the following options - attachment 2 and 3
> 
> The PSU Temperature readout in aida07 appears to not be working. Shows a permanent value of 160 degC - attachment 4
> Water inlet and outlet feel cold so believe it is an artiface
> aida07 also appears to quite often take longer to mount than the others. Often coming in 5 minutes later than them.
> 
> Only tested aida01 - aida04 for noise as we lacked the required HDMI type c cables to set up a full MACB branch
> structure.
> 
> We appear to have considerable noise issues in the FEE modules with adapter boards on but no DSSD cables
> connected but test input loop and pulsar connected.
> The pulsar appears to be the circuit of doom for this noise. With all test input ports connected to a pulsar
> chain each FEE is seeing in the region of 800k rate of good events with discriminator thresholds set to 255.
> Removing the pulsar cable from the end FEE but leaving the chain in place reduces this to around 500k.
> 
> Grounding an adapter board to the copper cooling plate also further reduces this noise somewhat but still leaves
> rates of around 300k.
> 
> If an adapter board is isolated from the chain so that it has nothing connected to it the rate drops to around
> 300-400 which is just the rate of the sync pulses.
> 
> For the following screenshots aida01 and aida02 are isolated adapter boards.
> aida03 and aida04 are connected via lemo cables via the test input to other FEE modules in their chains.
> Waveforms for the 4 FEEs - attachment 5-8
> Full statistics for the 4 FEEs - attachment 9-12
> Good events comparison - attachment 13
> 
> It appears we need to provide a low resistance path between the FEE cooling plates like we do in RIKEN.
> Currently the measured resistance between FEE 6 and 10 is around 2 ohm.
> 
> Also we may need to isolate the pulsar signals themselves
> 
> A potential difference of 5mV was also measured between the chassis of the NIM modules and the cooling plates.
> Resistance was 0ohm when the power was on.

Re:- Memory check problem..... the required log file is not at /MIDAS/XilinxLinux/...... The file is a log file placed
in the FEE64 root file system. The code will be altered in time to reflect the new location by using a local variable.

Re:- aida07. This module should be urgently removed and sent back to Daresbury for repair. This measurement could be
indicative of the worse problems with this module.

Re:- Earthing. Please could you add some photos of the installation. ( apologies if they are already here ;-) If I
recall it is worth adding a low resistance connection ( big copper braid ) from the NIM bin to the adaptor board earths.

10:08 Normal system restart ok
      System currently in 8DSSD configuration as was left in June 2018

      Bias and leakage currents ok - attachment 1
      Temp +24.0 deg C, H 46.8%, D.P +11.7 deg C
      FEE Temperatures ok - attachment 2
      System wide checks all ok (aida7 fails ADC)
      
      DAQ Started with pulser test input
      Statistics ok - attachment 3
      Hit rate laout for first 30 FEEs (Note second ASIC in FEE29 still missing) - attachment 4
      Waveforms for Layout 7 (Front side) ok - attachment 5
      Waveforms for Layout 8 (Back side) good (nnaida 7 missing due to failed ADC check) - attachment 6

14:25 Moved from 8x DSSD setup to 6x DSSD
      Top two detectors removed.
      Detector alignments in snout unchanged so can use the 6xDSSD mark from June 2018

      Rev F adapter cards have been put in place for all backside FEEs
      All adapter boards are now Rev E or F
      All adapter boards have an additional ground cable from the screw on the copper block to the lemo ground
connection

      Power on and detectors biased (Cabling changed to reflect new 1-6)
      Leakage and bias ok - attachment 1
      System wide checks all ok
      FEE temperatures ok - attachment 2

      DAQ Started
      Stats are looking very good - attachment 3
      Rates look ok (Lin and Log) - attachments 4 & 5

      Front side waveforms look ok - attachment 6
      Backside waveforms look ok - attachment 7

      Pulser widths are showing double peaking. - attachments 8 & 9
      we are not sure if this is down to the pulser or not

      Expanded timescales of the waveform, 0, 10, 100 and 1000 cycles
      Layout 7 - attachments 10 - 14
      Layout 8 - attachments 15 - 18

15:24 Pulser turned off in prep for alpha run
      Slow comp set to 0x64

15.42 DAQ start file Sep18/R2

      Test and calibration using background alpha particles

      ASIC settings 2018Jun22-11.28.18
      *except* slow comparator 0xa -> 0x64

      BNC PB-4 pulser OFF

      Merger c. 30k items/s TapeServer 230kb/s

09.10 DAQ continues OK - current file Sep18/R2_6

09.15 all system wide checks OK *except* nnaida7 fails ADC calibration

      detector biased & leakage currents OK - see attachments 1 & 2


      FEE64 temperatures OK - see attachment 3
      good event statistics OK - see attachment 4
      nnaida2 rate due to one hot channel

      Merger OK - see attachment 5
      TapeServer OK - see attachment 6
      no errors or warnings shown by server terminal sessions 

      1.8.W - 20us FSR - see attachments 7 & 8

      all spectra zeroed
      
16.05 temperature +24.2 deg C, RH 45.4%, d.p. +11.6deg C
      Julabo FL11006 water level c. 75-80% (i.e. approx mid-way between middle and upper level indicators)

https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit?pli=1#gid=282765719

09.10 DAQ continues OK - current file Sep18/R2_16

09.15 all system wide checks OK *except* nnaida7 fails ADC calibration

      detector biases & leakage currents OK - see attachments 1 & 2

      FEE64 temperatures OK - see attachment 3
      good event statistics OK - see attachment 4
      nnaida2 rate due to one hot channel

      Merger OK - see attachment 5
      TapeServer OK - see attachment 6
      no errors or warnings shown by server terminal sessions 

      1.8.L spectra - see attachments 7 & 8
      1.8.W spectra 20us FSR - see attachments 9 & 10
      
09.24 temperature +25.5 deg C, RH 43.4%, d.p. +11.9 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level c. 75%

16.25 Julabo FL11006 refill to just below H indicator

      DAQ continues OK - current file Sep18/R2_19