Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

After tests in the Daresbury T9 system.

Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees.

To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF

The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected.

Alternatively STOP acquisition, set the ADC Control  register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.

The latest version of MIDAS has a new page to check the WR timestamp of each FEE.

In AIDA Hardware control click: GSI White Rabbit Control
In expert options click: Collect all Timestamps
Verify every FEE has a good timestamp.

To Synchronize the FEE64 ASIC clocks to rise at the same timestamp time use the System function Synchronise the ASIC clocks in the System Wide Checks browser page.

The Server will read the current timestamp value and calculate, based on the number of FEEs and the access delay, a timestamp value sometime in the future. F_stamp

F_stamp is written to each FEE and the synchronization is enabled. 

In a state machine in the FEE ASIC clock control, logic is enabled to synchronize its ASIC clock. It compares the current timestamp with F_stamp and if the current timestamp is >= F_stamp then it will start the ASIC clock.

The report in the browser log window gives the 3 LSBs of the timestamp at the instant the ASIC clock is started. They should all be the same. They are from the 10ns timestamp.

Here is a document showing how and where to calibrate the LMK3200 devices that lock to the system clock and generate the clocks for the ADCs and the FPGA internal logic.

If the merger doesn't show "xfer Links => Merger" at the top, it is probably the merger setting has been corrupted somehow. This is how to reset it

  If you go to the web page from which you started the Merge and TapeServer

    (probably    localhost:8115)

 Then select the item  NetVar Service  which is at the left hand end.

  Click on Inquire Registers

  then  enter    NetVar.MERGE.RunOptions  into the "Register Name" entry    and  when finished move the mouse out of the entry box.

 NetVar.MERGE.RunOptions will appear in the center of the top row.   (it is an item in the menu but not so easy to find)

 Click on Read and I expect you will get  0

 Type 1 into the data entry box     and click on Write

  Now return to the Merger and  click on  Reload

  The Merger State will now end as    xfer Links => Merger

 The button  Output:   will now toggle between this and   xfer links => Merger => Storage.

Figures of intense high frequency noise pickup in AIDA.

Will investigate source in S4.

Edit: Period is approx 40 cycles / approx 2.5 MHz (?)

Looking at the HV bias in a scope to see if it is the source of noise especially on the p+n side

Channel 1: HV Core channel 4
Channel 2: HV Braid Channel 4
Math: Ch1 - Ch2

100V bias (100 uA leakage current for 1 MOhm)

No obvious frequency components or large frequency components?
I do not know how this compares to RIKEN or STFC-DL

Updated to put the right frequency range (misread before)

Fig 3-4: Normalish looking data 

Fig 5-6: Examples of noise appearing randomly on math line

Fig 7: 1ms long trace, noise stopped appearing

Update 26.02.2020

Fig 8: Trace with all 4 HV channels off

Fig 9: Trace with only oscilloscope biased

Fig 10: Trace with braids unplugged from all 3 FEE64s (only channel 4 connected)

Things to try:

Joining clean earth & chassis ground?

Checking HV jumper?

Finding 'scope probes and testing NIM socket?

Tests performed

• After leaving overnight noticed the pulser was not fully inserted into aida09
  • Little change in rates observed in 9 but otherwise nothing else
• Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
  • No change observed in rates
• Also tested covering the snout in light tight material
  • No Changes
• Unplugged the system monitor from 6 and 8, no change
• Connected the snout to ground with copper tape
  • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
  • Removed the ground and powercycled and rates restored to where previously
• Changed the bias supply jumper inside to have the bias line connected to the rack ground
  • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
    • Recrimped and inserted again
  • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
  • Pulser widths however larger than yesterday

p+n width

n+n width

• Connected up entire pulser circuit to n+n FEEs

Switching bias supply back

• because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
• DAQ was then left overnight as had to leave to catch the bus
• Rates are observed to be back at their higher values - attachment 2
• n+n 1.8.L spectra - attachment 3
• p+n 1.8.L spectra - attachment 4

p+n widths

n+n widths

• The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

n+n

• Don't see a pulser peak in 4 or 2
• Will replace cable between 4 and 2 and 2 and 6
• Statistics at 0x64 with pulser on - atachment 5
• Now see peaks - attachment 6
• Statistics at 0xa - attachment 7

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

• Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
• 0x64 stats - attachment 7
• 0xa statistics - attachment 8
• p+n 1.8.L - attachment 9
• p+n waveform - attachment 10
• n+n waveform - attachment 11
  • Same high frequency component in both
  • 35 channel spacing corresponds to a frequency of ~1.4MHz

Next tests

• Returned braid to FEE 4 and 8
  • Recovered earlier rates and performance
• Re-added the ground cable to the snout
  • Much worse noise performance across all FEEs
  • p+n rates in 200k
• Removed the ground from the snout
  • Got really good rates originally - attachment 12
  • Noticed 1 and 5 failed ADC calibration
  • Calibrated them and there rates increased massively - attachment 13
• Notice that the pulser peak in FEE2 and 6 keeps dropping out
  • As the cables have been changed multiple times thought it could be that an adaptor board was broken
  • 4 seemed the most likely suspect with its high rates and triple peaks
  • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
  • Pulser peaks were still muissing
  • Issue could be then with adaptor board 2 possibly

Conclusions

• Rates on the FEEs are in a better place now than they were at the start of the week
  • p+n in particular have shown a large amount of improvement
• There are still however issues with the n+n FEEs
• I think there is likely an issue with one of the adaptor boards
  • Possible aida02 at the pulser connection point
  • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
  • Would recomend swapping out aida02 as a test next week
• Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
  • Nic would like to use something other than clingfilm in future
  • One suggestion would be mylar which could be held in place with strips of double sided tape
  • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
• Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
• I am at a slight loss of what else to try

• I have left the daq running at 0x64 overnight to check for alphas
• Histograms and statistics zeroed

For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 

- Ch 2 and 3 of CAEN HV module changed to negative polarity, Ch 0,1 still with positive polarity
- Bias changed from the n+n to the p+n side (now Ch 2,3) - att. 1

- EXPERIMENTS/AIDA/2025Oct28-14.55.15 -> Slow comparitor -> 0x64 (1MeV) p+n Pos. bias & n+n Neg. bias

- Attachment 7: aida01 had its threshold lowered to 500 keV -> Rate increased from 500 to 6500 Hz
- Pulser inputs plugged in.


We have input a 1V pulser input into the data and have measured a peak in aida01 with a FWHM = 75 channel see attachment 9 and a FWHM in aida 14 (DSSD2) of also 76 channels which is similar to the results obtained in June from Nabil.


We cannot see anything from the n+n side as the signals are too noisy. Probably best to go with a higher trigger rate on the pulser and check with log scale tomorrow. It might also be worth checking the grounding on the y-strips as the problem with biasing with positive polarity is still seen in aida02 and aida04 (high noise repetition0 signal, attachment 2).

Conclusion is that we have reproduced in part the result from Nabil.

Tomorrow we will continue with the tests by lowering the thresholds and hopefully collect some data :-)

Inspection of side FEEs (2,6,4,8) and possible grounding improvements:

- all screws and nuts for copper connections were tightenedFee

- FEE 6 was isolated from the copper connection grounding FEE2 with insulating tape (attachment 1)

- A piece of tape was found below FEE6 (attachments 2,3) --> shall this be removed?

Bias and power on of FEES, preparation of two settings with lower thresholds:

- note: cannot calibrate ADC of aida15 even after trying multiple times (?)

- 500-keV thresholds settings saved as EXPERIMENTS/AIDA/2025Oct29-09.49.01

- 200-keV thresholds settings saved as EXPERIMENTS/AIDA/2025Oct29-09.55.31

Remember that 'ADC calibration' refers to the setup of the fast serial output of the AD9222 Octal, 50MSPS, 14 bit ADC which generates the waveforms - if the FEE64 is unable to calibrate all that happens is that you will not be able to see preamp output waveforms. Energy spectra will work regardless as they use a different ADC.

During testing for the S452 experiment yesterday we were running through the steps of a restart.

The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.

Checking the folder manually it could be seen that the Options file was there.

At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.

This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.

During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.

At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally.

09:59 Noticed that the right hand ribbon cable for aida07 was offset vertically by 1. i.e. Only 1 of the 2 rows of pins in. Unbiassed detectors and reconnected.
      Rebiassed detectors - leakage current on DSSD2 increased to 9.96uA - Attachment 1

      Power cycling FEEs

      Following restart the FEEs show some improvement in rates - attachment 2
      Rate spectra - attachment 3
      Still issues with aida11 in the rate spectra (Likely a misaligned ERNI connector)



      Pulser on rate spectra - attachment 4
      Pulser off rate spectra - attachment 5
      We can see that with the pulser on there are events in the channels in aida01 and aida11
      With the pulser off these channels disappear.
      ERNI connector is then likely aligned the issue is more likely the DSSD ribbon cable is misaligned.

      Taking bias off to investigate.


10:39 Could not observe anything out of alignement about either the connectors or the adaptor boards
      Note that after pushing on the connector AIDA01 had largely recovered

13:30 Top of snout taken off while mounted in S4. It was observed that one of the ribbon cables for aida11 was off by 1 row.
      This was corrected in situ and the snout was put back together and made light tight again.
      Oddly leakage current still the same - DSSD2 has come down though - attachment 6
      FEEs powercycled for new tests

      Half the channels in aida11 had appeared again
      Took apart once more and found the other header also off by one row which has been corrected

      FEEs powercycled for new test

      All of aida11 now showing - attachment 7
      Statistics at 0xa shown in attachment 8
      All p+n for DSSD1 are good (aida01, 03, 09, 10, 11 and 12)
      p+n for DSSD2 are so so (aida05, 07, 13, 14, 15 and 16)
      All n+n 2, 4, 6 and 8 are poor still


14:50 p+n waveforms - attachment 9
      n+n waveforms - attachment 10

      Current jumper configuration
      LK1 (Bias ground) on FEEs 6 and 8
      LK2&4 on all p+n
      LK3 on aida03 and aida07 (Bottom middle)

15:00 Investigated whether there was a light leak. Placed black material over snout.
      Leakage on DSSD2 dropped to 5.6 - attachment 11
      There was a light leak
      Stats have decreased slightly - attachment 12

16:00 Move raspberry pis atop 5mm alminimum plate to keep away from FEE64 PSUs
      No change

16:30 Remove pulser network from all FEEs 
      Rates get noticably worse in many FEEs... why?
      Stats - attachment 13

      Possible ideas include braid touching things it shouldn't (Adapter PCBs?) 

Bias tests of AIDA on individual wafers and in parallel. Spreadsheet can be found in attachment 1, and graphic results can be seen in Attachment 2. Summary: Upstream detector cannot be biased in parallel nor individually. Downstream detector can be biased with positively and each wafer can individually be biased negatively. In parallel, it was not possible to bias the downstream detector negatively as indicated by the tests below.

Test downstream DSSSD with positive polarity bias

Configuration as follows:

CAEN N1419ET ch #3 connected to LHS FEE64 adaptor PCB ( looking upstream ) - LK1 *not* fitted LK1 fitted to 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cables from/to GND terminals of 3x ( top )

FEE64 adaptor PCBs *and* LHS FEE64 adaptor PCB 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything, LK3 fitted Total 5x adaptor PCBs installed

No other LKs fitted

Nominal V-I curve, stable leakage current. Attachment 3.

Following this success we attempted to repeat test using negative polarity bias

Configuration as follows:

CAEN N1419ET ch #1 connected to ( top, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cable from/to GND terminals of LHS and ( top, left )

FEE64 adaptor PCBs 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything,

LK3 fitted Total 5x adaptor PCBs installed No other LKs fitted With detector bias

-20V we continue to observe the leakage current cycling between 0 and ~2uA with a frequency ~1Hz ( as before )

Copy configuration used for upstream DSSSD test ( which was successful albeit there was detector breakdown at bias voltages > c. 90V )

CAEN N1419ET ch #1 connected to ( bottom, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( bottom )

FEE64 adaptor PCBs Ground cable jumpered from/to GND terminals LHS, ( left, bottom ), RHS and all 3x top FEE64 adaptor PCBs ( bottom, middle )

FEE64 adaptor PCB LK3 fitted Total 8x adaptor PCBs installed

No other LKs fitted With detector bias -20V we observe leakage current of ~2-3uA.

Current unstable - variations 10-100nA over periods of several seconds Although the leakage current is unstable this is an improvement over previous tests with negative bias. The duplication of upstream and downstream configurations suggests that for some unknown reason it is necessary to connect all 8x FEE64 adaptor PCBs whereas our expectation was that only 4x were necessary.

Summary: Upstream DSSSD Si wafers 1 & 2 breakdown for bias > c. 90V Si wafer 3 OK to 120V Positive bias - not tested Negative bias OK - leakage current stable to c. 90V Downstream DSSSD Si wafers 1, 2 & 3 OK to 120V Positive bias OK Negative bias - leakage current unstable

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

2) Check that all ribbon cables are properly seated in the adaptor PCBs

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

14:11 - Alpha has been running most of morning

Just saw rates in tape spike to 6 MB/s... 
Stop output to tape and look:
ASIC check & load... all rates except aida04 back to 0
Start output to tape
Back to ~300 KB/s

R6_49 will be affected by this.
Others seem OK

System wide check failures:

		 Base 		Current 	Difference
aida01 fault 	 0xb405 : 	 0xb406 : 	 1  
aida02 fault 	 0xefc7 : 	 0xefc8 : 	 1  
aida03 fault 	 0xdaab : 	 0xdaac : 	 1  
aida04 fault 	 0x8f7c : 	 0x8f7d : 	 1  
aida05 fault 	 0xb5bd : 	 0xb5be : 	 1  
aida06 fault 	 0xeff1 : 	 0xeff2 : 	 1  
aida07 fault 	 0x8f57 : 	 0x8f58 : 	 1  
aida08 fault 	 0xbef7 : 	 0xbef8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x200 : 	 512  
aida13 fault 	 0x0 : 	 0x61 : 	 97  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page)

Temps & Rates OK

Will now stop Alpha run and check noise situation

Run stopped at R6_49

Switching to nostorage (NOTAPE/R2)

Set ASICs to threshold 0xa, Rates OK
Rate to Tape/MBS ca 9 MB/s

No change to situation since yesterday... good sign

During MBS test aida13 rebooted... crash log from ttyUSB5 attached

It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again...
cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :)

A power cycle of all FEEs was performed after aida13's reboot to make sure things are good.
All systems check OK

05.01 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      9x < 20k, max 228k aida08

      per FEE64 Rate spectra - attachment 4


08.18 DSSSD bias & leakage current - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 8


14.37 DSSSD bias & leakage current - attachment 9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 12

      Writing to disk file S181/R1 (continuing alpha background file)

14.38 End run S181/R1 and start new run file S181/R2

17.09 08.18 DSSSD bias & leakage current - attachment 13

      FEE64 temperatures OK - attachment 14

      MBS correlation scaler data - attachment 15
       data to aida01, aida02, aida03 and aida10

      ADC data item stats - attachment 16
      9x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 17

      Merger/TapeServer etc - attachments 18-19