Firefox browser proxy setting changed to 'Auto-detect proxy settings for this network' ( Firefox -> Edit -> Settings -> Network Settings -> Settings )

When using the VETAR connected to the HDMI port of the root MACB the settings for all the MACB in the system 0x3.

This setting takes the Clock and Data line from the input HDMI and outputs it via teh HDMI output ports.

When using the Emulator connected via the SMA connectors on the back of the root MACB then the setting for the root MACB should be 0xD and all others should be 0x3.

Attached is the .jed file for programming the MACB and the .vhd source file to help with understanding of the settings.

The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible

Investigating AIDA noise with a TA041 differential probe and oscilloscope 

AC Mains (DESPEC platform AC, L-N)
Probe attentuation = 1:100

Fig 1: Main AC waveform [X: 5ms/div, Y: 100 V/div]
Fig 2: Zoomed in at peak (20 V FSR, any less and the waveform clipped) [X: 10us/div, Y: 20V/div]
Fig 3: Longer time base and FFT of 0-5 MHz. No significant frequency harmonics noticed [X: 5ms/div, Y:20 V, FFT X: 500 kHz/div, Y: 10 dBm/div]

No significant noise or distortion present, fully within any AC specification. 
Note that at the moment there is almost no load on AC
Equipment on on DESPEC rack: AIDA NIM, AIDA Raspberry Pis, bPlas PC (+ WR) + 2x DESPEC NIM crates
 No autofill, VME crate or detectors
All big machines at GSI (SIS, FRS) off (suspect pumps are on)

Ion catcher not on (I think under repair)

-

FEE PSU studies
Probe connected to 5V exposed power pin on FEE64 (+v) and to grounding crimp on FEE64 (-v)
No adapter board connected
Attentuation = 1:10


Fig 4: FFT when FEEs are *off* - essentially probe+scope noise [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 5: FFT when FEEs are *on* - notice 1.4 MHz peak in FFT, also seen on ADC waveform readout before (fig 6) [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]

Fig 7: 500 ns/div 5V output on FEE, single FEE on the PSU [X: 500 ns/div, Y: 100 mV/div]

Note average max voltage is 5.31 V (power on) and ~ 70 mV "peak to peak" -might be from probe/scope?

Also see voltage changes with FEE power draw:

Power on  : 5.45 V (different scale to above)
SETUP ran : 5.51 V 
FADCs off : 5.86 V
ACQ Go:   : Unchanged; ASIC threshold 0xa: Unchanged

Also check situation on a fully loaded PSU (8 fees connected and powered on)

Power on:   5.29 V (fig 8)
SETUP ran:  5.36 V (fig 9)
FADCs off:  5.64 V (fig 10)

All X: 500 ns/div, Y: 100 mV/div

Both cases observe voltage rises as current draw drops (as expected for voltage drop along a cable)

Noise on 'scope seems to get slightly worse with reduced current (and higher voltage)

No sign of strong 100 kHz noise as seen in ADC traces beforehand

Todo:

- Check -6V and 7V rails
- Check 5V and noise when front-end card is added and pulser/HV connected
- Check between two FEE64 grounds 
- Check direct out of PSU vs ground to see if 1.4 MHz appears on PSU side or FEE64 side

-
11.10.22 Updates

Attachement 11 - 5V PSU on upper PSU with no FEEs attached whatsoever. No 1.4 MHz (on FFT) but clear low frequency beats from switching - presumably low/no load behaviour

Attachement 12 - 5V PSU on aida12 with 8 FEEs on PSU. Longer time base to allow lower frequencies in FFT. 1.4 MHz switching spikes visible but nothing around 100 kHz region

Attachments 13-16: 5V PSU on aida12 at 20 mV/div vertical and 1, 0.5, 2, 5 us/div horizontal respectively

12.10.22 Updates

Attachment 17: -6V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: -6.21 V

Attachment 19: 7V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: 7.46V

Measurement between AIDA12 ground and Reference ground/copper bar
+ve (red probe) attached to copper bar at ground point (not strong connection at present)
-ve (black probe) attached to ground crimp on aida12 (connected to cooling plate)
aida12 no adapter board connected: connections are PSU, Ethernet, HDMI and TTY only

Attachment 21: 5 us/div 100 mv/div waveform, big oscillations present. Not seen before FEEs turned on (8 FEES, 1-7+12)
Attachment 22: 10 ms/div for FFT, sharp peak at exactly 100 kHz observed...


Attachment 23: Between 5V PSU (+ve) and 19" rack (-ve) with  no FEEs connected to PSU
 See strong 100 kHz oscillations too, note that voltage isn't 5V as PSU is floating w.r.t. ground
 Looks to be common mode noise (on both 5V and Return of PSU)

Attachment 24: Same as 21 but using thick crocodile clips on probe to ground and aida12. Noise is attenuated but still present

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use

Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 

The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system

In no particular order

1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)

   observe +/- 6V, 12V, 24V lines with/without load 

   try new CAEN NIM bin and/or NIM bin of different type

2) Measure actual voltages at FEE64 power connector input

   OH suggests fab of power adaptor for safe observation - contact EW

3) rev B adaptor PCB

   invert 125 way ERNI - check for mech conflicts
   paired HV input (avoid Lemo-00 T pieces)
   consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
   straight jumpers
   shrouded Samtec headers - consider mech issues/consequences of using eject clips too
   re-visit HV filtering & separate trace ground

4) isolation transformer

   as practical matter may be necessary to operate all platform from isolation transformer
   consider hire of appropriate unit

   need method to measure isolation - will require permit to work or equiv

5) investigate S4 area ac mains

   NH discussing with GSI electricians

6) Systematic measurement of AIDA PSU noise

   Spec linear AIDA FEE64 PSU

7) Redesign of snout
   Return to 1mm welded box Al for lower stage of snout  for added rigidity

8) Revisit calculation of cable lengths. Particularly for the triple

9) 



   

All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 

11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?

Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after

00:07 Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_896 - attachment 1
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6 & 7

00:15 Check ASIC control all FEE64s, all ASICs


02:03

analysis of file S505/R5_914 - attachment 8
 zero timewarps
 deadtime all FEE64s << 1% 

per FEE64 1.8.H spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 94 ch FWHM

per FEE64 stat & rate spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today


03:41

analysis of file S505/R5_926 - attachment 19
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24


07:00

analysis of file S505/R5_954 - attachment 25
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

07:45 Spoke to David and the beam has been gone since about 05:30
      Reason for the loss of beam is a vacuum issue before the FRS
      They are waiting for the experts

08:31 Statistics ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage currrents ok - attachment 3
      ASIC clock check ok
	
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2
	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  

      Currently on file R5_771
      Analysis of file R5_770 (No beam) - attachment 4
      Around 0.25% deadtime on AIDA02 rest even less

09:02 Current free HDD space 965 GB
      Current tape server rate 4979 kB/s
      Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours

11:41 Statistics ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7
      ASIC clock check ok
	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
      Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently?

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change

09:51 Taken over from Tom following the night shift
      Experiment is still running smoothly
      Compression of the files is complete up to the start of R5.
      Have started compression of files in R5 which should run up to R5_499
      Currently on R5_528

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      System wide checks:
          		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc594 : 	 87  
      aida08 fault 	 0xf1be : 	 0xf271 : 	 179  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 	Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Current merger rate is 2E6-4E6 events per second
      Current tapeserver rate is 5800 kB/s
      Free HDD space is 1.1 TB
      At current rates will last 54 hours which should see out the experiment
      Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
      
      Analysis of R5_528 - attachment 4
      Deadtime of AIDA02 currently around 15%

11:56 Statistics ok - attachment 5
      Temps ok - attachment 6
      Bias and leakage currents ok - attachment 7
      System wide checks:
      Clocks all ok 
      
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59c : 	 95  
      aida08 fault 	 0xf1be : 	 0xf27a : 	 188  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of file R5_458 - attachment 8
      Deadtime in AIDA02 only 11.5% in this file
      TapeServer rate still 5.5 MB/s

14:36 Has been no beam for the last while or so.
      With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
      Stats- attachment 9
      Temp - attachment 10
      Bias and leakage currents ok - attachment 11
      System wide checks:
      Clock ok
      	
        		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59d : 	 96  
      aida08 fault 	 0xf1be : 	 0xf27c : 	 190  
      White Rabbit error counter test result: Passed 6, Failed 2
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3c : 	 18  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

13:30 Beam taken to change an ion source

14:57 Beam back

16:21 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14

      System wide checks - ASIC clocks ok 
	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5a3 : 	 102  
      aida08 fault 	 0xf1be : 	 0xf285 : 	 199  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3e : 	 20  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Currently on file R5_592
      Analysis of R5_591 - Attachment 15
      Deadtime of AIDA02 sitting at 17%

[NH taking over for OH so he can get home]

18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
        Statitics ok - attachement 16
        Temps ok - attachement 17
        Bias & leakage ok - attachement 18

        System wide checks - 
        Clocks OK
        ADC Calibration N/A 
	WR Decoder -
        		 Base 		Current 	Difference
        aida07 fault 	 0xc53d : 	 0xc5ac : 	 111  
        aida08 fault 	 0xf1be : 	 0xf28c : 	 206  
        White Rabbit error counter test result: Passed 6, Failed 2
        FPGA -
         Base 		Current 		Difference
        aida07 fault 	 0x2a : 	 0x3f : 	 21  
        PLL OK

        Currently on file R5_621
        Analysis of R5_620 - Attachement 19
        aida02 deadtime only 5%, all others negligible 

20:20 Stats ok - attachment 20
      Temps ok - attachment 21
      Bias and leakage currents ok - attachment 22
      System wide checks:
      Clock ok
    	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ad : 	 112  
      aida08 fault 	 0xf1be : 	 0xf293 : 	 213  
      White Rabbit error counter test result: Passed 6, Failed 2

     	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3f : 	 21  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R6_637 - attachment 23

22:13 Statistics ok - attachment 24
      Temperatures ok - attachment 25
      Bias and leakage currents ok - attachment 26
      ASIC clock check ok

	
		         Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ba : 	 125  
      aida08 fault 	 0xf1be : 	 0xf2a2 : 	 228  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x40 : 	 22  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_658 - attachment 27

23.43 Check ASIC control
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

All system wide checks OK *except* WR & FPGA errors - attachments 1 & 2

DSSSD bias & leakage currents OK - attachment 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6

analysis of file S505/R5_415 - attachment 7
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 1%



02:36

analysis of file S505/R5_449 - attachment 8
 zero timewarps
 max deadtime aida02 c. 15%, all other FEE64s < 1%

per FEE64 rate & stat spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 99 ch FWHM

per FEE64 1.8.H spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


05:38

analysis of file S505/R5_483 - attachment 19
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24



07:27

analysis of file S505/R5_502 - attachment 25
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

Took over the night shift from Magda

18:00 : attachments 1-3

20:00 : attachments 4-6

22:00 : attachments 7-9

00:00 : attachments 10-12

Took over the night shift from Tom.

7:00 The beam is back.

8:00 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

14:00 : attachments 10-12

16:00 : attachments 13-15