Initial results: copy from DESPEC elog.

We have managed to obtain the time difference between the bPlast White rabbit and the AIDA Fast time. We had to gate out the zero fast-time events and then also condition that we only take the data from aida10. We see this in 684/1. Then if we look at the time difference between the fast time and the bPlast WRT we see a sharp peak at zero, this makes sense as the AIDA fast time discriminator is being triggered by a pulser which is then being used as a trigger for bPlasts DAQ so these events should be virtually arriving without delay. The delay we do see is infact around 750 ns.

The data collected overnight was also analysed as shown in 684/3 this is the data accumulated with the 22Na source close to the snout and a global threshold in the fast discriminator of 0x0f. The centre peak was roughly fitted with a gaussian:    

  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  Constant     1.21277e+03   2.74526e+01   5.66915e-01  -5.61074e-07
   2  Mean        -1.78513e+02   8.50699e+00   1.62947e-01   1.58554e-06
   3  Sigma        3.48571e+02   4.03993e+00   6.26343e-05  -1.86863e-03

Yesterday we spent time essentially performing a dry run to get AIDA DAQ and bPlast into the time sorter, there were some issues with the AIDA mbs PC x86l-119 which was related to some boot issue, it was booting to a newer version of debian while the Relay for AIDA to MBS is on an older version (scratch) (the machine is quite old). We spent a bit of time getting the thresholds correct for each of the ASICs. In the end we just elected to have a blanket level of 0x0f for all of the ASICs when we removed the pulser.

Last night we left the setup with the following thresholds on all of the FEEs to collect data. 681/1. The pulser was also turned off and the 22Na source was moved close to the snout.

This morning c. 9:25 we returned and checked the temperatures, HV and statistics and everything seems to be ok. The only problem was that the timesorter has crashed this morning at 8 am.

WR timestamp errors resolved after reseating the HDMI cables to the MACB

Bias tests of AIDA on individual wafers and in parallel. Spreadsheet can be found in attachment 1, and graphic results can be seen in Attachment 2. Summary: Upstream detector cannot be biased in parallel nor individually. Downstream detector can be biased with positively and each wafer can individually be biased negatively. In parallel, it was not possible to bias the downstream detector negatively as indicated by the tests below.

Test downstream DSSSD with positive polarity bias

Configuration as follows:

CAEN N1419ET ch #3 connected to LHS FEE64 adaptor PCB ( looking upstream ) - LK1 *not* fitted LK1 fitted to 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cables from/to GND terminals of 3x ( top )

FEE64 adaptor PCBs *and* LHS FEE64 adaptor PCB 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything, LK3 fitted Total 5x adaptor PCBs installed

No other LKs fitted

Nominal V-I curve, stable leakage current. Attachment 3.

Following this success we attempted to repeat test using negative polarity bias

Configuration as follows:

CAEN N1419ET ch #1 connected to ( top, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cable from/to GND terminals of LHS and ( top, left )

FEE64 adaptor PCBs 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything,

LK3 fitted Total 5x adaptor PCBs installed No other LKs fitted With detector bias

-20V we continue to observe the leakage current cycling between 0 and ~2uA with a frequency ~1Hz ( as before )

Copy configuration used for upstream DSSSD test ( which was successful albeit there was detector breakdown at bias voltages > c. 90V )

CAEN N1419ET ch #1 connected to ( bottom, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( bottom )

FEE64 adaptor PCBs Ground cable jumpered from/to GND terminals LHS, ( left, bottom ), RHS and all 3x top FEE64 adaptor PCBs ( bottom, middle )

FEE64 adaptor PCB LK3 fitted Total 8x adaptor PCBs installed

No other LKs fitted With detector bias -20V we observe leakage current of ~2-3uA.

Current unstable - variations 10-100nA over periods of several seconds Although the leakage current is unstable this is an improvement over previous tests with negative bias. The duplication of upstream and downstream configurations suggests that for some unknown reason it is necessary to connect all 8x FEE64 adaptor PCBs whereas our expectation was that only 4x were necessary.

Summary: Upstream DSSSD Si wafers 1 & 2 breakdown for bias > c. 90V Si wafer 3 OK to 120V Positive bias - not tested Negative bias OK - leakage current stable to c. 90V Downstream DSSSD Si wafers 1, 2 & 3 OK to 120V Positive bias OK Negative bias - leakage current unstable

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

2) Check that all ribbon cables are properly seated in the adaptor PCBs

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

17.32 Power and detector bias cycle

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2
       *except* aida02 ASIC temp u/s

      ASIC settings 2024Mar27-11.25.32
       slow comparator p+n FEE64s 0xa (100keV), n+n FEE64s 0xf (150keV)

      BNC PB-5 pulser
      amplitude 10.0V
      attenuation x10
      frequency 25Hz
      polarity -
      tau_d 1ms
      tail pulse

      test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

      test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

      All system wide checks OK *except* aida02 & aida03 WR decoder status errors - attachment 10

      WR timestamps OK - attachment 11

      ADC data item stats - attachment 3
       12 of 16 < 20k, all < 100k

      per FEE64 Rate spectra - attachment 4

      per n+n FEE64 1.8.L spectra - attachment 5

      1.8.W spectra - 20us FSR - attachments 6-9
       preamplifier output noise generally very good

*Current ground configuration*

CAEN N1419ET LK fitted (non floating outputs)

ground cable from aida04-aida12-aida02-aida09-aida01-aida05
HV#0 aida12-aida03-aida15

ground cable from aida08-aida16-aida06-aida10-aida14-aida13
HV#1 aida16-aida07-aida11

LK3 fitted aida03, aida07

LK1 fitted aida02, aida04, aid06, aida08

test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

all DSSSD ribbon cable drain wires grounded to their resepctive AIDA adaptor PCBs

JB observed open circuit between AIDA Al snout and AIDA support frame ( as expected )

AIDA PSU cabling as reported https://elog.ph.ed.ac.uk/DESPEC/560

Snout configuration:

- all bPlast cables are disconnected and floating. 
- CC taped up all the cables and exit points in snout.
- CC, JB covered the exit of the snout with two additional layers of aluminium foil which were also taped shut.


To Do:

- repower detector, see if similar conditions persist.
- disconnect test - pulser, again see if conditions persist.

- aida04 asic#1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?




       

09.38 CC completed install of Bplast driver PCBs yesterday evening.
      All flat ribbon cables connected - all drain wires grounded
      PSU on but not enabled - return terminals grounded to PSU front panel ground
      SiPm bias off

      Water pressure and temperature OK
      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      ADC data item stats - attachments 5

      per FEE64 Rate spectra - attachments 6-7

      per 1.8.W spectra - 20us FSR - attachments 8-11

      ASIC settings 2024Mar27-11.25.32
       LEC slow comparator p+n FEE64s 0xa, n+n FEE64s 0xf

      BNC PB-5 pulser - attachment 12



10.45 CC returns
      bPlas ON

      ADC data item stats - attachments 13
       8x < 20k, max c. 310k

      per FEE64 Rate spectra - attachments 14

      per 1.8.W spectra - 20us FSR - attachments 15-16

      downstream DSSSD n+n and bottom left & right p+n FEE64s noisy


11.40 per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 55 ch FWHM ~38keV FWHM - no change cf. before installation of bPas *except* aida16


12.00 Photos of snout, bPlas driver PCBs, cabling, grounding and PSUs courtesy JB - attachments 18-27

      N.B outputs of PN300 PSU at base of AIDA support stand are *not* ground ref'd - attachment 23


12.20 Slow comparator -> 0x64
      Pulser OFF
      All histograms zero'd

13.17 JB: returned from lunch. Current status of AIDA modules given by attachment 29.

15.55 While bPlast thresholds were being set the noise increased substantially.

16.51 Replaced mezzanine of aida14, reinstalled and biasing detector. Resulting for noise conditions in the detector given by attachments 30-33.

17.31 With a multimeter it was found that there is continuity between the snout and the frame.

18.50 We tried to disconnect the cables of the short side of the bPlast detector and all of the grounds. This seemed to show an open line OL on the multimeter and whence connecting the bPlast grounds back excluding the short side ribbon cable grounds, the multimeter still read OL. 
      We also tried to wedge paper between the short side ribbon cable of bPlast and the snout, but this did not work, the detector still reading continuity between the frame, booster board and snout. 
      The results after booting up the detector again are given by the attachments 34-36. 
      The noise condition is appreciably better than before with 12 out of the 16 FEE64 modules with sub 20 kHz rates.
      Disconnect the BB7 preamp. ground from the frame.

18.54 We powered up bPlast. The noise condition three FEEs got worse, aida01, aida11 and 06. The problem may be associated with downstream grounding. The results are given by the attachments 37-43. 10 out of the 16 FEE64 modules showed sub 20 kHz rates.

19.07 We turned off the power supply on the base of the snout support. PN300 Attachment 23. Rates did not change - attachment 44.

19.10 We turned off the mesytec PSU that is located in the bPlast NIM crate. No change was observed in the rates - attachment 45.

19.13 We turned off the R&SRMP4040 PSU that is located above the AIDA crate. And the noise situation did not change attachment 46.

19.22 Leaving the R&SRMP4040 PSU off we turned back on the mesytec PSU and PN300 PSU, attachment 47.

Summary:

It is clear that with no continuity on the short side we are able to reduce the noise back to the scenario where bPlast was not connected. However, powering bPlast introduced substantial noise in both DSSSDs that did not go away when turning all the PSUs off. There might be some hysteresis in the system (this is just speculation).

The situation is still fair considering that 10 out of the 16 FEE64 modules are in the sub 20 kHz rate level (good noise condition).

TO-DO for 11.04.2024

- Try bringing bPlast ground back to the PSU ground.

- Recheck the downstream detector bias and ground scheme.
       

12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3

      During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
      aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )

      AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
       *except* aida02 ASIC temp - known fault

      System wide checks OK *except* WR/FPGA errors - attachments 3-4

      WR timestamps OK - attachment 5

      ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
       ADC data items 10/16 < 20k, max 143k

      per FEE64 Rate spectra - attachment 11

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-13


15.10 Synchronise ASIC clocks
      Re-calibrate ALL ADCs 

      ASIC settings 2024Mar27-11.25.32

      Changes ( https://elog.ph.ed.ac.uk/CARME/499 )

       IBias LF feedback from 0xf to 0x8
       Diode link threshold from 0xbf to 0xca

      New ASIC settings saved as 2024Apr19-15.22.49

      All FEE64 slow comparators -> 0x14

      Data file S100_alpha/R12

      Pulser ealkthrough (test +)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity +
      Tail pulse
      tau_d 1ms

16.35 DAQ ends OK file S100_alpha/R12_24

      per p+n FEE64 1.8.L spectra - attachment 14
       aida09 pulser peak width 54 ch FWHM 

17.38 Data file S100_alpha/R13

      Pulser walkthrough (test -)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity -
      Tail pulse
      tau_d 1ms

18.43 DAQ ends OK file S100_alpha/R13_20

      per p+n FEE64 1.8.L spectra - attachment 15

18.55 Current status

      FEE64 power ON
      DSSSD bias ON

      All FEE64 slow comparator 0x14

      DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )


To Do

- S4 currently closed/controlled access for FRS startup

If we have access to S4 tomorrow

- switch test - to test + ( not critical )
- test AIDA interlock 
- further alpha background?



22.47 DSSSD bias & leakage current OK - attachments 16-17

      FEE64 temps OK - attachment 18
       *except* aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Data link, Tape Server & Merger - attachments 21-23

       

Status before dismount

14:26 DSSSD#1 119.99 V @ 17.968 uA

          DSSSD#2 119.95 V @ 11.212 uA

          Leakage currents slightly elevated from Monday.

S4 temperature at 26 °C.

14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.

Plan:

- Dismount snout.

- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

16:30 We moved the platform in. The system was still biased and the DAQ was still running.

The system seems to be okay, temperatures OK. Bias voltage OK 685/1 and 685/2. Two FEEs - aida07 and aida12 are receiving a lot of trigger data items from the fast discriminator. This is already a bit worrying. 685/3

15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.

           Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.

           Histograms okay, some FEE64s now have hot channels. Attachment 2.

           9/16 aidas < 20kHz - max rate 162k

18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.

TO-DO (kicking the can down the road):

- Some work to do on noise, but we might have to accept the situation as it currently is.

- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.

- Try different grounding configuration:

    - Grounding drain wires to frame. Observe AIDA conditions.

    - Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.

15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).

          TEMP OK.

          Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.

          9/16 <20kHz, max 176k

15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.

          No change in noise observed.

          9/16 <20kHz, max 187k

15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.

          10/16 <20kHz, max 158k.

Current bPlast grounding scheme in attachments 8-9.

Three BB18-1000 triples AIDAs collected on 14.09.2024

Find attached visual of the wafer and bond wire + factory bias tests accompanying the DSSSDs. elog:669/1 elog:669/2 elog:669/3

Visual inspection carried out showed that bond wires have been fixed + fingerprint on one DSSSD removed and wires repaired. elog:669/4 elog:669/5 elog:669/6 elog:669/7 elog:669/8 elog:669/9 elog:669/10 elog:669/11

DSSSD 1 (defect bias issue 80V): 3208-10 / 3208-18 / 3208-20

DSSSD 2  (3208-6 dysfunctional): 3208-6 / 3208-9 / 3208-16

DSSSD 3 (defect fingerprint): 3131-5 / 3131-10 / 3131-12

AIDA analysis and presentation by J. Bormans for AIDA made in the HISPEC-DESPEC collaboration meeting.

https://docs.google.com/presentation/d/1hlZ30r294UqbVKGy3jg-wompFpc7TSfD/edit?usp=sharing&ouid=102131181856760114019&rtpof=true&sd=true

Snout assembly

DSSSDs

Upstream   3208-10/3208-18/3208-20
Downstream 3131-5/3131-10/3131-12

p+n junction side bias continuity checked OK for *all* wafers
 all c. 22 Ohm as expected

n+n Ohmic side bias continuity checked OK for both DSSSDs
 upstream 73 Ohm, downstream 92 Ohm

n+n Ohmic side bias wafer #2 
 positioned at top of snout assembly ( lower stage snout side marked 'T' )

p+n junction side faces downstream


Distances

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.0cm
ref - upstream DSSSD 11.0cm
ref - downstream DSSSD 12.0cm
ref - middle of downstream bPlas 14.0cm

Measurements consistent to +/-1mm between LHS and RHS of snout assembly as viewed from top side 


13.00 At this position the Kapton PCBs bulged in/out and there was some concern that when the upper stage
snout was installed it would the Kapton PCBs into the 'active' area. It was decided to move the 2x bPlas
and 2x DSSSDs c. 5mm downstream.

Jeroen and Phillip cut sections from 4x PEEK 6mm spacers so that they would clip onto 3mm dia support rods.

We raised the assembly fro the inter stage and inserted the modified spacers between the inter stage and 
the other spacers above. The Kapton PCBs now run fairly straight to the DSSSD connectors.

Distances re-measured as

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.6cm
ref - upstream DSSSD 11.5cm
ref - downstream DSSSD 12.4cm
ref - middle of downstream bPlas 14.5cm

EMC tests with NK (Orsay)

Stack configuration (upstream->downstream)

bPlas 
AIDA DSSSD #0
AIDA DSSSD #1
bPlas
3x MSL type BB7(DS)-1000 (1x aida10, 2x Mesytec preamplifiers)

AIDA DSSSD #0 bias -100V leakage current -13.23uA
AIDA DSSSD #1 bias -100V leakage current -9.68uA

BNC PB-5 settings
Amplitude 1.0V
Attenuator x1
Frequency 25Hz
Polarity +
tau_d 1ms
Tail pulse

14.39 per p+n FEE64 1.8.W spectra - 20us FSR - attachment 1

      per p+n FEE64 1.8.L spectra - attachment 2

      Pulser peak widths ch FWHM

      DSSSD #0 (DSSSD - FEE64 cables with heavy duty braid - electrically connected to ground of adaptor PCB and internally connected to ground within snout)

9  1   5
120 201  21
15  3  12
114 180  broad

      DSSSD #1 (DSSSD - FEE64 cables with mesh - currently not connected i.e. floating to adaptor PCB and internally connected to ground within snout)
10 14  13
-   22   374
11  7  16
broad 417 20

       Pulser peak widths c. 20 ch FWHM probably indicate cable from DSSSD *not* connected to FEE64 adaptor PCB

        

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

16:51 Bias and leakage currents ok but starting to increase in DSSD 1 predominantly - attachment 1
      Statistics ok - attachment 2
      FEE temp ok - attachment 3
      System wide checks all ok 
     
16:57 Histograms zeroed

17:17 ASIC control check done

17:40 Layout 1 - attachment 4
      Layouts 3 and 4 - attachments 4 and 5
      Layouts 5 and 6 - attachments 6 and 7

19:06 Earlier in the shift two extra 8TB drives were formatted and mounted
      Named SecondDrive and ThirdDrive
      TapeData folder created on SecondDrive
      Plan is before the end of this shift to move the symbolic link from the filling up drive to the new drive
      Current prediction is drive would otherwise be full by 12:20 tomorrow.
      Playing it safe and switching early

21:57 Bias and leakage ok - attachment 8
      Statistics ok - attachment 9
      FEE Temps ok - attachment 10

0.00 checks - all looks ok

Voltage/currents - attachment 1

Rates attachment 2 + 3

Temps - attachment 4

Merger - attachment 5

ucesb - attachment 6

Grafana - attachment 7

1.30 -aida1 in merger Links, aida02 in rate spectra, has stopped - resetting DAQ.

Datafile R7_961 being written when stopped

1.40 - seems ok

1.50 checks - seems ok

Voltage/currents - attachment 8

Rates attachment 9 + 10

Temps - attachment 11

Merger - attachment 12

ucesb - attachment 13

Grafana - attachment 14

3.30 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1010 being written when stopped

3.38 - seems ok

3.40 checks - all looks ok

Voltage/currents - attachment 15

Rates attachment 16 + 17

Temps - attachment 18

Merger - attachment 19

ucesb - attachment 20

Grafana - attachment 21

Zeroed spectra - attachments 22-28

5.39 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1062 being written when stopped

AIDA out of the main data

Could not get it to restart folowing the notes on the elog - SOAP errors as shown in attachment 29. called Tom who''s logged in 

6.10 - seems to be working again. Not sure what happened. Tom thinks multiple restarts fixed it

Started data file R8

AIDA back in the main data

FRS shifters report a problem with the AIDA white rabbit time stamp - attachment 30

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  17D8C4ED 8F46CF7B , WR/10=>   2627A17C18714BF, Readout Time =>   2627A17C2A74000
aida02 : White Rabbit=>  17D8C4ED A0A1B623 , WR/10=>   2627A17C3435F03, Readout Time =>   2627A17C4410000
aida03 : White Rabbit=>  17D8C4ED AFDACE68 , WR/10=>   2627A17C4C914A4, Readout Time =>   2627A17C5D3C000
aida04 : White Rabbit=>  17D8C4ED C0516FC2 , WR/10=>   2627A17C66E8B2D, Readout Time =>   2627A17C7AC4000
aida05 : White Rabbit=>  17D8C4ED D3B93DAB , WR/10=>   2627A17C85F52F7, Readout Time =>   2627A17C9AEC000
aida06 : White Rabbit=>  17D8C4ED E8C9B243 , WR/10=>   2627A17CA7A91D3, Readout Time =>   2627A17CB704000
aida07 : White Rabbit=>  17D8C4ED FFE475FD , WR/10=>   2627A17CCCA0BCC, Readout Time =>   2627A17CDD94000
aida08 : White Rabbit=>  17D8C4EE 103232A6 , WR/10=>   2627A17CE6B6B77, Readout Time =>   2627A17CF7BC000
aida09 : White Rabbit=>  17D8C4EE 20A1F612 , WR/10=>   2627A17D0103235, Readout Time =>   2627A17D112C000
aida10 : White Rabbit=>  17D8C4EE 307D8FE7 , WR/10=>   2627A17D1A627FD, Readout Time =>   2627A17D2958000
aida11 : White Rabbit=>  17D8C4EE 4402829D , WR/10=>   2627A17D399D9DC, Readout Time =>   2627A17D4570000
aida12 : White Rabbit=>  17D8C4EE 6288B04D , WR/10=>   2627A17D6A744D4, Readout Time =>   2627A17D7D94000
aida13 : White Rabbit=>  17D8C4EE 751E1306 , WR/10=>   2627A17D88301E7, Readout Time =>   2627A17D9C34000
aida14 : White Rabbit=>  17D8C4EE 894E8806 , WR/10=>   2627A17DA87DA67, Readout Time =>   2627A17DBFF4000
aida15 : White Rabbit=>  17D8C4EE 9EEEB997 , WR/10=>   2627A17DCB178F5, Readout Time =>   2627A17DDC18000
aida16 : White Rabbit=>  17D8C4EE AF0BC923 , WR/10=>   2627A17DE4DFA83, Readout Time =>   2627A17DF5C8000

Tom thinks everything appears ok with AIDA

Have converted WR timestamp to data/time - see attachment 40

At the ~second timescale AIDA WR timestamp looks OK, i.e. no gross errors. Will need to check correlations with other detector sub-systems to confirm WR on the ~us time timescale.. 

However, ~10us offset reported by DESPEC online crew is correct (AIDA timestamps data at a later point in the signal processing cycle than other detector sub-systems). Have suggested online crew contact Nic or Calum to check what the AIDA WR timediff spectra should look like.

7.00 checks - all looks ok

Voltage/currents - attachment 31

Rates attachment 32 + 33

Temps - attachment 36

Merger - attachment 37

ucesb - attachment 39

Grafana - attachment 38

Handed over to Tom

8am checks

temps ok - attachment 2

rates ok - attachment 8

voltage ok - attachment 5

merger ok - attachment  6

9am checks

temps ok - attachment 10

rates ok - attachment 9

voltage ok - attachment 12

merger ok - attachment  11

(Ignore  Grafana plots 13-43 inc - not refreshed)

9.16 beam stopped to optimise beam intensity

10.00 checks - still no beam

temps ok - attachment 17

rates ok - attachment 15+16

voltage ok - attachment 14

merger ok - attachment  18

11.00 checks - still no beam

temps ok - attachment 23

rates ok - attachment 21+22

voltage ok - attachment 20

merger ok - attachment  24

12pm checks

temps ok - attachment 28

rates ok - attachment 26+27

voltage ok - attachment 29

merger ok - attachment  30

12.45 - beam back

temps ok - attachment 34

rates ok - attachment 32+33

voltage ok - attachment 35

merger ok - attachment  36

2pm checks

temps ok - attachment 40

rates ok - attachment 38+39

voltage ok - attachment 41

merger ok - attachment  42

/tmp/R4_276 added - attachment 44

max deadtime 10.5% (aida08)

n+n FEE64s deadtime 

3pm checks

temps ok - attachment 48

rates ok - attachment 46+47

voltage ok - attachment 45+50

merger ok - attachment  49

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :     25     10      4      3      1      2      2      3      3      3      6   : 36500
aida03 :     27      8      5      2      3      4      3      3      2      3      6   : 36092
aida04 :     30      4      4      5      0      2      4      2      3      3      6   : 36472
aida05 :     19      5      7      1      3      2      2      2      2      4      6   : 37060
aida06 :     18     13      1      2      1      3      2      3      3      3      6   : 36544
aida07 :     24      7      3      2      2      3      1      3      2      4      6   : 37384
aida08 :     15     11      2      5      2      4      1      4      3      3      6   : 37076
aida09 :      7      3      4      1      5      6      5      4      3      2      6   : 36308
aida10 :      8      3     13      4      3      3      1      4      2      3      6   : 36040
aida11 :     18     11     17      7      4      4      4      3      2      3      6   : 36752
aida12 :     12      7      7      7      4      4      2      3      2      3      6   : 36024



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021



17:27.  Checked system stats, temps, leakage etc. looks ok 


18:49. Tape server writing to disk /NULL/R21_9+
       /TapeData points to /media/SecondDrive/ - 3.6Tb free

19.15. checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9     16      2      2      2      1      3      3      3      6   : 36424
aida02 :      1      3      5      4      1      3      3      2      3      3      6   : 36268
aida03 :     24     13     11      6      3      3      2      4      2      3      6   : 36472
aida04 :      5      4      4      4      3      3      3      2      3      3      6   : 36404
aida05 :     17      8      4      2      2      2      2      2      2      4      6   : 36996
aida06 :     26     11      3      2      0      3      2      3      3      3      6   : 36528
aida07 :     24      7      2      1      1      1      2      3      2      4      6   : 37272
aida08 :     13     12      6      3      3      4      1      4      3      3      6   : 37140
aida09 :     22     15     19      2      6      5      4      4      3      2      6   : 36416
aida10 :     16     11      6      4      3      3      1      4      2      3      6   : 36024
aida11 :     12      4      5      4      2      3      3      3      2      3      6   : 35872
aida12 :     18     12      9      4      4      4      2      3      2      3      6   : 36024


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


19.53.  Checked system stats, temps, leakage etc. looks ok 

20:30.  Checked system stats, temps, leakage etc. looks ok 

20:59.  Checked system stats, temps, leakage etc. looks ok 



21:31 System checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      9     17      3      2      2      1      3      3      3      6   : 36492
aida02 :      5      1      1      6      0      2      2      3      3      3      6   : 36332
aida03 :      2     10     11      6      4      4      3      3      2      3      6   : 36296
aida04 :     27     12      3      4      0      2      3      2      3      3      6   : 36220
aida05 :     19      9      7      2      1      2      2      2      2      4      6   : 36996
aida06 :     28      5      0      4      0      3      1      3      3      3      6   : 36248
aida07 :     19     13     10      6      2      2      1      3      2      4      6   : 37524
aida08 :     18      9      6      0      1      4      1      3      3      3      6   : 36400
aida09 :     17     19     12      3      6      5      4      4      3      2      6   : 36348
aida10 :     23      9      7      4      2      3      1      4      2      3      6   : 35988
aida11 :     19     10      3      3      2      3      3      3      2      3      6   : 35884
aida12 :     12      6      0      3      2      4      2      3      2      3      6   : 35648


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


22.06.  Checked system stats, temps, leakage etc. looks ok 

22.18CEST BNC PB-5 amplitude changed from 1V to 2V
          fioler NULL/R21_67

22.23CEST All histograms zero'd

22.43.  Checked system stats, temps, leakage etc. looks ok 
23.21.  Checked system stats, temps, leakage etc. looks ok 


23.46 system checks
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     17     12      7      0      3      3      2      4      2      3      6   : 36180
aida02 :     35      3      5      2      0      2      1      3      3      3      6   : 36148
aida03 :     22      6     10      2      1      3      2      4      2      3      6   : 36136
aida04 :     27      9      3      5      2      4      3      3      2      3      6   : 36100
aida05 :     20      9      3      3      2      2      2      2      2      4      6   : 37032
aida06 :     25     11      1      2      0      3      1      3      3      3      6   : 36236
aida07 :     20      6      6      2      3      3      2      4      1      4      6   : 37216
aida08 :     25     10      0      2      1      3      1      3      3      3      6   : 36276
aida09 :     10      4      3      4      4      5      4      4      3      2      6   : 35960
aida10 :     26      9      3      0      1      3      1      4      2      3      6   : 35744
aida11 :      2      3      4      2      2      3      3      3      2      3      6   : 35744
aida12 :      4      9      7      2      2      4      2      3      2      3      6   : 35720
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021