10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

Double peaking normally indicates periodic baseline variations with period comparable to shaping time

Rates in all FEEs much higher than before - waveform shows very noisy in all systems. - attachment 1

Good event statistics - attachment 2

DISC in 7 and 3 above 300k

ASIC check load performed.

Good event statistics - attachment 3

Thesholds at 0xa for all FEE

Disc rate now 0 across all FEE


Pulser settings: attachment 4

pulser peak widths -   FEE         width(ch)
                              1             128
                              2             145
                              3             84
                              4             74
                              5             207
                              6             125
                              7             172
                              8             80
                              9             193
                              10            183
                              11            238
                              12            138

Attachments 5 & 6: pulser peaks for all FEE64

16.24: DAQ stop

       slow comparator threshold changed to 0x64

       ASIC check performed

       Alpha run/DAQ start (w/ data transfer enabled) - writing to file 31Oct19/R13

       Merger and TapeService ok - attachments 7 & 8

ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2

Analysis of approx 20GB of data since Thursday night (not continuous - estimate of hours incominG)

Figures 1-3: Energy Front vs Energy Back (No F-B energy gate)

Figures 4-6: 2D hit pattern

Figures 7-9: 1D hit pattern

aida10 does not seem to be reading out still, unsure of reason? Looks fine with pulser.
Slow comparator not setting or unusual gain?

Otherwise good

All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.

HDMI cables in aida09 checked and good.

> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > 
> > HDMI cables in aida09 checked and good.
> 
> The problem would be the cable , one end or the other. 
> I think ( if I recall ) a setup would restart the WR decoder.
> 
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

Cable will be replaced once a spare is available.

Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.

WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

> > > All 12 FEEs have valid WR Timestamps
> > > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > > 
> > > HDMI cables in aida09 checked and good.
> > 
> > The problem would be the cable , one end or the other. 
> > I think ( if I recall ) a setup would restart the WR decoder.
> > 
> > I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
> 
> Cable will be replaced once a spare is available.
> 
> Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
> 
> WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

An update/clarification:
Although the upper bits are zero I believe actually it is a total failure to synchronise to WR:

aida09
WR Time Item 0x80500000 0x0fbd8000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80400249 0x0fbd8000; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80500000 0x0fbdc000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80400249 0x0fbdc000; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80500000 0x0fbe0000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
WR Time Item 0x80400249 0x0fbe0000; Time (28:47)=0x249; Time (0:27)=0x0fbe0000

aida10
WR Time Item 0x8050022e 0x0bde8000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x804e29d5 0x0bde8000; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x8050022e 0x0bdec000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x804e29d5 0x0bdec000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x8050022e 0x0bdf0000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
WR Time Item 0x804e29d5 0x0bdf0000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

Attached is corrupted database and correct database

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could well cause you problems with readout as it won't understand some of the new data.

Hope this helps.

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

If ypou now check the actual values in the aida10 Options screen are the correct ?

It can happen that corruptions once loaded are propagated when Options are generally saved. 

Looking at the dates the files were modified does that make sense to you. It doesn't look as if the corrupted database was changed except when they all were ? Can you check ? Have you checked the other Options files ? I seem to recall that RIKEN AIDA had a program to to this ?

Alpha run has been running over the weekend. Stats looking positive with notable exception that aida10 is not sending much data. Will attempt a powercycle.

Temperatures and Leakage Currents very good to excellent

Noted that the slow rate of aida10 seems to be slowing down the rate data is sent to tape - perhaps being buffered until an ADC event arrives?

Update 05.11.2019

Most rates OK, aida10 only recorded 1 event over night, aida09 spectra had a number of weird channels.
Performed ASIC check/load, aida10 found another 201 events but mostly in the HEC channel (as did all FEEs)
All other FEEs look very nice however

15:44 CET: Check/load performed as aida07 rate went very high (in HEC it seems) - now back to 0

Update 07.11.2019
Run stopped to move DESPEC platform, overnight the USB relay disconnected/reconnected powering down the FEEs (probably someone knocked the interlock wire)

3x MSL type BB18(DS)-1000 installed 

detector bias -160V, leakage current c. 1uA @ +21 deg C for all 3x DSSSDs

all FEE64 good event rates (slow comparator 0xa, LEC fast comparator 0xff) c. 120k, or less, typically 30-50k, overall merge rate c. 1.6M data item/s

See https://elog.ph.ed.ac.uk/DESPEC/70



Outstanding issues

- occasional loss of bits 48-63 of WR timestamp for aida09 - can be recovered by power cycle - replace HDMI cable?

- aida09 asic 1 not producing ADC or disc data - replace aida09? 
  ASIC Check works OK
  aida09 asic temp c. 30deg C < other asic temps

- evidence of c. 1MHz extrinsic noise for most FEE64s

- acquisition of waveform data not robust - most ASIC channels do not produce data or quickly stop producing data  - PJCS to review firmware rev for Jan/Feb 2020

- require spare DSSSDs, FEE64s, HDMI cables etc

>
> 
> - evidence of c. 1MHz extrinsic noise for most FEE64s


Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom

11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding

Strange error about Database options appearing during run stop.
Does not seem to affect stopping/starting/data?

Beginning Alpha run

Thresholds set to 100 (LEC/Slow) 255 (LEC/Fast) 2 (HEC)

Pulser is OFF

Recording to MIDAS: /TapeData/31Oct19/R1

Recording to MBS: /lustre/gamma/nhubbard/AIDA/Alphas311019_

Disk rate approximately 0 as expected.

01.11.19 19:11 CET: FEE10 Stat histogram was empty - performed a check/load and now rare events come in - Merger also emitted 10 blocks to storage quickly (presumably waiting on FEE10 data)
All FEEs now slowly showing alpha data

Testing merger with 3 DSSDs connected, with thresholds of 10 (slow comparator), 2 (HEC) and 255 (fast discriminator)

Merger handling rate of 1.6 million events per second comfortably. Forwarding to MBS fine.

Network usage: 130 Mbps (AIDA network) and 20 Mbps (MBS network)

1) Login to Raspberry Pi 

ssh pi@nnrpi2

2) Kill any instances of the TclHttpd server

ps -ef | grep TclHttpd
kill -9 <pid>

3) Start server

/MIDAS/TclHttpd/linux-arm/TclHttpd-server &

10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
      Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%

10.15 DSSSD stack as follows

      DSSSD  serial  thickness  depletion 
      #              (um)       voltage (V)
      1      3208-7  1019       125              upstream
      2      3208-13 1020       120
      3      3208-14 1020       130              downstream

11.30 AIDA power relays sequenced on
      
      DAQ start

      system wide checks ok

11.33 attachment 2 - fee temperatures - ok

      attachment 3 - good event statistics with slow comparator threshold = 100

      attachment 4 - good event statistics with slow comparator threshold = 50

      attachment 5 - good event statistics with slow comparator threshold = 20

      attachment 6 - good event statistics with slow comparator threshold = 10
     
13.51 attachment 7 - Rates
                   - aida09 1 ASIC not working (no data)

      attachment 8 - good event statistics

      attachments 9/10 - 1.8.L spectra
                       - no peak observed in aida09

      attachments 11/12 - 1.8.W spectra
                        - waveform only observed in naida 2,4,7,8,12

      pulser peak widths -   FEE         width(ch)
                              1             71
                              2             96
                              3             66
                              4             79
                              5             74
                              6             96
                              7             89
                              8             83
                              9           no peak
                              10            82
                              11            153
                              12            120

17.26 FATIMA moved upstream around AIDA

      attachment 13 - good event statistics

      ASIC check-load performed

      attachment 14 - good event statistics following checkload

12.51 - detector bias and leakage currents - detector 3 has no current
                                           - other detectors ok

13.13 - tried changing from HV3 cables to HV4 - still open circuit 

      - tried different permutations of moving HV-4 braid and core cables (high voltage to ground connections)

       (dssd3)

      (all directions defined as if looking upstream)

      - core -> bottom, braid -> left     open - circuit

      - core -> top, braid -> right       trips at 8V

      - core -> bottom, braid -> right     open - circuit

      - core -> top, braid-> left          trips at 8V

      - removed jumpers - still trips

13.36 - testing continuity of bond wires on dsssd 3208-14 & 3208-15

      - observe 20 ohm resistance on p+p and p+n bond wires 

19:41 - dsssd 1 and 2 not biasing, 3 ok.

      - changed bias configuration -> HV core 1 & 2 now on top, braid on right

      - ribbon cables for DSSSD #1 & #2 p+n junction strips down were misaligned

      - all detectors biased succesfully

HV cable configuration (looking upstream): HV-1 Core -> top-inner (FEE 1)
                        
                                           HV-2 Core -> top-middle (FEE 5)

                                           HV-3 Core -> bottom-outer (FEE 11)

                                           HV-1 Braid -> right-inner (FEE 2)

                                           HV-2 Braid -> right-middle (FEE 6)

                                           HV-3 Braid -> left-outer (FEE 12)

Attachment 3 - rough sketch of HV cable configuration, looking upstream.