This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could well cause you problems with readout as it won't understand some of the new data.

Hope this helps.

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

Attached is corrupted database and correct database

Alpha run has been running over the weekend. Stats looking positive with notable exception that aida10 is not sending much data. Will attempt a powercycle.

Temperatures and Leakage Currents very good to excellent

Noted that the slow rate of aida10 seems to be slowing down the rate data is sent to tape - perhaps being buffered until an ADC event arrives?

Update 05.11.2019

Most rates OK, aida10 only recorded 1 event over night, aida09 spectra had a number of weird channels.
Performed ASIC check/load, aida10 found another 201 events but mostly in the HEC channel (as did all FEEs)
All other FEEs look very nice however

15:44 CET: Check/load performed as aida07 rate went very high (in HEC it seems) - now back to 0

Update 07.11.2019
Run stopped to move DESPEC platform, overnight the USB relay disconnected/reconnected powering down the FEEs (probably someone knocked the interlock wire)

> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom

>
> 
> - evidence of c. 1MHz extrinsic noise for most FEE64s


Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

3x MSL type BB18(DS)-1000 installed 

detector bias -160V, leakage current c. 1uA @ +21 deg C for all 3x DSSSDs

all FEE64 good event rates (slow comparator 0xa, LEC fast comparator 0xff) c. 120k, or less, typically 30-50k, overall merge rate c. 1.6M data item/s

See https://elog.ph.ed.ac.uk/DESPEC/70



Outstanding issues

- occasional loss of bits 48-63 of WR timestamp for aida09 - can be recovered by power cycle - replace HDMI cable?

- aida09 asic 1 not producing ADC or disc data - replace aida09? 
  ASIC Check works OK
  aida09 asic temp c. 30deg C < other asic temps

- evidence of c. 1MHz extrinsic noise for most FEE64s

- acquisition of waveform data not robust - most ASIC channels do not produce data or quickly stop producing data  - PJCS to review firmware rev for Jan/Feb 2020

- require spare DSSSDs, FEE64s, HDMI cables etc

> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > 
> > HDMI cables in aida09 checked and good.
> 
> The problem would be the cable , one end or the other. 
> I think ( if I recall ) a setup would restart the WR decoder.
> 
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

Cable will be replaced once a spare is available.

Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.

WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding

Strange error about Database options appearing during run stop.
Does not seem to affect stopping/starting/data?

> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

Beginning Alpha run

Thresholds set to 100 (LEC/Slow) 255 (LEC/Fast) 2 (HEC)

Pulser is OFF

Recording to MIDAS: /TapeData/31Oct19/R1

Recording to MBS: /lustre/gamma/nhubbard/AIDA/Alphas311019_

Disk rate approximately 0 as expected.

01.11.19 19:11 CET: FEE10 Stat histogram was empty - performed a check/load and now rare events come in - Merger also emitted 10 blocks to storage quickly (presumably waiting on FEE10 data)
All FEEs now slowly showing alpha data

Testing merger with 3 DSSDs connected, with thresholds of 10 (slow comparator), 2 (HEC) and 255 (fast discriminator)

Merger handling rate of 1.6 million events per second comfortably. Forwarding to MBS fine.

Network usage: 130 Mbps (AIDA network) and 20 Mbps (MBS network)

All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.

HDMI cables in aida09 checked and good.

1) Login to Raspberry Pi 

ssh pi@nnrpi2

2) Kill any instances of the TclHttpd server

ps -ef | grep TclHttpd
kill -9 <pid>

3) Start server

/MIDAS/TclHttpd/linux-arm/TclHttpd-server &

10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
      Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%

10.15 DSSSD stack as follows

      DSSSD  serial  thickness  depletion 
      #              (um)       voltage (V)
      1      3208-7  1019       125              upstream
      2      3208-13 1020       120
      3      3208-14 1020       130              downstream

11.30 AIDA power relays sequenced on
      
      DAQ start

      system wide checks ok

11.33 attachment 2 - fee temperatures - ok

      attachment 3 - good event statistics with slow comparator threshold = 100

      attachment 4 - good event statistics with slow comparator threshold = 50

      attachment 5 - good event statistics with slow comparator threshold = 20

      attachment 6 - good event statistics with slow comparator threshold = 10
     
13.51 attachment 7 - Rates
                   - aida09 1 ASIC not working (no data)

      attachment 8 - good event statistics

      attachments 9/10 - 1.8.L spectra
                       - no peak observed in aida09

      attachments 11/12 - 1.8.W spectra
                        - waveform only observed in naida 2,4,7,8,12

      pulser peak widths -   FEE         width(ch)
                              1             71
                              2             96
                              3             66
                              4             79
                              5             74
                              6             96
                              7             89
                              8             83
                              9           no peak
                              10            82
                              11            153
                              12            120

17.26 FATIMA moved upstream around AIDA

      attachment 13 - good event statistics

      ASIC check-load performed

      attachment 14 - good event statistics following checkload

12.51 - detector bias and leakage currents - detector 3 has no current
                                           - other detectors ok

13.13 - tried changing from HV3 cables to HV4 - still open circuit 

      - tried different permutations of moving HV-4 braid and core cables (high voltage to ground connections)

       (dssd3)

      (all directions defined as if looking upstream)

      - core -> bottom, braid -> left     open - circuit

      - core -> top, braid -> right       trips at 8V

      - core -> bottom, braid -> right     open - circuit

      - core -> top, braid-> left          trips at 8V

      - removed jumpers - still trips

13.36 - testing continuity of bond wires on dsssd 3208-14 & 3208-15

      - observe 20 ohm resistance on p+p and p+n bond wires 

19:41 - dsssd 1 and 2 not biasing, 3 ok.

      - changed bias configuration -> HV core 1 & 2 now on top, braid on right

      - ribbon cables for DSSSD #1 & #2 p+n junction strips down were misaligned

      - all detectors biased succesfully

HV cable configuration (looking upstream): HV-1 Core -> top-inner (FEE 1)
                        
                                           HV-2 Core -> top-middle (FEE 5)

                                           HV-3 Core -> bottom-outer (FEE 11)

                                           HV-1 Braid -> right-inner (FEE 2)

                                           HV-2 Braid -> right-middle (FEE 6)

                                           HV-3 Braid -> left-outer (FEE 12)

Attachment 3 - rough sketch of HV cable configuration, looking upstream.
       

Ribbon cables fitted with kapton PCB couplers -> 3 isolated + 1 non-isolated for each cable length/DSSSD

Ribbon cable lengths:

without kapton PCB coupler (cm) - 40, 47, 51.5

with kapton PCB coupler (cm) - 50.5, 57.5, 62

gap between plastic and dsssd - 15 mm

length between top of snout and dsssd - 5cm