HV core (-160 V) now attached to bottom 3 FEEs (with grounded kapton)

DSSDs bias OK, leakages unchanged

No appreciable change in waveforms

Some estimates

aida01/aida05/aida09 VHF noise: 5 channels frequency = 10 MHz ?

All FEEs:
Oscillation at ~25-35 channels = 1.4 to 2 MHz ?
Varies a bit with each FEE.

No obvious issue with upper FEEs unsure why 10 MHz is present - fee09 has been replaced but showed it before as well.

Lower frequency noise on every FEE may be ground noise from e.g. switching PSU?

No more access to S4 to check thoroughly again.

AIDA Waveforms immediately after beam at 21.11.19 8am or so.

> AIDA Waveforms immediately after beam at 21.11.19 8am or so.

Update 25.11.19 with all FEE64 waveforms added

Things to note:
aida01, aida05, aida09 are all on TOP - show VHF noise, esp. aida01?
TOP is where HV is connected (-ve core)

Everything noisy in general though

21:35 (NH) - A few merger timestamp errors were appearing - Decided to powercycle all FEE64s, all checks OK.
 No Screenshot due to forgetting before I closed merger 
 FEEs were often not stopping properly and needing merger reset as well.

Beam Plan:

40Ar 300 MeV/u Primary Beam
and/or
34Si Fragment from Ar

22:45

FEES were restart near these timestamps in the TapeData, evidently
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:17 R1_361
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:36 R1_362

Run number was kept the same (Tape Server was not stopped @ FEE restart?)

All system wide checks are okay (except of course the master clock fails)
Temps look okay (attachment 1, dated 21/Nov by mistake)
Leak currents all around 0.9 uA as before (attachment 2)
Stats are behaved. We don't have beam yet (attachment 3)
Merger and tape server happy (see attachment 4)
Data rate histos while beam is still off (see attachment 5)

==Nov 21==

0:00
TD noted that the data rates have gone up at some point (about double).  
Now we increased the slow comparator thresholds from 0xa to 0xb (data rate drops from 24 Mb/s to 16 Mb/s).
We increased it again from 0xb to 0xc, but near that time the beam was also turned on, and the data rate was about 25 Mb/s.

Beam was turned on near R1_472 or a bit earlier.

TD requested waveforms, but at the moment they are not appearing.  Will enable WFs after a condition change.

Beam seen in all three DSSSDs, see attachment 6.  Beam was turned off before I could get more screencaps.

Degrader change ... near R1_481

00:32 waveforms now enabled under Run Control.  They also appear to be enabled on ASIC1 undre LED and Waveform controls.  However, we are unable to see the waveforms in the relevant histograms.

00:44 3 g/cm^2 inserted R1_488

00:45 3.2 g/cm^2 degrader in R1_489.

1:00 Several degraders and combinations are being tested since last note.

1:10 It seems 1 g/cm^2 of energy loss is not accounted for by LISE++.  Trying to understand the reason.

1:28 Satisfied with a total degrader thickness of 2.6 g/cm^2 -- seems to calculate that fragments should implant mainly in DSSSD2?  Issue seems to be resolved updating the LISE++ file.
  Near R1_507

Can see the implant samples attached as #7 and #8

1:43 According to the stats, most of the beam energy is being deposited in DSSSD1.  See #9.
  Present beam burst repetition is around 0.1 Hz (each ~10 sec)

2:00 better example of stats histos (see #10)

2:45 tuning the FRS degraders (ones for RIB separation rather than AIDA implantation depth).

3:17 Just started fragments.  Disregard previous filenames with "34Si".  Finally no degrader in the FRS was needed to get Z separation.
  R1_559
  As the DAQs are not synchronized, a quick call to the BNC PB-5 pulser shifiting from 2 -> 50 -> 2 Hz was used to make a signal burst.
  S4 degrader is 5.2 g/cm^2

3:22 Updated biases in Google Sheets.

Having trouble seeing the fragments in the AIDA histograms.  Since we just started fragments, zero the histos.

3:35 Attachment 11, with 34Si fragments as Stats.  Difficult to click Update on the Rates at the right time.  
   From the beam monitors, fragments should be at around 3000 cps.

3:45 (Up to now, some optics tuning with slits, etc)

3:46 Optics settings accepted, AIDA at R1_572, accept this condition for data runs on ~34Si fragments.

3:47 Sent a pulser burst 2 -> 50 -> 2 Hz

3:50 zero histograms again.
   Temperatures attachment 12
   Biases attachment 13
   Statistics attachment 14
   Merger / Tape Server attachment 15

3:56   
   StatHistos attachment 16

4:09 FRS starts new run #56, AIDA R1_583.  Pulser 2-> 50 -> 2 Hz

4:56 Stat histograms in attachment 17.  Not zeroed since noted at 3:50

19 November

Waiting for 40Ar beam to be delivered for tests.
Objective today is to assess degrader thickness suitable to stop 40Ar in 2nd DSSD layer.
Energy expected at 300 MeV/A, around 1E3-4 beam intensity - to be determined.

Tomorrow fragments could be expected.

DSSD / FEE64 pairing as follows

DSSD3 (most downstream): FEEs 9,10,11,12
DSSD2 (middle): FEEs 5,6,7,8
DSSD1 (most upstream): FEEs 1,2,3,4
+plastic scintillator furthest upstream

New AIDA Google spreadsheet created with leakage current, to monitor them during beamtime. First entry added. Bias attached.
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023

DESPEC platform has already been moved. No effect on rates - see attached. FEE1 is missing one channel. FEE9, 10 noisy as expected. See attached.


System wide checks *all OK* except
Master clock failed (normal)
ADC calibration fails on 6 and 12. Recalibrated via FADC Align. All OK now.

MBS transfer OK. See attached.

Good event statistics OK. See attached.

FEE64 temperatures OK. See attached.

Merger and data rate OK. See attached.

Waiting for beam.



20 November

01:07 - Still no beam. 

System wide checks OK. 
Stats OK. 
Temps OK. 
MBS OK. 
Merger and data rate OK.


02:35  - Beam is expected soon
Tried enabling writing data to disk. Merger crashed. FEE64 stopped responding to commands.
Reset Merger. FEE64 now respond to commands.
Tried enabling writing to disk again. Merger crashed once more. Reset merger.
Reset Tape Server interface. Now writing to disk.

No data currently being sent to MBS, but operators say they "are adding a new detector". May be normal.

DAQ & Merger going OK. Writing to Nov19/R1

No beam yet.

Temperatures OK
System wide checks OK
Stats OK. See attached.
MBS not working.
Merger and data rate OK. See attached.

~02:48 Beam on. 40Ar. 300 MeV/A, 500 pps. Degrader settings unclear.

03:54 Degrader settings have been changing between 4.5, 1 and 6 ug/cm2
Currently on 6 g/cm2.
AIDA seen through MBS shows no HE on detector 3, and Det 1 and 2 show no counts on positive y (FEE6 and 1). That is not the case from MIDAS.

04:30 The aim of the tests above was to have no 40Ar at all in DSSD3. It does not appear that 6 g/cm2 is sufficient, as we still see events. The reduction between 1 & 2 vs. 3 is about a factor of 2-10 depending on ADC channel. See attached.

~04:30 Moved to 6.2 g/cm2 degrader. See attached.

~04:45 Moved to 5.8 g/cm2. See attached.

~05:09 Moved to 6.4 g/cm2. See attached.

05:31 High energy channel spectra (1.8.H) for all FEE64 w/ 6.4 g/cm2 degrader - See attachments 13 & 14

05:38: Moved to 1.0 g/cm2. See attached. Difference is not huge which may indicate these spectra are misleading, as partly expected. A more detailed analysis will be carried out off line.

06:00: Beam stopped. ~R77. AIDA keeps running background over the day. Beam should restart around 11 pm later today.

10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

Double peaking normally indicates periodic baseline variations with period comparable to shaping time

Rates in all FEEs much higher than before - waveform shows very noisy in all systems. - attachment 1

Good event statistics - attachment 2

DISC in 7 and 3 above 300k

ASIC check load performed.

Good event statistics - attachment 3

Thesholds at 0xa for all FEE

Disc rate now 0 across all FEE


Pulser settings: attachment 4

pulser peak widths -   FEE         width(ch)
                              1             128
                              2             145
                              3             84
                              4             74
                              5             207
                              6             125
                              7             172
                              8             80
                              9             193
                              10            183
                              11            238
                              12            138

Attachments 5 & 6: pulser peaks for all FEE64

16.24: DAQ stop

       slow comparator threshold changed to 0x64

       ASIC check performed

       Alpha run/DAQ start (w/ data transfer enabled) - writing to file 31Oct19/R13

       Merger and TapeService ok - attachments 7 & 8

ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2

Analysis of approx 20GB of data since Thursday night (not continuous - estimate of hours incominG)

Figures 1-3: Energy Front vs Energy Back (No F-B energy gate)

Figures 4-6: 2D hit pattern

Figures 7-9: 1D hit pattern

aida10 does not seem to be reading out still, unsure of reason? Looks fine with pulser.
Slow comparator not setting or unusual gain?

Otherwise good

All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.

HDMI cables in aida09 checked and good.

> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > 
> > HDMI cables in aida09 checked and good.
> 
> The problem would be the cable , one end or the other. 
> I think ( if I recall ) a setup would restart the WR decoder.
> 
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

Cable will be replaced once a spare is available.

Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.

WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

> > > All 12 FEEs have valid WR Timestamps
> > > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > > 
> > > HDMI cables in aida09 checked and good.
> > 
> > The problem would be the cable , one end or the other. 
> > I think ( if I recall ) a setup would restart the WR decoder.
> > 
> > I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
> 
> Cable will be replaced once a spare is available.
> 
> Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
> 
> WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

An update/clarification:
Although the upper bits are zero I believe actually it is a total failure to synchronise to WR:

aida09
WR Time Item 0x80500000 0x0fbd8000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80400249 0x0fbd8000; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80500000 0x0fbdc000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80400249 0x0fbdc000; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80500000 0x0fbe0000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
WR Time Item 0x80400249 0x0fbe0000; Time (28:47)=0x249; Time (0:27)=0x0fbe0000

aida10
WR Time Item 0x8050022e 0x0bde8000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x804e29d5 0x0bde8000; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x8050022e 0x0bdec000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x804e29d5 0x0bdec000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x8050022e 0x0bdf0000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
WR Time Item 0x804e29d5 0x0bdf0000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

Attached is corrupted database and correct database

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could well cause you problems with readout as it won't understand some of the new data.

Hope this helps.

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

If ypou now check the actual values in the aida10 Options screen are the correct ?

It can happen that corruptions once loaded are propagated when Options are generally saved. 

Looking at the dates the files were modified does that make sense to you. It doesn't look as if the corrupted database was changed except when they all were ? Can you check ? Have you checked the other Options files ? I seem to recall that RIKEN AIDA had a program to to this ?

Alpha run has been running over the weekend. Stats looking positive with notable exception that aida10 is not sending much data. Will attempt a powercycle.

Temperatures and Leakage Currents very good to excellent

Noted that the slow rate of aida10 seems to be slowing down the rate data is sent to tape - perhaps being buffered until an ADC event arrives?

Update 05.11.2019

Most rates OK, aida10 only recorded 1 event over night, aida09 spectra had a number of weird channels.
Performed ASIC check/load, aida10 found another 201 events but mostly in the HEC channel (as did all FEEs)
All other FEEs look very nice however

15:44 CET: Check/load performed as aida07 rate went very high (in HEC it seems) - now back to 0

Update 07.11.2019
Run stopped to move DESPEC platform, overnight the USB relay disconnected/reconnected powering down the FEEs (probably someone knocked the interlock wire)

3x MSL type BB18(DS)-1000 installed 

detector bias -160V, leakage current c. 1uA @ +21 deg C for all 3x DSSSDs

all FEE64 good event rates (slow comparator 0xa, LEC fast comparator 0xff) c. 120k, or less, typically 30-50k, overall merge rate c. 1.6M data item/s

See https://elog.ph.ed.ac.uk/DESPEC/70



Outstanding issues

- occasional loss of bits 48-63 of WR timestamp for aida09 - can be recovered by power cycle - replace HDMI cable?

- aida09 asic 1 not producing ADC or disc data - replace aida09? 
  ASIC Check works OK
  aida09 asic temp c. 30deg C < other asic temps

- evidence of c. 1MHz extrinsic noise for most FEE64s

- acquisition of waveform data not robust - most ASIC channels do not produce data or quickly stop producing data  - PJCS to review firmware rev for Jan/Feb 2020

- require spare DSSSDs, FEE64s, HDMI cables etc

>
> 
> - evidence of c. 1MHz extrinsic noise for most FEE64s


Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom

11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding

Strange error about Database options appearing during run stop.
Does not seem to affect stopping/starting/data?

Beginning Alpha run

Thresholds set to 100 (LEC/Slow) 255 (LEC/Fast) 2 (HEC)

Pulser is OFF

Recording to MIDAS: /TapeData/31Oct19/R1

Recording to MBS: /lustre/gamma/nhubbard/AIDA/Alphas311019_

Disk rate approximately 0 as expected.

01.11.19 19:11 CET: FEE10 Stat histogram was empty - performed a check/load and now rare events come in - Merger also emitted 10 blocks to storage quickly (presumably waiting on FEE10 data)
All FEEs now slowly showing alpha data

Testing merger with 3 DSSDs connected, with thresholds of 10 (slow comparator), 2 (HEC) and 255 (fast discriminator)

Merger handling rate of 1.6 million events per second comfortably. Forwarding to MBS fine.

Network usage: 130 Mbps (AIDA network) and 20 Mbps (MBS network)