18.30 AIDA powered up for brief test

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
       ASIC settings at power up - slow comparator 0x64

      per p+n FEE64 1.8.L spectra - attachment 4
       pulser peak width aida9 55 ch FWHM (39keV FWHm), aida15 53 ch FWHM (37keV FWHM)

      p+n FEE64 aida15 1.8.W spectra - 20us FSR - attachments 5-6

      saved *.L, *.H, *.W spectra for aida15 - attachments 7-11
       *.W spectra - 20us, 200us, 2ms & 20ms FSR 


19.27 DAQ STOP
      DSSSD bias OFF
      FEE64 power OFF

Start 00:43 22.2.25 https://elog.gsi.de/despec/G-24-00302/57
AIDA data file R9_85

Converting hexadecimal timestamp to decimal: 1740181266259754800
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Friday, February 21, 2025 11:41:06.259 PM
Your time zone: Friday, February 21, 2025 11:41:06.259 PM GMT+00:00


End 12:56 22.2.25 https://elog.gsi.de/despec/G-24-00302/85 (start of next degrader setting)
AIDA data file R9_199

Converting hexadecimal timestamp to decimal: 1740224934731157800
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Saturday, February 22, 2025 11:48:54.731 AM
Your time zone: Saturday, February 22, 2025 11:48:54.731 AM GMT+00:00

Analysis data files FEB25/R9_85 and FEB25/R9_199
 - no timewarps
 - deadtime aida05 c. 37% and 39% due to high SC41 scaler input to MACB - issue subsequently fixed 
 - deadtime aida04 c. 4%, all other FEE64s <<1%




FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4

n+n Ohmic FEE64s 2, 4

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (507 of 512 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/- 168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   151keV < LEC energy < 1000keV
    to select candidate beta events
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0
   (x,y) strips corresponding to maximum energy
   p+n junction and n+n Ohmic side HEC 

- HEC veto 
   not available - only 1x AIDA DSSSD installed

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 and overall event length < 33us





per FEE64 LEC ADC data items 268ms/channel - attachments 3-5
- all
- 150keV < energy < 1500keV
- energy > 1500keV

absence of high instantaneous rates on spill cf. S181, S100, S505 etc
spill structure visible for energies > 1500keV - probably to be expected as majority of such events should be light ions

per FEE64 HEC ADC data items 268ms/channel - attachments 6-8

- all
- 100MeV < energy < 1000MeV
- energy > 1000MeV

Implant & decay event rates 262us/channel - attachment 9

LEC m_p versus m_n - attachment 10

LEC e_p versus e_n - 20keV/channel - attachment 11

HEC x strip versus y strip - attachment 12

- HEC-LEC implant decay time difference <1s
- HEC-LEC implant decay time difference <100s

shows x-y window used to select 82Nb events

HEC m_p versus m_n - attachment 13

HEC e_p versus e_n - 20MeV/channel - attachment 14


HEC-LEC implant-decay time difference (4ms/channel) versus - attachment 15
- HEC energy 
- LEC energy
- x & y strip 

x strips 130 & 136 disabled

HEC & LEC p strip - n strip time difference (2us/channel, offset=2000 channels) - attachment 16

HEC E (20MeV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 17

LEC E (20keV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 18

LEC e_p - e_n (5.6keV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 19

HEC x & y strip versus implant-decay time difference (4.194ms/channel) - attachment 20

HEC-LEC implant-decay time difference (8.389ms/channel) for candidate 82Nb events - attachment 21
- blue - forward
- cyan - backward

Note
- no FRS PID
- no HEC dE
- no HEC veto
- no HEC energy gates - assume all events within x-y window *stop* in AIDA DSSSD - 82Nb events known to overlap in x-y with (longer-lived, more numerous) 83Nb events
- no front-back time difference
- no clustering
- *all* LEC channels with valid ADC offsets (507 of 512) included *except* 2x strips 130 & 136

HEC-LEC implant-decay time difference (8.389ms/channel) for candidate 82Nb events - attachment 22

Weighted least squares fit channels 1-12 half life 55(9)ms cf. NNDC 50.0(3)ms.

WR timestamp errors resolved after reseating the HDMI cables to the MACB

Analysis data files R4_351-396


Data file R4_351 first WR ts 0x17D7B4C72B0BA9C8
Converting hexadecimal timestamp to decimal: 1718040550378809900
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Monday, June 10, 2024 5:29:10.378 PM
Your time zone: Monday, June 10, 2024 6:29:10.378 PM GMT+01:00 DST
Relative: 8 months ago

Data file R4_396 first WR ts 0x17D7BE86408888E8
Converting hexadecimal timestamp to decimal: 1718051266682718500
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Monday, June 10, 2024 8:27:46.682 PM
Your time zone: Monday, June 10, 2024 9:27:46.682 PM GMT+01:00 DST
Relative: 8 months ago

Attachments 1-2 - analysis data files R4_351 and R4_396
 max. *time averaged* deadtime FEE64 #7 (aida08) 1.7% and 1.7% respectively
 all other FEE64 deadtimes < 1%





FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4
DSSSD#2 11  7 16 10 14 13  6  8

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (1012 of 1024 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/- 11200keV (wide open for first pass analysis)

- HEC energy difference +/- 1.68GeV

- valid LEC events

   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1

   LEC energy > 151keV
    to select candidate beta  and alpha events - will include light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 and overall event length < 33us




Number of events observed

 *** scaler # 1 count:  35941696 DSSSD#1 decay events       LEC m_p = 1 and LEC m_n = 1, ADC data > 151.2keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler # 2 count:  31960753 DSSSD#2 decay events       LEC m_p = 1 and LEC m_n = 1, ADC data > 151.2keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler # 3 count:   1499844 DSSSD#1 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler # 4 count:   1297356 DSSSD#2 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler # 5 count:   5446969 DSSSD#1 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler # 6 count:  75030747 DSSSD#2 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8

DSSSD#1 implant events
x=m_p=0 ?
y=m_n=0 ?

DSSSD#2 implant events
x=m_p=0 ?
y=m_n=0 ?


Attachments 3-6 - per FEE64 LEC data item rates 268ms/channel - common x and y scales
 - no conditions
 - 150keV < energy < 1500keV
 - energy > 1500keV

Some deadtime observed on spill, none observed off spill. No significant variation in per FEE64 LEC data rates.

Attachments 7-8 - per FEE64 LEC hit patterns
 - 150keV < energy < 1500keV
 - energy > 1500keV

Aside from usual hot channels at cable/DSSSD boundaries good hit pattern observed. For > 1500keV events observe flat field in x-plane, focussed in y-plane.

Attachments 9-11 - per FEE64 HEC data item rates 268ms/channel - common x and y scales
 - no conditions
 - 100MeV < energy < 1000MeV
 - energy > 1000MeV

Note hot HEC channel in aida08 - should be disabled.

Attachment 12 - per DSSSD decay and implant rates 262us/channel - common x and y scales

Attachment 13 - per DSSSD LEC m_p versus m_n 
 - no conditions

Attachment 14 - per DSSSD LEC p strip versus n strip
 - no conditions

Attachments 15-16 - per DSSSD LEC E_p versus E_n - x and y-axes 20keV/channel
 - LEC energy difference +/- 2000 channels (+/- 11200keV)

Attachments 17-18 - per DSSSD p strip versus n strip
 - HEC-LEC time difference <1s
 - HEC-LEC time difference <100s

Attachment 19 -  per DSSSD HEC m_p versus m_n

Attachment 20 - per DSSSD HEC p strip versus n strip
 - no conditions
 - z_hec=1 => implant stops in DSSSD#1
 - z_hec=3 => implant hits DSSSD#1 and DSSSD#2 but does not necessarily stop in DSSSD#2

Attachment 21 - per DSSSD HEC E_p versus E_n - x and y axes 20MeV/channel

Attachment 22 - DSSSD#1 HEC E_p versus DSSSD#2 HEC E_p - x and y axes 20MeV/channel
 - few events stop in DSSSD#2
 - most events lower Z and A - fission fragments?

Attachment 23 - per DSSSD per pixel HEC-LEC time difference 4.096us/channel
 - events observed to <<100us

Attachment 24 - per DSSSD per pixel HEC-LEC time difference (1s/channel) versus LEC energy (20keV/channel)
 - alpha events long lived 
 - some evidence of effect of on spill deadtime in implant-decay correlations (cf. S100 and S505)

Attachment 25 - per DSSSD implant and decay event p strip - n strip time difference (2us/channel)
 - wider distribution for implant events as expected due to number of ASIC active channels in implant events
 - most decay events +/-2us - lower ASIC occupancy for decay events so most events will be from same (or adjacent) clock cycle

18.18 bPlas and BB7 installs complete 
      Restart AIDA DAQ, Merger, Tape Server and re-test

      per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 69 ch FWHM = 42keV FWHM

      per FEE64 Rate Spectra - attachment 2
       BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
       BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK

      per FEE64 1.8.W spectra - 20us FSR - attachments 3-5

      ADC data item stats - attachment 6
       aida01  aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
       BB7 aida10 19k

      WR timestamps OK - attachment 7

      Merger, TapeSever etc - attachments 8-9
       disk directpry /TapeData/FEB25
       working OK
     
      DSSSD bias & leakage current OK - attachment 10

      System wide checks OK - attachment 11-15
       note global clock status 6 errors reported earlier today have now gone

      FEE64 temperatures OK - attachment 16

18.38 Transition to safe state

      DAQ STOP
      disable data transfer 1
      detector bias OFF
      FEE64 power OFF


      Can restart as follows

1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
     sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps

If all OK can re-connect to Merger/TapeServer as follows

1) DAQ STOP
2) enable data transfer 1
3) DAQ GO


Can disconnect from Merger/TapeServer as follows

1) DAQ STOP
2) disable data transfer 1
   

AIDA analysis and presentation by J. Bormans for AIDA made in the HISPEC-DESPEC collaboration meeting.

https://docs.google.com/presentation/d/1hlZ30r294UqbVKGy3jg-wompFpc7TSfD/edit?usp=sharing&ouid=102131181856760114019&rtpof=true&sd=true

Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000

N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.

n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted

BB7 aida10 (asics #1-2 p+n, #3-4 n+n)

BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream) 

10.30 FEE64 power ON

      DSSSD bias ON (BB18 only) - attachment 1
       leakage current OK

      ASIC settings 2024Dec13-17.02.45 restored
       p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf

      Attachments 2-7 WR timestamp aida05 & aida12, system wide check
       aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
       WR decoder status aida02

      per FEE64 Rate spectra - attachment 8
       aida10 high rate - BB7 not biased
       rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
       BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments

      ADC data item stats - attachment 9
       rates generally OK
       aida10 high - BB7 not biased
       aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
       all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!

      BNC PB-5 settings - attachment 10

      WR timestamps OK - attachment 11

      FEE64 temperatures OK - attachment 12

12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet

      ADC data item stats - attachment 13
       all rates higher cf. attachment 9

      per FEE64 Rate spectra - attachment 14

      per p+n FEE64 1.8.L spectra - attachment 15
       aida09 pulser peak width 69 ch FWHM = 49keV FWHM 
       aida14 pulser peak width 50 ch FWHM = 35keV FWHM

      per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17

      per n+n FEE64 1.8W spectra - 20us FSR - attachment 18

      DSSSD bias & leakage current OK - attachment 19
       ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%

      FEE64 temperatures OK - attachment 20

      DSSSD bias volatge & leakage current OK - attachment 21
       CAEN N1419ET ch#0 BB18, ch#1 BB7

      Install of bPlas drivers, cabling, grounds complete

      ADC data item stats - attachment 22
       
      per FEE64 Rate spectra - attachment 23
       all BB18 p+n FEE64s *except* aida05 show very low rates of noise

      per p+n FEE64 1.8.L spectra - attachment 24
       aida09 pulser peak width 62 ch FWHM = 42keV FWHM
       BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26

Aanlysis data files R3_9 - R3_128 (207Hg setting)

Data file R3_9 first WR ts 0x16FB110A37376D88
GMT: Wednesday, June 22, 2022 10:13:43.562 PM
Your time zone: Wednesday, June 22, 2022 11:13:43.562 PM GMT+01:00 DST

Data file R3_69 first WR ts 0x16FB2286FAD71896
GMT: Thursday, June 23, 2022 3:34:11.118 AM
Your time zone: Thursday, June 23, 2022 4:34:11.118 AM GMT+01:00 DST

Data file R3_128 first WR ts 0x16FB35DF179BD866
GMT: Thursday, June 23, 2022 9:28:40.278 AM
Your time zone: Thursday, June 23, 2022 10:28:40.278 AM GMT+01:00 DST

Attachments 1-3 - analysis data files R3_9, R_69, and R3_128
 max. *time averaged* deadtime FEE64 #1 (aida02) 15.7%, 8.6% and 15.6% respectively
 all other FEE64 deadtimes < 2%


 *** scaler( 1):            120442206  DSSSD#1 decay events       LEC m_p = 1 and LEC m_n = 1, 151.2keV < ADC data < 1008keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler( 2):             51126024  DSSSD#2 decay events       LEC m_p = 1 and LEC m_n = 1, 151.2keV < ADC data < 1008keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler( 3):              5630250  DSSSD#1 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler( 4):              4983984  DSSSD#2 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler( 5):             42439339  DSSSD#1 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler( 6):             22316289  DSSSD#2 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler( 7):                    0
 *** scaler( 8):                    0
 *** scaler( 9):                    0
 *** scaler(10):           2464294601  FEE64 #1 LEC data items
 *** scaler(11):           7641292106
 *** scaler(12):           3547670559
 *** scaler(13):           6144404472
 *** scaler(14):            637262222
 *** scaler(15):           2302716728
 *** scaler(16):           4511293677
 *** scaler(17):           3857002960  FEE64 #8 LEC data items
 *** scaler(18):                    0
 *** scaler(19):                    0
 *** scaler(20):              7697923  FEE64 #1 HEC data items
 *** scaler(21):              7169894
 *** scaler(22):             10028407
 *** scaler(23):              3045622
 *** scaler(24):              4246633
 *** scaler(25):              5564911
 *** scaler(26):              7196158
 *** scaler(27):              1960591  FEE64 #8 HEC data items
 *** scaler(28):                    0
 *** scaler(29):                    0
 *** scaler(30):                    0
 *** scaler(31):                    0
 *** scaler(32):                    0

DSSSD#1 implant events
x=m_p=0 1334587
y=m_n=0 4860983

DSSSD#2 implant events
x=m_p=0 1159736
y=m_n=0 2757735


Attachment 4 - per DSSSD LEC E_p versus E_n - x and y-axes 20keV/channel
 - LEC energy difference +/- 2000 channels (+/- 11200keV)
 - Do not observe the off-axis energy correlations observed in the postrun background/calibration runs
 - Do observe expected inter strip charge sharing and charge sharing at DSSSD active area boundary

All subsequent spectra LEC energy difference +/-20 channels (+/- 112keV)

Attachments 5-7 - per FEE64 LEC data item rates (Hz) 268ms/channel - common x and y scales
 - no conditions
 - 150keV < energy < 1500keV
 - energy > 1500keV
Note structure in > 1500keV data not restricted to on spill

Attachments 8-10 - per FEE64 HEC data item rates (Hz) 268ms/channel - common x and y scales
 - no conditions
 - 100MeV < energy < 1000MeV
 - energy > 1000MeV

Attachment 11 per DSSSD decay and implant rates (Hz) 268ms/channel =- common x and y scales
 - significant on spill deadtime for DSSSD#1

Attachment 12 - per DSSSD LEC m_p versus m_n 
 - no conditions
 - z_hec=1 (DSSSD#1) and z_hec=3 (DSSSD#2)

Attachment 13 - per DSSSD LEC p strip versus n strip

Attachment 14 - per DSSSD LEC E_p versus E_n - x and y axes 20keV/channel

Attachment 15 - per DSSSD per pixel time difference between successive LEC events (4.096us/channel)
 - minimum time difference c. 4us

Attachment 16 -  per DSSSD HEC m_p versus m_n
 - many more m=0 observed cf. RIKEN data e.g. https://elog.ph.ed.ac.uk/AIDA/583
 - more low energy HEC events?

Attachment 17 - per DSSSD HEC p strip versus n strip
 - no conditions
 - z_hec=1 (DSSSD#1) and z_hec=3 (DSSSD#2)
 - z_hec=1 => implant stops in DSSSD#1
 - z_hec=3 => implant hits DSSSD#1 and DSSSD#2 but does not necessarily stop in DSSSD#2

Attachment 18 - per DSSSD HEC E_p versus E_n - x and y axes 20MeV/channel

Attachment 19 - DSSSD#1 HEC E_p versus DSSSD#2 E_p - x and y axes 20MeV/channel

Attachment 20 - DSSSD#1 LEC E_p versus DSSSD#2 E_p - x and y axes 20keV/channel
 - y=0 12816818
 - x=0 7374435
 - x>0 and y>0 2295693
 
Attachment 21 - per DSSSD per pixel time difference between successive HEC events (4.096us/channel)
 - minimum time difference c. 36us which make sense - c. readout time of ASIC with all 16 channels active
 - origin of 9 channel period spectrum structure? 

Attachments 22-23 - per DSSSD per pixel time difference between HEC and LEC events (4.096us/channel)
Attachments 24 - per DSSSD per pixel time difference between HEC and LEC events (1s/channel)
 - for ions stopped in DSSSD#1 or DSSSD'2 - no PID selection
 - minimum time difference c. 88us
 - effect of high on spill deadtime apparent per S100 analysis

Attachment 25 - per DSSSD per pixel time difference between HEC and LEC events (1s/channel) versus
 - HEC energy
 - LEC energy
 - p and n strip

Attachment 26 - per DSSSD implant and decay event p strip - n strip time difference (2us/channel)
 - wider distribution for implant events expected due to number of ASIC active channels in implant events
 - most decay events +/-2us - lower ASIC occupancy for decay events so most events will be from same (or adjacent) clock cycle

Attachment 27 - per DSSSD decay event p strip - n strip time difference (2us/channel) versus decay (E_p - E_n) (20keV channel) 

First timestamp of R2_105 0x16FAEC5E54AABF6C

GMT: Wednesday, June 22, 2022 11:01:42.903 AM
Your time zone: Wednesday, June 22, 2022 12:01:42.903 PM GMT+01:00 DST


First timestamp of R3_150 0x16FB3CE72ED53CB6

GMT: Thursday, June 23, 2022 11:37:31.609 AM
Your time zone: Thursday, June 23, 2022 12:37:31.609 PM GMT+01:00 DST


First timestamp of R5_35 0x16FB89F33F9B44E2

GMT: Friday, June 24, 2022 11:09:25.825 AM
Your time zone: Friday, June 24, 2022 12:09:25.825 PM GMT+01:00 DST


First timestamp of R5_299 0x16FBD8CC5DCBE912

GMT: Saturday, June 25, 2022 11:14:20.247 AM
Your time zone: Saturday, June 25, 2022 12:14:20.247 PM GMT+01:00 DST


First timestamp of R5_546 0x16FC220EA8ADB132

GMT: Sunday, June 26, 2022 9:36:49.320 AM
Your time zone: Sunday, June 26, 2022 10:36:49.320 AM GMT+01:00 DST

Attachments 1-5 - analysis of data files R2_105, R3_150, R5_35, R5_299, and R5_536 (c. one data file per day of S505)
 FEE64 #1 (aida02) *time averaged* deadtime c. 12-18%, deadtime of all other FEE64s much lower

Attachments 6-10 - per DSSSD LEC p+n junction versus n+n Ohmic energy - X & y-axes 20keV/channel

No evidence of off-axis energy correlation observed in the postrun background/calibration runs - see https://elog.ph.ed.ac.uk/DESPEC/670

Offline analysis of S505 data files R3_150

first WR ts
First timestamp of R3_150 0x16FB3CE72ED53CB6

Epoch converter says ...

GMT: Thursday, June 23, 2022 11:37:31.609 AM
Your time zone: Thursday, June 23, 2022 12:37:31.609 PM GMT+01:00 DST

last WR ts
First timestamp of R3_151 0x16FB3D3042B9E8E6

Epoch converter says ...

GMT: Thursday, June 23, 2022 11:42:45.475 AM
Your time zone: Thursday, June 23, 2022 12:42:45.475 PM GMT+01:00 DST

Analysis of data file R3_150 - attachment 1
 max time averaged deadtime FEE64 #1 (aida02) 18.3%



FEE64 configuration

FEE64   a  
      b    d
        c

         a  b  c  d 
DSSSD#1  3  4  1  2
DSSSD#2  7  8  5  6

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 100keV

- LEC energy difference +/-168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   DSSSD #1
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   DSSSD #2
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1

   151keV < LEC energy < 1008keV
    to select candidate beta events and veto higher energy events e.g. light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 *or* difference in first and last WR timestamp of event < 33us


Attachment 2 per FEE64 LEC data rate (Hz) 268ms/channel
Attachment 3 per FEE64 LEC data rate (Hz) 268ms/channel: 150keV < energy < 1500keV
Attachment 22 per FEE64 LEC data rate (Hz) 268ms/channel: energy > 1500keV

- observe high instantaneous rate on spill
- rate dominated by low energy (<1500keV) events
- rate of higher energy events dominated by on spill events i.e. light ions as expected
- significant deadtime on spill for n+n Omic FEE64 aida02, low deadtime off spill
- deadtime low/zero for all other FEE64s on/off spill

Attachment 4 per FEE64 HEC data rate (Hz) 268ms/channel
Attachment 5 per FEE64 HEC data rate (Hz) 268ms/channel: 100MeV < energy < 1000MeV
Attachment 6 per FEE64 HEC data rate (Hz) 268ms/channel: energy > 1000MeV

- all HEC events on spill as expected
- significant deadtime on spill for n+n Ohmic FEE64 aida02, low deadtime off spill
- deadtime low/zero for all other FEE64s on/off spill


Attachment 7 per DSSSD decay and implant rate (Hz) 268ms/channel

Attachment 8 log scale
yellow - FEE64 aida01 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida01 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #1 decay rate (Hz) 268ms/channel
green  - DSSSD #1 implant rate (Hz) 268ms/channel

Attachment 9 log scale
yellow - FEE64 aida02 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida02 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #1 decay rate (Hz) 268ms/channel
green  - DSSSD #1 implant rate (Hz) 268ms/channel

Attachment 10 log scale
yellow - FEE64 aida05 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida05 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #2 decay rate (Hz) 268ms/channel
green  - DSSSD #2 implant rate (Hz) 268ms/channel


Attachment 11 per DSSSD LEC m_p versus m_n 

Attachment 12 per DSSSD LEC x versus y 

Attachment 13 per DSSSD LEC p+n junction versus n+n Ohmic energy - x-axis & y-axis 20keV/channel


Attachment 14 per DSSSD HEC m_p versus m_n 

Attachment 15 per DSSSD HEC x versus y 

Attachment 16 per DSSSD HEC p+n junction versus n+n Ohmic energy - x-axis & y-axis 20MeV/channel

Attachment 17 HEC DSSSD#2 p+n junction versus DSSSD#1 p+n junction energy - x-axis & y-axis 20MeV/channel
 to identify which ions stop in DSSSD#2 (z_loc=2)

Attachment 18 decay & implant dx versus dy 

Attachment 19 LEC DSSSD#2 p+n junction versus DSSSD#1 p+n junction energy - x-axis & y-axis 20keV/channel

Attachment 20 per DSSSD decay & implant dt (2us/channel)

Attachment 21 per DSSSD decay dt (2us/channel) versus p+n junctionj - n+n Ohmic energy difference (20keV/channel)


Summary

- high instantaneous data rates on spill
- significant deadtime FEE64 #2 on spill
   magnitude, structure, position of DSSSD#1 decay rate variations differs from FEE64 #2 deadtime - events merging?
- all other FEE64s OK on/off spill

aida06 crashed and rebooted sometime ( 14:48 - 23:25 ) yesterday. System console log appended.


10:01:25/14:58:29|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
10:01:25/14:58:29|------------[ cut here ]------------
10:01:25/23:25:49|kernel BUG at mm/slab.c:2974!
10:01:25/23:25:49|Oops: Exception in kernel mode, sig: 5 [#1]
10:01:25/23:25:50|PREEMPT Xilinx Virtex440
10:01:25/23:25:50|Modules linked in: aidamem xdriver xh_spidev_register
10:01:25/23:25:50|NIP: c009211c LR: c00920b0 CTR: 00000006
10:01:25/23:25:50|REGS: c632dc60 TRAP: 0700   Not tainted  (2.6.31)
10:01:25/23:25:50|MSR: 00021000 <ME,CE>  CR: 22022048  XER: 00000000
10:01:25/23:25:50|TASK = c62064c0[375] 'AidaExecV10' THREAD: c632c000
10:01:25/23:25:50|GPR00: 00000001 c632dd10 c62064c0 c680daf0 c694003c 0000000b c6940020 0000000a 
10:01:25/23:25:50|GPR08: 0000001b c680dae0 00000d80 c680dae0 22008022 10054d4c c03a0000 00000020 
10:01:25/23:25:50|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 
10:01:25/23:25:50|GPR24: 00100100 c632c000 00000000 c680dae8 c680dae0 c680ae00 00000005 c680e400 
10:01:25/23:25:50|NIP [c009211c] cache_alloc_refill+0x130/0x608
10:01:25/23:25:50|LR [c00920b0] cache_alloc_refill+0xc4/0x608
10:01:25/23:25:50|Call Trace:
10:01:25/23:25:50|[c632dd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
10:01:25/23:25:50|[c632dd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
10:01:25/23:25:50|[c632dd90] [c0042420] __sigqueue_alloc+0x50/0xb8
10:01:25/23:25:50|[c632ddb0] [c0042938] __send_signal+0x78/0x260
10:01:25/23:25:50|[c632dde0] [c0042f78] group_send_sig_info+0x70/0x9c
10:01:25/23:25:51|[c632de10] [c00438a8] kill_pid_info+0x48/0x8c
10:01:25/23:25:51|[c632de30] [c0038e8c] it_real_fn+0x1c/0x30
10:01:25/23:25:51|[c632de40] [c0050c40] hrtimer_run_queues+0x184/0x240
10:01:25/23:25:51|[c632dea0] [c0040ba8] run_local_timers+0x10/0x2c
10:01:25/23:25:51|[c632deb0] [c0040bf4] update_process_times+0x30/0x70
10:01:25/23:25:51|[c632ded0] [c005a000] tick_periodic+0x34/0xe8
10:01:25/23:25:51|[c632dee0] [c005a0d4] tick_handle_periodic+0x20/0x120
10:01:25/23:25:51|[c632df20] [c000af70] timer_interrupt+0xa4/0x10c
10:01:25/23:25:51|[c632df40] [c000e9c4] ret_from_except+0x0/0x18
10:01:25/23:25:51|Instruction dump:
10:01:25/23:25:51|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 
10:01:25/23:25:51|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 
10:01:25/23:25:51|Kernel panic - not syncing: Fatal exception in interrupt
10:01:25/23:25:51|Call Trace:
10:01:25/23:25:51|[c632dab0] [c0005de8] show_stack+0x44/0x16c (unreliable)
10:01:25/23:25:51|[c632daf0] [c00345bc] panic+0x94/0x168
10:01:25/23:25:51|[c632db40] [c000bd44] die+0x178/0x18c
10:01:25/23:25:51|[c632db60] [c000c000] _exception+0x164/0x1b4
10:01:25/23:25:51|[c632dc50] [c000e978] ret_from_except_full+0x0/0x4c
10:01:25/23:25:51|[c632dd10] [c00920b0] cache_alloc_refill+0xc4/0x608
10:01:25/23:25:52|[c632dd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
10:01:25/23:25:52|[c632dd90] [c0042420] __sigqueue_alloc+0x50/0xb8
10:01:25/23:25:52|[c632ddb0] [c0042938] __send_signal+0x78/0x260
10:01:25/23:25:52|[c632dde0] [c0042f78] group_send_sig_info+0x70/0x9c
10:01:25/23:25:52|[c632de10] [c00438a8] kill_pid_info+0x48/0x8c
10:01:25/23:25:52|[c632de30] [c0038e8c] it_real_fn+0x1c/0x30
10:01:25/23:25:52|[c632de40] [c0050c40] hrtimer_run_queues+0x184/0x240
10:01:25/23:25:52|[c632dea0] [c0040ba8] run_local_timers+0x10/0x2c
10:01:25/23:25:52|[c632deb0] [c0040bf4] update_process_times+0x30/0x70
10:01:25/23:25:52|[c632ded0] [c005a000] tick_periodic+0x34/0xe8
10:01:25/23:25:52|[c632dee0] [c005a0d4] tick_handle_periodic+0x20/0x120
10:01:25/23:25:52|[c632df20] [c000af70] timer_interrupt+0xa4/0x10c
10:01:25/23:25:52|[c632df40] [c000e9c4] ret_from_except+0x0/0x18
10:01:25/23:25:52|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:28:52|
Finished copying zImage to RAM 
10:01:25/23:28:53|
Found 0 errors checking kernel image
10:01:25/23:28:54|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
10:01:25/23:28:54|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
10:01:25/23:28:54|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
10:01:25/23:28:54|SPIconfig done now to set up the LMK3200 registers
10:01:25/23:28:54|LMK #0 : regInit[0]=0x80000000
10:01:25/23:28:54|LMK #0 : regInit[1]=0x10070600
10:01:25/23:28:54|LMK #0 : regInit[2]=0x60601
10:01:25/23:28:54|LMK #0 : regInit[3]=0x60602
10:01:25/23:28:55|LMK #0 : regInit[4]=0x60603
10:01:25/23:28:55|LMK #0 : regInit[5]=0x70624
10:01:25/23:28:55|LMK #0 : regInit[6]=0x70605
10:01:25/23:28:55|LMK #0 : regInit[7]=0x70606
10:01:25/23:28:55|LMK #0 : regInit[8]=0x70627
10:01:25/23:28:55|LMK #0 : regInit[9]=0x10000908
10:01:25/23:28:55|LMK #0 : regInit[10]=0xA0022A09
10:01:25/23:28:55|LMK #0 : regInit[11]=0x82800B
10:01:25/23:28:55|LMK #0 : regInit[12]=0x28C800D
10:01:25/23:28:55|LMK #0 : regInit[13]=0x830020E
10:01:25/23:28:55|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
10:01:25/23:28:55|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
10:01:25/23:28:55|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
10:01:25/23:28:55|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
10:01:25/23:28:56|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
10:01:25/23:28:59|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
10:01:25/23:29:05|Using Xilinx Virtex440 machine description
10:01:25/23:29:06|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
10:01:25/23:29:06|Zone PFN ranges:
10:01:25/23:29:06|  DMA      0x00000000 -> 0x00007000
10:01:25/23:29:06|  Normal   0x00007000 -> 0x00007000
10:01:25/23:29:06|Movable zone start PFN for each node
10:01:25/23:29:06|early_node_map[1] active PFN ranges
10:01:25/23:29:06|    0: 0x00000000 -> 0x00007000
10:01:25/23:29:06|MMU: Allocated 1088 bytes of context maps for 255 contexts
10:01:25/23:29:06|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
10:01:25/23:29:06|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
10:01:25/23:29:06|PID hash table entries: 512 (order: 9, 2048 bytes)
10:01:25/23:29:07|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
10:01:25/23:29:07|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
10:01:25/23:29:07|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
10:01:25/23:29:07|Kernel virtual memory layout:
10:01:25/23:29:07|  * 0xffffe000..0xfffff000  : fixmap
10:01:25/23:29:07|  * 0xfde00000..0xfe000000  : consistent mem
10:01:25/23:29:07|  * 0xfde00000..0xfde00000  : early ioremap
10:01:25/23:29:07|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
10:01:25/23:29:07|NR_IRQS:512
10:01:25/23:29:07|clocksource: timebase mult[a00000] shift[22] registered
10:01:25/23:29:07|Console: colour dummy device 80x25
10:01:25/23:29:07|Mount-cache hash table entries: 512
10:01:25/23:29:07|NET: Registered protocol family 16
10:01:25/23:29:07|PCI: Probing PCI hardware
10:01:25/23:29:07|bio: create slab <bio-0> at 0
10:01:25/23:29:07|NET: Registered protocol family 2
10:01:25/23:29:07|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
10:01:25/23:29:07|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
10:01:25/23:29:07|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
10:01:25/23:29:07|TCP: Hash tables configured (established 4096 bind 4096)
10:01:25/23:29:08|TCP reno registered
10:01:25/23:29:08|NET: Registered protocol family 1
10:01:25/23:29:08|ROMFS MTD (C) 2007 Red Hat, Inc.
10:01:25/23:29:08|msgmni has been set to 214
10:01:25/23:29:08|io scheduler noop registered
10:01:25/23:29:08|io scheduler anticipatory registered
10:01:25/23:29:08|io scheduler deadline registered
10:01:25/23:29:08|io scheduler cfq registered (default)
10:01:25/23:29:08|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
10:01:25/23:29:08|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
10:01:25/23:29:08|console [ttyS0] enabled
10:01:25/23:29:08|brd: module loaded
10:01:25/23:29:08|loop: module loaded
10:01:25/23:29:08|Device Tree Probing 'ethernet'
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
10:01:25/23:29:08|XLlTemac: DCR address: 0x80
10:01:25/23:29:08|XLlTemac: buffer descriptor size: 32768 (0x8000)
10:01:25/23:29:08|XLlTemac: Allocating DMA descriptors with kmalloc
10:01:25/23:29:08|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
10:01:25/23:29:08|XTemac: PHY detected at address 7.
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
10:01:25/23:29:09|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09|Using buffer write method
10:01:25/23:29:09|cfi_cmdset_0001: Erase suspend on write enabled
10:01:25/23:29:09|cmdlinepart partition parsing not available
10:01:25/23:29:09|RedBoot partition parsing not available
10:01:25/23:29:09|Creating 5 MTD partitions on "fc000000.flash":
10:01:25/23:29:09|0x000000000000-0x000000500000 : "golden_firmware"
10:01:25/23:29:09|0x000000500000-0x000000800000 : "golden_kernel"
10:01:25/23:29:09|0x000000800000-0x000000d00000 : "user_firmware"
10:01:25/23:29:09|0x000000d00000-0x000000fe0000 : "user_kernel"
10:01:25/23:29:09|0x000000fe0000-0x000001000000 : "env_variables"
10:01:25/23:29:09|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
10:01:25/23:29:10|SPI: XIlinx spi: bus number now 32766
10:01:25/23:29:10|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
10:01:25/23:29:10|SPI: XIlinx spi: bus number now 32765
10:01:25/23:29:10|mice: PS/2 mouse device common for all mice
10:01:25/23:29:10|Device Tree Probing 'i2c'
10:01:25/23:29:10| #0 at 0x81600000 mapped to 0xD1030000, irq=22
10:01:25/23:29:10|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
10:01:25/23:29:10|TCP cubic registered
10:01:25/23:29:10|NET: Registered protocol family 17
10:01:25/23:29:10|RPC: Registered udp transport module.
10:01:25/23:29:10|RPC: Registered tcp transport module.
10:01:25/23:29:10|eth0: XLlTemac: Options: 0x3fa
10:01:25/23:29:11|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
10:01:25/23:29:11|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
10:01:25/23:29:11|eth0: XLlTemac: speed set to 1000Mb/s
10:01:25/23:29:13|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
10:01:25/23:29:13|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
10:01:25/23:29:13|Sending DHCP requests ., OK
10:01:25/23:29:14|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.6
10:01:25/23:29:14|IP-Config: Complete:
10:01:25/23:29:14|     device=eth0, addr=192.168.11.6, mask=255.255.255.0, gw=255.255.255.255,
10:01:25/23:29:14|     host=aida06, domain=dl.ac.uk, nis-domain=nuclear.physics,
10:01:25/23:29:14|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06
10:01:25/23:29:14|Looking up port of RPC 100003/2 on 192.168.11.99
10:01:25/23:29:14|Looking up port of RPC 100005/1 on 192.168.11.99
10:01:25/23:29:14|VFS: Mounted root (nfs filesystem) on device 0:12.
10:01:25/23:29:14|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
10:01:25/23:29:15|Starting sysinit...
10:01:25/23:29:15|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
10:01:25/23:29:16|		Press 'I' to enter interactive startup.
10:01:25/23:29:16|Setting clock  (utc): Fri Jan 10 22:29:16 GMT 2025 [  OK  ]
10:01:25/23:29:16|Building the cache [  OK  ]
10:01:25/23:29:16|Setting hostname aida06:  [  OK  ]
10:01:25/23:29:18|Mounting local filesystems:  [  OK  ]
10:01:25/23:29:19|Enabling /etc/fstab swaps:  [  OK  ]
10:01:25/23:29:21|Finishing sysinit...
INIT: Entering runlevel: 3
10:01:25/23:29:24|Entering non-interactive startup
10:01:25/23:29:24|FATAL: Module ipv6 not found.
10:01:25/23:29:26|Bringing up loopback interface:  [  OK  ]
10:01:25/23:29:29|FATAL: Module ipv6 not found.
10:01:25/23:29:29|Starting system logger: [  OK  ]
10:01:25/23:29:30|Starting kernel logger: [  OK  ]
10:01:25/23:29:30|Starting rpcbind: [  OK  ]
10:01:25/23:29:32|Mounting NFS filesystems:  [  OK  ]
10:01:25/23:29:32|Mounting other filesystems:  [  OK  ]
10:01:25/23:29:33|Starting xinetd: [  OK  ]
10:01:25/23:29:33|Starting midas:  Starting MIDAS Data Acquisition for aida06
10:01:25/23:29:34|xaida: device parameters: base=0x81000000 size=0x200000
10:01:25/23:29:39|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
10:01:25/23:29:40|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
10:01:25/23:29:40|xaida: driver assigned major number 254
10:01:25/23:29:40|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
10:01:25/23:29:45|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
10:01:25/23:29:45|AIDAMEM: aidamem: driver assigned major number 253
10:01:25/23:29:45|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
10:01:25/23:29:52|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
10:01:25/23:29:52|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
10:01:25/23:29:52|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
10:01:25/23:29:52|Computer Name = aida06; Temp Directory = /tmp/tcl361
10:01:25/23:29:57|
10:01:25/23:30:02|AIDA Data Acquisition Program Release 10.0.Jul  6 2022_15:10:57 starting
10:01:25/23:30:02|
10:01:25/23:30:02|Built without pthreads
10:01:25/23:30:02|
10:01:25/23:30:02|Creating NetVars
10:01:25/23:30:02|Output buffer length = 65504; format option = 4; transfer option = 3
10:01:25/23:30:02|EB transfer option = 3
10:01:25/23:30:02|NetVars created anxaida: open:
10:01:25/23:30:02|d initialised
10:01:25/23:30:02|SAIDAMEM: aidamem_open:
10:01:25/23:30:02|tatistics thread starting
10:01:25/23:30:02|Data Acquisition task has PID 375
10:01:25/23:30:02|Statistics thread created
10:01:25/23:30:02|Stat/Rate creation thread starting
10:01:25/23:30:02|Stat/Rate creation thread created
10:01:25/23:30:02|Hit/Rate creation thread starting
10:01:25/23:30:02|Hit/Rate creation thread created
10:01:25/23:30:02|AIDA Heartbeat thread starting
10:01:25/23:30:02|Heartbeat thread created
10:01:25/23:30:02|Installing signal handlers
10:01:25/23:30:02|Done
10:01:25/23:30:02|ModuleNum = 0
10:01:25/23:30:02|Aida Initialise complete. AidaExecV10.0: Build Jul  6 2022_15:10:57. HDL version : 03350706 
10:01:25/23:30:02|package limit is not available: can't find package limit
10:01:25/23:30:02|Running with default file descriptor limit
10:01:25/23:30:02|Spectra table initialised
10:01:25/23:30:02|AIDA Data Acquisition now all ready to start
10:01:25/23:30:02|SIGBUS, SIGSEGV and SIGPIPE traps setup
10:01:25/23:30:02|package setuid is not available: can't find package setuid
10:01:25/23:30:04|Running as user 0 group 0
10:01:25/23:30:04|[  OK  ]
10:01:25/23:30:12|
10:01:25/23:30:13|DENX ELDK version 4.2 build 2008-04-01
10:01:25/23:30:13|Linux 2.6.31 on a ppc
10:01:25/23:30:13|
10:01:25/23:30:13|aida06 login: /debug user "debug" password "ztl9xf9b3o0l"
10:01:25/23:30:18|httpd started on port 8015
10:01:25/23:30:18|
10:01:25/23:30:18|Cannot use /MIDAS/config/TclHttpd/aida06@8015/startup.tcl
10:01:25/23:30:18|Custom startup from /MIDAS/config/TclHttpd/aida06/startup.tcl
10:01:25/23:30:18|XAIDA Access package 1.0
10:01:25/23:30:20|/XAIDAAccessServer
10:01:25/23:30:21|XAD9252 Access package 1.0
10:01:25/23:30:21|/XAD9252AccessServer
10:01:25/23:30:21|/DataBaseAccessServer
10:01:25/23:30:21|/NetVarService
10:01:25/23:30:21|/SigTaskService
10:01:25/23:30:21|Loaded MemSasAccess
10:01:25/23:30:21|/SpectrumService
10:01:25/23:30:21|loading tcl/AIDARunControl.tcl for namespace ::
10:01:25/23:30:21|/DataAcquisitionControlServer
10:01:25/23:30:21|DefineMessage unknown
10:01:25/23:30:21|Run Control Server Implementation for AIDA
10:01:25/23:30:21|RunControlServer loaded
10:01:25/23:30:21|loading Html/RunControl/implementation.tcl
10:01:25/23:30:22|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
10:01:25/23:30:22|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
10:01:25/23:30:22|Created UI registers
10:01:25/23:30:23|RunControl loaded
10:01:25/23:30:23|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
10:01:25/23:30:23|mkdir returned 1
10:01:25/23:30:23|filepath = /MIDAS/FEE_ASIC
10:01:25/23:30:23|filename = /MIDAS/FEE_ASIC/aida06/ASICs.txt
10:01:25/23:30:23|AIDA RunControl loaded
10:01:25/23:30:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
10:01:25/23:30:24|xaida: open:
11:01:25/14:36:47|do_GetState returned z=0 and 8
11:01:25/14:59:08|do_GetState returned z=0 and 8
11:01:25/14:59:20|do_GetState returned z=0 and 8
11:01:25/14:59:32|do_GetState returned z=0 and 8
11:01:25/14:59:44|do_GetState returned z=0 and 8
11:01:25/14:59:56|do_GetState returned z=0 and 8
11:01:25/15:00:08|do_GetState returned z=0 and 8
11:01:25/15:00:20|do_GetState returned z=0 and 8
11:01:25/15:00:33|do_GetState returned z=0 and 8
11:01:25/15:00:45|do_GetState returned z=0 and 8
11:01:25/15:00:57|do_GetState returned z=0 and 8
11:01:25/15:01:09|do_GetState returned z=0 and 8
11:01:25/15:01:21|do_GetState returned z=0 and 8
11:01:25/15:01:33|do_GetState returned z=0 and 8
11:01:25/15:01:45|do_GetState returned z=0 and 8
11:01:25/15:01:58|do_GetState returned z=0 and 8
11:01:25/15:02:10|do_GetState returned z=0 and 8
11:01:25/15:02:22|do_GetState returned z=0 and 8
11:01:25/15:02:34|do_GetState returned z=0 and 8
11:01:25/15:02:46|do_GetState returned z=0 and 8
11:01:25/15:02:59|do_GetState returned z=0 and 8
11:01:25/15:03:11|do_GetState returned z=0 and 8
11:01:25/15:03:23|do_GetState returned z=0 and 8
11:01:25/15:03:35|do_GetState returned z=0 and 8
11:01:25/15:03:47|do_GetState returned z=0 and 8
11:01:25/15:03:59|do_GetState returned z=0 and 8
11:01:25/15:04:11|do_GetState returned z=0 and 8
11:01:25/15:04:24|do_GetState returned z=0 and 8
11:01:25/15:04:36|do_GetState returned z=0 and 8
11:01:25/15:04:48|do_GetState returned z=0 and 8
11:01:25/15:05:00|do_GetState returned z=0 and 8
11:01:25/15:05:12|do_GetState returned z=0 and 8
11:01:25/15:05:24|do_GetState returned z=0 and 8
11:01:25/15:05:36|do_GetState returned z=0 and 8
11:01:25/15:05:49|do_GetState returned z=0 and 8
11:01:25/15:06:01|do_GetState returned z=0 and 8
11:01:25/15:06:13|do_GetState returned z=0 and 8
11:01:25/15:06:25|do_GetState returned z=0 and 8
11:01:25/15:06:37|do_GetState returned z=0 and 8
11:01:25/15:06:50|do_GetState returned z=0 and 8
11:01:25/15:07:02|do_GetState returned z=0 and 8
11:01:25/15:07:14|do_GetState returned z=0 and 8
11:01:25/15:07:26|do_GetState returned z=0 and 8
11:01:25/15:07:38|do_GetState returned z=0 and 8
11:01:25/15:07:51|do_GetState returned z=0 and 8
11:01:25/15:08:03|do_GetState returned z=0 and 8
11:01:25/15:08:15|do_GetState returned z=0 and 8
11:01:25/15:08:27|do_GetState returned z=0 and 8
11:01:25/15:08:40|do_GetState returned z=0 and 8
11:01:25/15:08:52|do_GetState returned z=0 and 8
11:01:25/15:09:04|do_GetState returned z=0 and 8
11:01:25/15:09:17|do_GetState returned z=0 and 8
11:01:25/15:09:29|do_GetState returned z=0 and 8
11:01:25/15:09:41|do_GetState returned z=0 and 8
11:01:25/15:09:53|do_GetState returned z=0 and 8
11:01:25/15:10:06|do_GetState returned z=0 and 8
11:01:25/15:10:18|do_GetState returned z=0 and 8
11:01:25/15:10:30|do_GetState returned z=0 and 8
11:01:25/15:10:43|do_GetState returned z=0 and 8
11:01:25/15:10:55|do_GetState returned z=0 and 8

14.37 Detector bias & leakage current OK - attachment 1
  
      FEE64 temperatures OK - attachment 2

      WR timestamps - attachment 3
       aida06 out of sync

      global clock status - attachment 4
       aida06 fails

      ADC calibration - attachment 5
       all fail - probably due to ASIC synchronise *after* ADC calibration

      WR decoder & FPGA TS status - attachments 6-7 
       aida06 status 0xd

      DAQ run control - attachment 8
       aida06 undefined => aida06 has rebooted

      Merger & Tape Server - attachments 9-10
       merger, tape server and data transfer have stopped

      See https://elog.ph.ed.ac.uk/DESPEC/691
       aidao06 system console - aida06 rebooted sometime yesterday


*** Recovery

For FEE64 aida06 *only*

DAQ reset
DAQ setup


For all FEE64s

DAQ stop
Restart merger, merger setup, merger go 
 Following two merger restart cycles all FEE64s stopped OK
Recovery complete

       All system wide checks OK

       Disable data transfer, DAQ go

       per FEE64 rate spectra = attachment 11
        aida04 high rates

       per FEE64 1.8.W spectra - 20us FSR - attachments 12-13

       DAQ stop, enable data transfer, DAQ go

       Merger and server consoles - attachment 14
        merger and tape server (no storage mode) running OK but no data transfer to MBS server

       WR timestamps OK - attachment 15

       ADC and DISC data item stats - attachment 16-17
        aida04 & aida16 high rates - all others appear to be OK
        ASIC check load for all aida16 ASICs changes rate from c. 420k to 0 
        ASIC check load for all aida04 ASICs changes rate from c. 200k to c. 120k

18.00 No data transfer to MBS - nothing useful being donme at the moment - will place system in safe state

      DAQ stop
      Detector bias OFF
      FEE64 power OFF

LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.

The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).

I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.

> 
> LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> 
> The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> 
> I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.

Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18

"We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.

This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending data. (@Nic is this correct?)

This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."

> > 
> > LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> > 
> > The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> > 
> > I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
> 
> Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18
> 
> "We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.
> 
> This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending data. (@Nic is this correct?)
> 
> This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."

We have increased the all thresholds excluding aida10 to 0xff. To not trigger on the other FEEs, however better, the DAQ still occasionally runs into dead time and as such fails with making trees.

We have thus elected to turn off AIDA and move out the platform. The conclusion is that we are not able to do this fast timing test with a 200 keV threshold.

> > > 
> > > LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> > > 
> > > The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> > > 
> > > I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
> > 
> > Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18
> > 
> > "We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.
> > 
> > This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending 
data. (@Nic is this correct?)
> > 
> > This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."
> 
> We have increased the all thresholds excluding aida10 to 0xff. To not trigger on the other FEEs, however better, the DAQ still occasionally runs into dead time and as such fails with making trees.
> 
> We have thus elected to turn off AIDA and move out the platform. The conclusion is that we are not able to do this fast timing test with a 200 keV threshold.

Per our discussions in December 2024 this is exactly what would be expected.

16:30 We moved the platform in. The system was still biased and the DAQ was still running.

The system seems to be okay, temperatures OK. Bias voltage OK 685/1 and 685/2. Two FEEs - aida07 and aida12 are receiving a lot of trigger data items from the fast discriminator. This is already a bit worrying. 685/3

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

678/1 shows the AIDA OR64 trigger from the fast comparator of aida10.

678/2-4 show the bPlast accepted trigger after the bPlast DAQ has been triggered by the AIDA OR64. The images show a signal coming fast in time after the pulser and also a signal at around 80 us that appears to be coming from pile up.

The bPlast DAQ also now runs with the AIDA OR64 trigger after the NIM out signal from the MACB was sent to an octal discriminator to fix the width of the signal and also the pulse width which from TAMEX should be kept at or above 100 ns. The NIM signal from the MACB was mostly 100 ns but also jumping to signals with a 20ns pulse width.

The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.

These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.

This was set to have the HitRate in aida10 to be just above the noise.

Initial results: copy from DESPEC elog.

We have managed to obtain the time difference between the bPlast White rabbit and the AIDA Fast time. We had to gate out the zero fast-time events and then also condition that we only take the data from aida10. We see this in 684/1. Then if we look at the time difference between the fast time and the bPlast WRT we see a sharp peak at zero, this makes sense as the AIDA fast time discriminator is being triggered by a pulser which is then being used as a trigger for bPlasts DAQ so these events should be virtually arriving without delay. The delay we do see is infact around 750 ns.

The data collected overnight was also analysed as shown in 684/3 this is the data accumulated with the 22Na source close to the snout and a global threshold in the fast discriminator of 0x0f. The centre peak was roughly fitted with a gaussian:    

  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  Constant     1.21277e+03   2.74526e+01   5.66915e-01  -5.61074e-07
   2  Mean        -1.78513e+02   8.50699e+00   1.62947e-01   1.58554e-06
   3  Sigma        3.48571e+02   4.03993e+00   6.26343e-05  -1.86863e-03

8:01:25/11:42:48|0x000000d00000-0x000000fe0000 : "user_kernel"
08:01:25/11:42:48|0x000000fe0000-0x000001000000 : "env_variables"
08:01:25/11:42:48|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32766
08:01:25/11:42:48|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32765
08:01:25/11:42:48|mice: PS/2 mouse device common for all mice
08:01:25/11:42:48|Device Tree Probing 'i2c'
08:01:25/11:42:48| #0 at 0x81600000 mapped to 0xD1030000, irq=22
08:01:25/11:42:48|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
08:01:25/11:42:48|TCP cubic registered
08:01:25/11:42:48|NET: Registered protocol family 17
08:01:25/11:42:48|RPC: Registered udp transport module.
08:01:25/11:42:49|RPC: Registered tcp transport module.
08:01:25/11:42:49|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:42:49|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:42:51|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:42:51|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:42:51|Sending DHCP requests .
08:01:25/11:42:53|eth0: XLlTemac: PHY Link carrier lost.
08:01:25/11:42:53|..... timed out!
08:01:25/11:44:15|IP-Config: Reopening network devices...
08:01:25/11:44:15|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:44:16|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:44:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:44:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:44:18|Sending DHCP requests ...... timed out!
08:01:25/11:45:35|IP-Config: Auto-configuration of network failed.
08:01:25/11:45:35|Root-NFS: No NFS server available, giving up.
08:01:25/11:45:35|VFS: Unable to mount root fs via NFS, trying floppy.
08:01:25/11:45:35|VFS: Cannot open root device "nfs" or unknown-block(2,0)
08:01:25/11:45:35|Please append a correct "root=" boot option; here are the available partitions:
08:01:25/11:45:35|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
08:01:25/11:45:35|Call Trace:
08:01:25/11:45:35|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
08:01:25/11:45:35|[c6827f10] [c00345bc] panic+0x94/0x168
08:01:25/11:45:35|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
08:01:25/11:45:36|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
08:01:25/11:45:36|[c6827fd0] [c0341220] kernel_init+0x104/0x130
08:01:25/11:45:36|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
08:01:25/11:45:36|Rebooting in 180 seconds..

Yesterday we spent time essentially performing a dry run to get AIDA DAQ and bPlast into the time sorter, there were some issues with the AIDA mbs PC x86l-119 which was related to some boot issue, it was booting to a newer version of debian while the Relay for AIDA to MBS is on an older version (scratch) (the machine is quite old). We spent a bit of time getting the thresholds correct for each of the ASICs. In the end we just elected to have a blanket level of 0x0f for all of the ASICs when we removed the pulser.

Last night we left the setup with the following thresholds on all of the FEEs to collect data. 681/1. The pulser was also turned off and the 22Na source was moved close to the snout.

This morning c. 9:25 we returned and checked the temperatures, HV and statistics and everything seems to be ok. The only problem was that the timesorter has crashed this morning at 8 am.

See also https://elog.ph.ed.ac.uk/CARME/489

1. Establish port (to access carme-gsi)               ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

   Establish port (to access aida-gsi)                ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin

   Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes

2. Check AnyDesk password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)

11.54 re-established remote access via AnyDesk - details to follow

      confirmed AIDA FEE64 power OFF, detector HV OFF


1. Establish port (to access carme-gsi)               ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

   Establish port (to access aida-gsi)                ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin

   Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes

2. Check AnyDesk password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)

11:00 Visual inspection FEE64 adaptor PCBs & cabling
      aida02 ground from/to other FEE64s disconnected - re-connected
      aida01, aida09, aida12 - ground cabling screws to Lemo 00.250 housings loose - tightemed

12:50 Cooling water pressure & temperature OK - attachments 1  2

12:59 relay #1 power ON

13:01 relay #2 power ON

13:07 aida06 starts - panic during startup, automatic restart following 3 minute timeout

      DAQ reset, setup

      Check ASIC Control - browser tab timeout 
      AIDA MIDAS HTTPD server console log - attachment 3

      Appears to have restored ASIC settings 2024Dec13-17.02.45 saved yesterday
    
      aida10 ASICs #1 & #2 positive input, ASICs #3 & #4 negative input
      slow comparator 0xa (all p+n junction FEE64s and aida10), 0xf (n+n Ohmic FEE64s)

13:28 tar ASIC settings - attachment 4

[npg@aidas-gsi]$ cd /MIDAS/FEE_ASIC
[npg@aidas-gsi FEE_ASIC]$ tar cvf /tmp/FEE_ASIC.tar .

       System wide checks

       Sync ASIC clocks - attachment 5

       Clock, ADC calibration, WR decoder, FPGA timestamp, PLL checks - attachments 6-11
        all OK *except* aida02 WR decoder error

       WR timestamps OK - attachment 12
       FEE64 temps OK - attachment 13
       
13:45 Detector bias ON - attachment 14      

      BNC PB-5 pulser settings - attachment 15
      Pulser connected to all p+n junction FEE64s *except* aida10

      ADC, DISC, PAUSE and MBS correlation scaler stats - attachments 16-19
       aida02 rate significantly lower than yesterday - https://elog.gsi.de/despec/Implantation+Stack/8
       high rates observed for aida08, aida11 and aida14 - which are not connected to a DSSSD!

      per FEE64 Rate spectra - attachment 20

      per p+n junction FEE64 1.8.L spectra - attachment 21
       aida09 pulser peak width 56 ch FWHM = 39 keV FWHM
       consistent electronic noise for all p+n junction FEE64s (cabling+DSSSD)
       electronic noise of p+n junction FEE64s (cabling *only*) higher and more variable cf. https://elog.gsi.de/despec/Implantation+Stack/8 attachment 5

      per p+n junction FEE64 1.8.W spectra 20us FSR - attachments 22-23

      per n+n Ohmic FEE64 1.8.W spectra 20us FSR - attachment 24 

      WR timestamps OK - attachment 25

14:43 DAQ STOP
      Data transfer enabled
      Select Tape Server -> Next Run
      DAQ GO  data file R3

      Merger, Tape Server - attachments 26-27
       data transfer rate c. 900k data items/s cf c. 300k data items/s yesterday https://elog.gsi.de/despec/Implantation+Stack/8

16:45 DAQ STOP

      Data transfer disabled 

      Detector bias OFF

      FEE64 power OFF

Snout assembled for detector position and timing test.

The planned test is to use a BGO with a 22-Na source and the implantation stack to characterise the timing and position of back-to-back 511 keV events (proxy for mock beta decay event).

The implantation stack is assembled as follows:

Upstream ----> AIDA test frame (as a spacer) ----> AIDA DSSSD (3208-3/3208-21/3208-22, Upstream AIDA used in S100 & S181, April 2024) ----> bPlast (Downstream bPlast) ----> BB7 (mounted in test frame connected to AIDA adaptor) ----> Downstream

The AIDA DSSSD condition seems to be the same as when it was dismounted after S181 and it, apart from some specks of dust no bond wires appear to be broken to the naked eye.

BGO signals appear on the scope with a 22-Na source but the response is quite messy.

17:36

MSL type BB18(DS)-1000 24cm  x 8cm DSSSD upstream position within snout using 8x upstream (23") ribbon cables 

MSL type BB(DS)-1000 positioned downstream of downstream bPlast within snout using 1x downstream (29") ribbon cable. For BB7 'isolated' Right Coupler Kapton PCB replaced by unmodified Right Coupler Kapton PCB.

BB7 connected to aida07, sitting in a cardboard box

Update Tcl/Tml files in ASIC, ASIC4 and sys.tml frm PJCS email to allow setting per-ASIC polarity on FEE64
Restart MIDAS@8015

Power on all FEEs

Note nnrpi is not responding to ssh... MP investigates if it ahs been unplugged
 No, but a powercycle worked and fixed it

See error: /MIDAS/FEE_ASIC/aida06/ASICs.txt  not found, create the folders

We see some data in the L spectra, and changing the settings from one ASIC seems to work properly

Stopped for CC/MP to determine the which ASICs correspond to p+n and n+n

Follow up: Alpha data overnight or source?

Offline analysis of S100 data files R21_0 - R21_99 (162Eu setting)

first WR ts
First timestamp of R21_0 0x17CA09154AE3E636

Epoch converter says ...

GMT: Saturday, April 27, 2024 4:36:35.223 AM
Your time zone: Saturday, April 27, 2024 5:36:35.223 AM GMT+01:00 DST

last WR ts
First timestamp of R21_100 0x17CA16C1904150CE

GMT: Saturday, April 27, 2024 8:47:08.772 AM
Your time zone: Saturday, April 27, 2024 9:47:08.772 AM GMT+01:00 DST


FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4
DSSSD#2 11  7 16 10 14 13  6  8

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (1012 of 1024 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/-168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   DSSSD #1
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   DSSSD #2 
   0 < p+n junction side multiplicity < 8 
   and
   0 < n+n Ohmic side multiplicity < 8 

   151keV < LEC energy < 1008keV
    to select candidate beta events and veto higher energy events e.g. light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500

Attachments 1-4
per DSSSD p+n junction - n+n Ohmic strip time difference for HEC and LEC events (2us/channel) linear and log scale

- observe large (> 32us) time differences (on log scale)

- range of time differences increases with multiplicity ( DSSSD#1 cf. DSSSD#2 LEC events)

- distribution of HEC time differences can probably be understood in terms of most/all channels of ASIC being active during HEC event with low LEC thresholds

- AIDA is a triggerless DAQ producing streams of ADC data items *not* events
   at high instantaneous rates when events are constructed they may become aggregated in time i.e. > 32us readout time of all channels of one ASIC

- To investigate impose additional end of event criterion 
   difference in first and last WR timestamp of event < 33us

Attachments 5-6
per DSSSD p+n junction - n+n Ohmic strip time difference for HEC and LEC events (2us/channel) linear and log scale

- blue original end of event criteria, cyan new end of event criteria

- as expected range of time differences is restricted to +/- 32us 

- observe somewhat higher fraction of events with low time differences
 
  DSSSD #1 10363098 of 16104322 (64%) events +/-2us 

  DSSSD #2 860454912 of 1766618199 (49%) events +/-2us

Attachment 7
per DSSSD p+n junction - n+n Ohmic strip time difference for LEC events - x-axis 2us/channel, y-axis 20keV /channel


Attachment 8 per FEE64 LEC data rate (Hz) 268ms/channel
Attachment 9 per FEE64 LEC data rate (Hz) 268ms/channel: 150keV < energy < 1500keV
Attachment 10 per FEE64 LEC data rate (Hz) 268ms/channel: energy > 1500keV

- observe high instantaneous rate on spill
- rate dominated by low energy (<1500keV) events
- rate of higher energy events dominated by on spill events i.e. light ions as expected
- significant deadtime on spill for n+n Ohmic FEE64s, low deadtime off spill
- deadtime low/zero for p+n junction FEE64s on/off spill

Attachment 11 per FEE64 LEC hit pattern: 150keV < energy < 1500keV
Attachment 12 per FEE64 LEC hit pattern: energy > 1500keV


Attachment 13 per FEE64 HEC data rate (Hz) 268ms/channel
Attachment 14 per FEE64 HEC data rate (Hz) 268ms/channel: 100MeV < energy < 1000MeV
Attachment 15 per FEE64 HEC data rate (Hz) 268ms/channel: energy > 1000MeV

- rate dominated by low energy (>1GeV) events
- all HEC events on spill as expected (note FEE64 #7 has a single hot channel which can be disabled in software)
- significant deadtime on spill for n+n Ohmic FEE64s, low deadtime off spill
- deadtime low/zero for p+n junction FEE64s on/off spill

Attachment 16 per DSSSD p+n junction versus n+n Ohmic LEC energy - x-axis & y-axis 20keV/channel

Attachment 17 per DSSSD p+n junction versus n+n Ohmic HEC energy - x-axis & y-axis 20MeV/channel

Attachment 18 per DSSSD p+n junction versus n+n Ohmic HEC strip hit pattern: all HEC events

Attachment 19 per DSSSD p+n junction versus n+n Ohmic HEC strip hit pattern
 DSSSD #1 ions stopped in DSSSD #1 i.e. DSSSD #2 HEC multiplicity = 0
 DSSSD #1 shows x-y gate used ( 270 < x < 370, 20 < y < 90 ) to identify 166Tb implants
 DSSSD #2 ions stopped in DSSSD #2 *and* in transmission (can establish which ions stop in DSSSD#2 from DSSSD#2 HEC energy versus DSSSD#1 HEC energy - see https://elog.ph.ed.ac.uk/DESPEC/672

Attachment 20

DSSSD#1 HEC energy (20MeV/channel) versus HEC-LEC dt (1s/channel)

DSSSD#1 LEC energy (20keV/channel) versus HEC-LEC dt (1s/channel)

DSSSD#1 HEC strip # versus HEC-LEC dt (1s/channel)

- Observe # events in every third channel is lower 
- Probably reflects implant-decay correlation livetime
  For example (choosing some numbers for illustrative purposes)
  on spill: HEC livetime 75%, LEC livetime 75% (FEE64 deadtime common for HEC and LEC data) => implant decay correlation livetime 56%
  off spill: HEC live time 75%, LEC livetime 100% =>  implant decay correlation livetime 75%
- Observe 'hot' x channels 315, 318, 321, 324 - disabled for further analysis
- Do not observe any 'hot' y channels 


Attachments 21 & 22

DSSSD#1 per pixel HEC-LEC time (1s/channel): x,y,z gated to select 166Tb events

Naive (parent-daughter decay only, flat background) fit for data t=0-26s ( t1/2 = 27.1(3)s )

Fit ignores data for t=0, 3, 6, 10, 13, 16, 19, 22s to avoid bias of differences in implant-decay correlation deadtime

Suggestion of structure at c. 30s period? Does this reflect spill stucture? 10x spill cycles (30s), 9s spill off, ... etc

Sum of x,y,z gated HEC events (s2112 - see attachment 19) = 670441

Elapsed time of dataset 4h11m = 15060s
# pixels = 100 x 70 = 7000
=> # x,y,z gated HEC events per pixel = 670441/7000/15060 = 0.0064/s or mean time between x,y,z gated HEC event = 157s (estimate needs to be corrected for HEC deadtime)

Sum of implant decay correlations (s2220 - see attachment 21) t=0-150s = 273508 - flat background estimated as 150 x 500 = 75000 = 198508

=> efficiency c. 30% (presumably low due to implant-decay deadtime, LEC multiplicity, per pixel correlations and no clustering)


Summary

$64,000 question - what is the origin of the high instantaneous rate on spill  for DSSSD#1 ? On my to do list.


Attachment 23

LEC multiplicity with/without HEC data in event

per DSSSD LEC p+n junction multiplicity versus n+n Ohmic multiplicity
per DSSSD LEC p+n junction multiplicity versus n+n Ohmic multiplicity z_hec=1 and z_hec=2


With HEC data

DSSSD#1 p+n junction multiplicity ~ 17, n+n Ohmic multiplicity ~28
DSSSD#2 p+n junction multiplicity ~ 40, n+n Ohmic multiplicity ~23

Assume 200Hz HEC events => DSSSD#1 LEC rate = 200 x ( 17 + 28 ) => 9k LEC data items cf. >100k LEC data items (attachments 8 & 13)

i.e. not due to HEC events








 
 


 







   

Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Offline analysis of data files S505/R5_780 - R5_814 (corresponding to MBS data files 73-74) 

Can find an analysis of alpha background runs at RIBF, RIKEN for comparison at https://elog.ph.ed.ac.uk/AIDA/816

Beam off - background runs without sources

MBS 73 27.6.22 09:33-11:33 CEST https://elog.gsi.de/despec/S505/204

MBS 74 27.6.22 11:34-13:34 CEST https://elog.gsi.de/despec/S505/206

MBS 75 27.6.22 13:35-16:29 CEST https://elog.gsi.de/despec/S505/209

ADC offsets per https://elog.ph.ed.ac.uk/DESPEC/556

FEE64 configuration

FEE64   a b c d
DSSSD#1 3 4 1 2
DSSSD#2 7 8 5 6

p+n junction FE64s odd numbered


Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
- no clustering
- no p+n junction side - n+n Ohmic side correlation time gates
- valid LEC events
   0 < p+n junction side multiplicity < 8 
   and
   0 < n+n Ohmic side multiplicity < 8 


Attachments 1-2 - per DSSSD p+n versus n+n multiplicity

Attachment 3 - per DSSSD x versus y

Attachments 4-5 - per DSSSD p+n versus n+n energy (20keV/channel nominal)
                  all combinations of per DSSSD p+n junction and n+n Ohmic energies 
                  with projection of data within window onto x and y axes

                  too many events for natural (U decay series) background
                  off leading diagonal correlations anomalous
                  transverse width of leading diagonal correlation wider than expected - ADC offsets OK?
                 

Attachment 6 - per FEE64 WR timestamp (32.768us/channel)
               FEE64 sync test using pulser data - looks OK

Attachments 7-14 - per FEE64 ADC spectra (5.6keV/channel nominal)
                   note common x/y scale - pulser peak height proxy for peak width

per FEE64 1.8.L pulser peak widths (ch FWHM)

1 11.43
2 16.58
3 12.52
4 17.54
5 9.10
6 12.22
7 17.03
8 16.09

3 of 4 p+n junction FEE64 good (<70keV), 1 of 4 n+n Ohmic FEE64 good - all others < 100keV
Pulser peak indicates noise/gain/offset stable throughout background runs


Attachments 15-22 - per FEE64 ADC spectra (5.6kleV/channel nominal)
                    observe broad peak-like structures (c. 2.8MeV) in first channel of most ASICs? 

Attachment 23 per DSSSD p+n versus n+n energy (20keV/channel nominal)
              valid LEC events
              p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1

Attachment 24 p+n FEE64 #0 energy versus each of n+n FEE64s (#1, #3, #5 and #7) energies (20keV/channel nominal)
              valid LEC events
              p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1

Attachments 23 & 24 do not unambiguously identify which FEE64s are attached to which DSSSD
                  

Three BB18-1000 triples AIDAs collected on 14.09.2024

Find attached visual of the wafer and bond wire + factory bias tests accompanying the DSSSDs. elog:669/1 elog:669/2 elog:669/3

Visual inspection carried out showed that bond wires have been fixed + fingerprint on one DSSSD removed and wires repaired. elog:669/4 elog:669/5 elog:669/6 elog:669/7 elog:669/8 elog:669/9 elog:669/10 elog:669/11

DSSSD 1 (defect bias issue 80V): 3208-10 / 3208-18 / 3208-20

DSSSD 2  (3208-6 dysfunctional): 3208-6 / 3208-9 / 3208-16

DSSSD 3 (defect fingerprint): 3131-5 / 3131-10 / 3131-12

03.11 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4

05.33 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-11
       aida09 pulser walkthrough 60 ch FWHM

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-14

      Merger, TapeServer etc - attachments 15-16

05.40 analysis data file R9_1042 - attachment 17

      Check ASIC load x2

      aida02 zero rate
      restart Merger

05.55 Data file S181/R10
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 25Hz

      per p+n FEE64 1.8.L spectra - attachment 18

      per FEE64 Rate spectra - attachment 19

06.43 TapeServer no storage mode

09.14 Data file S181/R11
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 25Hz

09.52 TapeServer no storage mode

      per n+n FEE64 1.8.L spectra - attachment 20

15.18 DEFGAS/FATIMA arrays moved downstream to permit installation of calibration source at centre of each array - increased distance to AIDA/bPlas => lower rate


15.16 DSSSD bias & leakage current OK - attachments 21-22

      FEE64 temperatures OK - attachment 23

      ADC data item stats OK - attachment 24
       aida02 zero rate

      per FEE64 Rate spectra - attachment 25

      Merger restart
      ADC data item stats OK - attachment 26
       7x < 20k, max 207k aida16

15.54 bPlast debiased.


16.29 Data file S181/R12
      bPlas OFF but remiains cabled/connected/grounded

16.30 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30

      per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - see attachments 31-39
       note - not all FEE64s updated spectra for all settings

      ADC data item stats OK - attachment 40
       rates change cf. attachment 29
       sources installed/moved/removed, people moving around platform?

      per FEE64 Rate spectra - attachment 41

16.57 DSSSD bias OFF
      FEE64 power OFF

16.34 analysis data file S181/R12_7
      max deadtime 2.2% (aida08)
      ADC data rate 1.330M, HEC data rate 2.1k

16.52 analysis data file S181/R12_12
      max deadtime 1.7% (aida08)
      ADC data rate 945k, HEC data rate 1.8k

04.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4


07.52 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per p+n FEE64 1.8.L spectra - attachments 9-10
       aida09 pulser peak width 59 ch FWHM


15.01 DSSSD bias & leakage current OK - attachment 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats OK - attachment 13

      per p+n FEE64 1.8.L spectra - attachments 14-15
       aida09 pulser peak width 59 ch FWHM
       aida05 pulser peak width 68 ch FWHM
       aida12 pulser peak width 71 ch FWHM


      per FEE64 Rate spectra - attachment 16

      Merger, TapeSever etc - attachment 17
       no data transfer to MBS - MBS down?

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 


19.12 DSSSD bias & leakage current OK - attachment 19

      FEE64 temperatures OK - attachment 20

      ADC data item stats OK - attachment 21

      per FEE64 Rate spectra - attachment 22

23.34 DSSSD bias & leakage current OK - attachment 23

      FEE64 temperatures OK - attachment 24

      ADC data item stats OK - attachment 25

      per FEE64 Rate spectra - attachment 26

Beam lost due to septum failure c. 18.15 Friday 14 June

Cooling of recirculating water failed c. 19.00 - water temperature increased from the normal 17-18deg C to 25-26 deg C
FEE64 temperatures increased and FEE64s powered off and on-call engineer requested.

Cooling of recirculating water restored and AIDA restarted

NH started background run 23.37 - data file S181/R9

01.21 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3
       8x < 20k, max 238k aida08

      per FEE64 Rate spectra - attachment 4

      2x Check ASIC load

      Merger, TapeServer etc - attachments 5-6

      analysis data file R9_27 - attachment 7
       max deadtime 1.9%, all other FEE64s < 1%
       ADC data rate 951k, HEC data rate 1.6kHz

04.28 DSSSD bias & leakage current OK - attachment 8

      FEE64 temperatures OK - attachment 9

      ADC data item stats OK - attachment 10
       8x < 20k, max 261k aida08

      per FEE64 Rate spectra - attachment 11


08.30 DSSSD bias & leakage current OK - attachments 12-13

      FEE64 temperatures OK - attachment 14

      ADC data item stats OK - attachment 15
       all rates increased - gamma source in use?

      per FEE64 Rate spectra - attachments 16-17
       note aida08 asic #3 hot HEC channel

      per FEE64 1.8.L spectra - attachments 18-20
       aida09 pulser peak width 59 ch FWHM
       decay alphas aida05, aida12, aida02, aida04

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-23

      Merger, TapeServer etc - attachments 25-25

      analysis data file R9_152 - attachment 26
       max deadtime 2.4%, all other FEE64s < 1%
       ADC data rate 1.215M, HEC data rate 2.1kHz


12.22 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30



15.27 DSSSD bias & leakage current OK - attachment 31

      FEE64 temperatures OK - attachment 32

      ADC data item stats OK - attachment 33
       all rates increased - 370kBq 152Eu in use

      per FEE64 Rate spectra - attachments 34

      per FEE64 1.8.L spectra - attachments 35-36
       aida09 pulser peak width 59 ch FWHM
       decay alphas aida05, aida12, aida02, aida04

      analysis data file R9_275 - attachment 37
       max deadtime 4.4%, all other FEE64s < 2%
       ADC data rate 1.520M, HEC data rate 2.3kHz

18.52 DSSSD bias & leakage current OK - attachment 38

      FEE64 temperatures OK - attachment 39

      ADC data item stats OK - attachment 40

      per FEE64 Rate spectra - attachment 41


22.50 DSSSD bias & leakage current OK - attachment 42

      FEE64 temperatures OK - attachment 43

      ADC data item stats OK - attachment 44

      per FEE64 Rate spectra - attachment 45

Everything appears to be running smoothly. Attachments 1-4 were taken around 16:00

DSSSD Bias & Leakage Current - Check: OK (see attachment 1)
FEE64 Temperature - Check: OK (see attachment 2)
ADC Data Item Stats and Spectrum Browser - Check: OK (see attachments 3 and 4)

Another set of screenshots was taken around 18:00, and everything looks good.

DSSSD bias & leakage current  ok - attachment # 5-6
FEE64 temperatures  ok - attachment # 7
ADC data item stats - attachment # 8
Merger ok - Attachement #9
Tape service - attachement # 10
ucesb - attachment #11

At 19:17, AIDA03 temperatures gave no response (see attachment 12), but after two minutes, they returned to normal.

I noted that the  bplse in ucesb has no data (see attachment 13). It may need to change the disc thresholds again
 it does not affect AIDA

At 19:30 most aida temperature are high and I spoke to people on site via zoom one of them she is working with AIDA so she was able to help. She called Nick and it is found that there is a problem in the cooling system so she stopped AIDA

At 22 the cooling system fixed and Nic and the other person on site restarted AIDA

07.23 All histograms & stats zero'd

      DAQ continues data file S181/R8_35

      analysis data file R8_35 - attachment 1
      max deadtime 4.8% (aida08), all other FEE64s < 3%
      LEC data rate 1.677M HEC data rate 3.7k

08.46 DSSSD bias & leakage current OK - attachments 2-3

      FEE64 temps OK - attachment 4

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 6

      per p+n FEE64 1.8.L spectra - attachment 8
       aida09 pulser peak width 65 ch FWHM

      per FEE64 1.8.H spectra - attachment 7 & 10

      per FEE64 1.8.W spectra - 20us FSR - attachment 9 & 11

      Merger, TapeServer etc - attachments 12-13
       data file S181/R8_72


10.51 DSSSD bias & leakage current OK - attachments 14

      FEE64 temps OK - attachment 15

      ADC data item stats - attachment 16

      per FEE64 Rate spectra - attachment 17

10.54 analysis data file R8_122 - attachment 18
       max deadtime 7.8% (aida08), all other FEE64s < 3%
       ADC data rate 1.88M, HEC data rate 4.7k


14.24 DSSSD bias & leakage current OK - attachments 18

      FEE64 temps OK - attachment 19

      ADC data item stats - attachment 20

      per FEE64 Rate spectra - attachment 21

      aida02 no data

14.28 merger restarted and DAQ running OK 


14.38 analysis data file R8_213 - attachment 22
       max deadtime 6.6% (aida08), all other FEE64s < 3.5%
       ADC data rate 1.935M, HEC data rate 3.6k

0.00 checks - all looks ok

Voltage/currents - attachment 1

Rates attachment 2 + 3

Temps - attachment 4

Merger - attachment 5

ucesb - attachment 6

Grafana - attachment 7

1.30 -aida1 in merger Links, aida02 in rate spectra, has stopped - resetting DAQ.

Datafile R7_961 being written when stopped

1.40 - seems ok

1.50 checks - seems ok

Voltage/currents - attachment 8

Rates attachment 9 + 10

Temps - attachment 11

Merger - attachment 12

ucesb - attachment 13

Grafana - attachment 14

3.30 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1010 being written when stopped

3.38 - seems ok

3.40 checks - all looks ok

Voltage/currents - attachment 15

Rates attachment 16 + 17

Temps - attachment 18

Merger - attachment 19

ucesb - attachment 20

Grafana - attachment 21

Zeroed spectra - attachments 22-28

5.39 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1062 being written when stopped

AIDA out of the main data

Could not get it to restart folowing the notes on the elog - SOAP errors as shown in attachment 29. called Tom who''s logged in 

6.10 - seems to be working again. Not sure what happened. Tom thinks multiple restarts fixed it

Started data file R8

AIDA back in the main data

FRS shifters report a problem with the AIDA white rabbit time stamp - attachment 30

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  17D8C4ED 8F46CF7B , WR/10=>   2627A17C18714BF, Readout Time =>   2627A17C2A74000
aida02 : White Rabbit=>  17D8C4ED A0A1B623 , WR/10=>   2627A17C3435F03, Readout Time =>   2627A17C4410000
aida03 : White Rabbit=>  17D8C4ED AFDACE68 , WR/10=>   2627A17C4C914A4, Readout Time =>   2627A17C5D3C000
aida04 : White Rabbit=>  17D8C4ED C0516FC2 , WR/10=>   2627A17C66E8B2D, Readout Time =>   2627A17C7AC4000
aida05 : White Rabbit=>  17D8C4ED D3B93DAB , WR/10=>   2627A17C85F52F7, Readout Time =>   2627A17C9AEC000
aida06 : White Rabbit=>  17D8C4ED E8C9B243 , WR/10=>   2627A17CA7A91D3, Readout Time =>   2627A17CB704000
aida07 : White Rabbit=>  17D8C4ED FFE475FD , WR/10=>   2627A17CCCA0BCC, Readout Time =>   2627A17CDD94000
aida08 : White Rabbit=>  17D8C4EE 103232A6 , WR/10=>   2627A17CE6B6B77, Readout Time =>   2627A17CF7BC000
aida09 : White Rabbit=>  17D8C4EE 20A1F612 , WR/10=>   2627A17D0103235, Readout Time =>   2627A17D112C000
aida10 : White Rabbit=>  17D8C4EE 307D8FE7 , WR/10=>   2627A17D1A627FD, Readout Time =>   2627A17D2958000
aida11 : White Rabbit=>  17D8C4EE 4402829D , WR/10=>   2627A17D399D9DC, Readout Time =>   2627A17D4570000
aida12 : White Rabbit=>  17D8C4EE 6288B04D , WR/10=>   2627A17D6A744D4, Readout Time =>   2627A17D7D94000
aida13 : White Rabbit=>  17D8C4EE 751E1306 , WR/10=>   2627A17D88301E7, Readout Time =>   2627A17D9C34000
aida14 : White Rabbit=>  17D8C4EE 894E8806 , WR/10=>   2627A17DA87DA67, Readout Time =>   2627A17DBFF4000
aida15 : White Rabbit=>  17D8C4EE 9EEEB997 , WR/10=>   2627A17DCB178F5, Readout Time =>   2627A17DDC18000
aida16 : White Rabbit=>  17D8C4EE AF0BC923 , WR/10=>   2627A17DE4DFA83, Readout Time =>   2627A17DF5C8000

Tom thinks everything appears ok with AIDA

Have converted WR timestamp to data/time - see attachment 40

At the ~second timescale AIDA WR timestamp looks OK, i.e. no gross errors. Will need to check correlations with other detector sub-systems to confirm WR on the ~us time timescale.. 

However, ~10us offset reported by DESPEC online crew is correct (AIDA timestamps data at a later point in the signal processing cycle than other detector sub-systems). Have suggested online crew contact Nic or Calum to check what the AIDA WR timediff spectra should look like.

7.00 checks - all looks ok

Voltage/currents - attachment 31

Rates attachment 32 + 33

Temps - attachment 36

Merger - attachment 37

ucesb - attachment 39

Grafana - attachment 38

Handed over to Tom

Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.

However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).

Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.

Another set of sceenshots taken at around 23:00 - attachments 19 - 26.

Took over the shift from Marc.

08:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

10:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12

12:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 13-14
FEE64 temperatures  ok - attachment # 15
ADC data item stats - attachment # 16
Merger ok - Attachement # 17
Tape service - attachement # 18

12.34 analysis data file S181/R7_610 - attachment 19

max deadtime 8.7% (aida08), all other FEE64s < 2.7%

LEC data rate 2.000M, HEC data rate 4.7k

12.52 per p+n FEE64 1.8.L spectra - attachment 20

 aida09 pulser peak width 63 ch FWHM

per FEE64 1.8.H spectra - attachments 21-22

per FEE64 1.8.W spectra - 20us FSR - attachments 23-24

13.00 all histograms zero'd

14:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 19-20
FEE64 temperatures  ok - attachment # 21
ADC data item stats - attachment # 22
Merger ok - Attachement # 23
Tape service - attachement # 24

13.01 Beam OFF for S4 access

ADC data item stats - attachment 25

per FEE64 Rate spectra - attachment 26

14.12 analysis data file R7_654 - attachment 27

max deadtime 3.9% (aida08), all other FEE64s < 3%

LEC data rate 1.817M, HEC data rate 2.5k

14.00 data file R7_649 - attachment 28

max deadtime 4.5% (aida08), all other FEE64s < 3%

LEC data rate 1.756M, HEC data rate 2.5k

16:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 25-26
FEE64 temperatures  ok - attachment # 27
ADC data item stats - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30

Starting a new night shift - All good

1am full check:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

2:30am: AIDA02 FEE stopped working. DESPEC main DAQ was alerted and remobe AIDA from timestitcher while I was restarting AIDA DAQ.

2:45am: AIDA02  FEE is back and AIDA has been added to the timestitcher again in the main DAQ.

The AIDA runs at the time of the crash to look at are R7_348, 349.

All spectra were zeroed at this time and checked.

4:00 am full check:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - attachement # 11
Tape service - attachement # 12
Spectra - attachement # 13
ucesb - attachement # 14

6:00am full check:
DSSSD bias & leakage current  ok - attachment # 15-16
FEE64 temperatures  ok - attachment # 17
ADC data item stats - attachment # 18
Merger ok - attachement # 19
Tape service - attachement # 20
Spectra - attachement # 21
ucesb - attachement # 22

6:30 the nearline/offline analysis team reported possible noise issue in DSSD#2 and asked if the threshold were changed at any point. I can't find any mention of changes in the elog. 
I've not done a setup of the all DAQ when AIDA02 fee crashes earlier this morning. I just did a setup of the merger. Thus I don't think the threshold have changed when I stopeed and restarted the DAQ.
I can see that currently slow comparator threshod are set to 0xa for most of the DSSD2 fees except for AIDA06 which is set to 0xf. I'm not sure this is the thershold we would need to increase. Something to check during the next shift.

In the UCESB window I can see that the number of decay in DSSD2 is double the number of decay in DSSD1 while the number of implant is similar. Looking at other shift it looks that this was alsready the case in prvious shifts.

07.06 analysis of data file S181/R7_457 - attachment 23

 max deadtime 5.8% aida08, all other FEE64s < 1.4%

LEC data rate 1.788M, HEC data rate 4.3k

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13

9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20

11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27

All seems normal.

Screenshots attached.

07:10 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      Merger, TapeServer etc - attachment 5

      per p+n FEE64 1.8.L spectra - attachment 6
       aida09 pulser peak width 68 ch FWHM 
07.13  peak area 25253 total area 27548 in 221mins => 1.9-2.1Hz => deadtime < 5% or less which is consistent with data file analysis below

      analysis data file S181/R6_274 - attachment 
       max FEE64 deadtime 24% aida08
       HEC data rate 4.5kHz, LEC data rate 2.284M


14.00 DAQ crashed by accident
      FEE64s power cycled 
      DAQ restarted
      data file S181/R7

      DSSSD bias & leakage current OK - attachments 8-9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11

      WR timestamps OK - attachment 12

      All system wide checks *except* as reported attachments 13-15

      per FEE64 Rate spectra - attachment 16

      per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 66 ch FWHM

      per FEE64 1.8.H spectra - attachments 18-19

      per FEE64 1.8.W spectra - 20us FSR - attachment 20-21

      Merger, TapeServer etc - attachments 22-23
       data file S181/R7_16

      analysis data file S181/R7_16
       max deadtime 7.3% aida08, all other FEE64s < 2%
       LEC data rate 1.877M HEC data rate 4.7k

00.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 3

      ADC data items stats OK - attachment 2
   
      per FEE64 Rate spectra - attachment 7

      per p+n FEE64 1.8.L spectra - attachments 6
       aida09 pulser peak width 63 ch FWHM 

      Merger, TapeServer etc - attachments 4-5

      analysis data file S181/R6_274 - attachment 8
       max FEE64 deadtime 19% aida08
       HEC data rate 3.8kHz, LEC data rate 2.258M

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

03.12 Merger, TapeServer etc - attachments 11-12
       before & after Merger restart

      per FEE64 Rate spectra - attachment 13

      ADC data items stats OK - attachment 14
       aida02 zero rate

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage current OK - attachment 16

      analysis data file S181/R6_372 - attachment 17
       max FEE64 deadtime 22% aida08
       HEC data rate 4.4kHz, LEC data rate 2.295M

03.32 all histograms zero'd

03.57 per p+n FEE64 1.8.L spectra - attachment 18
       25 mins = 1500 @ 2Hz = 3000 cf. peak area c. 3000 => deadtime low

Per NH

"From ELOG (assuming times accurateish)

10:00 Source in (AIDA removed from DESPEC DAQ)
11:30: Source out (?)
11:45: Background Runs
13:00: bPlast thresholds changed
15:45: Beam Returned"
-
                  ADC data item rate
EOF time  file    LEC        HEC
09.02    R4_538   0.906M     1.7k
10.02      _551   0.927M     1.7k
11.00      _564   0.875M     1.7k
11.19      _568   0.872M     1.7k
12.18    R6_7     1.121M     1.9k
13.03     6_19    1.134M     1.9k
14.02     6_35    1.197M     2.0k
15.00     6_52    1.232M     2.0k
15.10     6_54    1.225M     2.0k
15.36     6_56    1.467M     2.0k
16.00     6_65    1.630M     3.0k
17.01     6_89    1.778M     3.7k
21.43     6_207   1.759M     2.7k
12.6.24
00.09     6_274   2.258M     3.8k
03.12     6_372   2.295M     4.4k
07.13     6_506   2.284M     4.5k
11.12     6_611   1.853M     3.6k
14.55     7_16    1.877M     4.7k
13.6.24
07.01     7_457   1.788M     4.3k
12.34     7_610   2.000M     4.7k
14.00     7_649   1.756M     2.5k
14.12     7_654   1.817M     2.5k
15.33     7_693   1.907M     3.5k
14.6.24
07.23     8_35    1.677M     3.7k
10.54     8_122   1.880M     4.7k
14.38     8_213   1.935M     3.6k
15.6.24
01.25     9_27    0.951k     1.6k  (beam off from c. 18.15 14.6.24)
08.56     9_152   1.215M     2.1k  gamma source in use?
15.27     9_275   1.520M     2.3k  370kBq 152Eu source in use
16.6.24
15.06     9_747   1.667M     2.3k

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 

All looks good.

Sceenshots attached.

All seems OK

Screenshots attached.

17.33 per FEE64 1.8.W spectra - attachments 8-9
      per p+n FEE64 1.8.L spectra - attachment 10
       aida09 pulser peak width 66 ch FWHM
       cf 
       c. 11:00 today 57 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/650
       cf
       c. 00:10 today 53 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/649

17.48 ADC data item stats - attachment 11
      All histograms zero'd

17.51 Analysis data file S181/R6_96 - attachment 12
      max deadtime 5% (aida08)
      HEC Data items 3.7kHz, LEC data items 1.8MHz
      cf.
      c. 12.29 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/650
      c. 00.09 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/649

10.57 DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2

      ADC, MBS correlation scaler data item stats - attachments 3-4
       ADC 9x < 20k, max 243k aida08

      ASIC settings aida01 & aida02 - attachments 5-6

      ALL system wide checks OK *except* attachments 7-8

      WR timestamps OK - attachment 9

      BNC PB-5 settings - attachment 10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L spectra - attachment 12
       aida09 pulser peak width 57 ch FWHM

      per FEE64 1.8.H spectra - attachments 13-14

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16
       no p+n FEE64 wavefroms - reason unknown

      Merger, TapeServer etc - attachments 17-18

11.25 Lost control of aida-gsi window system 
      All command terminals lost, could not start new command terminals

      aida-gsi restart

      FEE64 power cycle
      DSSSD bias cycle

      Note file manager, calculator apps now working as expected

12.11 DAQ starts 
      data file S181/R6

      ASIC settings 2024Apr19-15.22.49

      BNC PB-5 Pulser
      Amplitude 10.0V
      Attenuation x10
      Fall time 1ms
      Frequency 2Hz
      Polarity + (connected to test + signal daisy chain to all p+n FEE64s)

      Sync ASIC clocks OK
      All system wide checks OK *except* aida02 FPGA/WR decoder - attachments 19-20

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-22

12.29 analysis data file S181/R6_9 - attachment 23
      max dead time c. 2% (aida08)

15:34 For some reason the AIDA MBS wasn't connecting to the time sorter (or a check stream) [NH]
      Try to reset the AIDA MBS 
      Due to confusion the AIDA Merger was rebooted too, while the DAQ was running, this broke all the FEEs -> Powecycled
      Synced ASIC clocks, fixed all waveforms
      Masked all discriminators
      All looks ok -> Sending back to MBS and back in DESPEC DAQ

00.09 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      per p+n FEE64 1.8.L spectra - attachments 5-6
       aida09 pulser peak width 53 ch FWHM 

      per FEE64 1.8.H spectra - attachments 7-8

      per n+n FEE64 1.8.W spectra - 20us FSR - attachment 9
       p+n 1.8.W not running currently

      Merger, TapeServer etc - attachments 10-11

      analysis data file S181/R4_427 - attachment 12
       max FEE64 deadtime 1.7%
       HEC data rate 3.1kHz

00.28 all histograms zero'd


02.39 DAQ continues data file S181/R4_460
      Beam off since 02.08

      DSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data items stats OK - attachment 15
       9x < 20k, max 239k aida08
   
      per FEE64 Rate spectra - attachment 16


07.15 DAQ continues data file S181/R4_517
      Beam off since 02.08
      HKR report c 3-4 h until UNILAC fixed

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18

      ADC data items stats OK - attachment 19
       8x < 20k, max 238k aida08
   
      per FEE64 Rate spectra - attachment 20

Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

8am checks

temps ok - attachment 2

rates ok - attachment 8

voltage ok - attachment 5

merger ok - attachment  6

9am checks

temps ok - attachment 10

rates ok - attachment 9

voltage ok - attachment 12

merger ok - attachment  11

(Ignore  Grafana plots 13-43 inc - not refreshed)

9.16 beam stopped to optimise beam intensity

10.00 checks - still no beam

temps ok - attachment 17

rates ok - attachment 15+16

voltage ok - attachment 14

merger ok - attachment  18

11.00 checks - still no beam

temps ok - attachment 23

rates ok - attachment 21+22

voltage ok - attachment 20

merger ok - attachment  24

12pm checks

temps ok - attachment 28

rates ok - attachment 26+27

voltage ok - attachment 29

merger ok - attachment  30

12.45 - beam back

temps ok - attachment 34

rates ok - attachment 32+33

voltage ok - attachment 35

merger ok - attachment  36

2pm checks

temps ok - attachment 40

rates ok - attachment 38+39

voltage ok - attachment 41

merger ok - attachment  42

/tmp/R4_276 added - attachment 44

max deadtime 10.5% (aida08)

n+n FEE64s deadtime 

3pm checks

temps ok - attachment 48

rates ok - attachment 46+47

voltage ok - attachment 45+50

merger ok - attachment  49

23.47 DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 4-5 & 9

      WR timestamp - attachment 7

      all system wide checks OK *except* attachments 6 & 10

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-13

      Merger, TapeServer etc - attachments 14-15
       data file S181/R4, TapeServer No Storage mode

00.00 Beam ON 3s spill on, 2s spill off
      99%+ fission fragments
      ADC data item stats - attachment 16


02.03 Data storage enabled
 
      per FEE64 1.8.H spectra - attachments 17-18

02.35 analysis data file S181/R4_28 - attachment 19
      data file S181/R4
      max deadtime aida08 18%
      n+n FEE64s 12-18%, all p+n FEE64s < 2%

02.41 Noted that there is disc data written to disk - all disc data now disabled for all FEE64s

04.08 DSSSD bias & leakage current OK - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Rate spectra - attachment 23

04.15 aida02 no data
      restarted merger per https://elog.ph.ed.ac.uk/DESPEC/644

06.00 DSSSD bias & leakage current OK - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27

AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.

17:00

DSSSD bias & leakage current - attachment 2

FEE64 temperatures OK - attachment 3

Statistics  attachment 4

17:39

Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.

23:19 Flange removed. Starting to take beam.

1) Select 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode please ignore this step

2) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

3) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

4) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

5) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

6) Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

8) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

9) Unselect 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode at the start of this procedure please ignore this step


   If you continue to have problems, please request expert assistance.

23.58 8.6.24

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

03.43 slow comparator 0x64

      DSSSD bias & leakage current OK - attachments 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 1.8.H spectra - attachments 8-9

      per FEE64 Rate spectra - attachment 10


09.11 DSSSD bias & leakage current OK - attachments 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats - attachment 13

      per FEE64 Rate spectra - attachment 14


15.05 online spectra
      per DSSSD HEC E_p versus E_n and x versus y
      observe flat field of c. 500MeV at per spull rate c. 5Hz - fission fragments?

14.09 aida02 WR stall
      DAQ STOP
      restart Merger, Mrger SETUP, Merger GO
      DAQ GO

      all histograms & stats zero'd

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 4

      WR timestamp before/after Merger restart - attachments 3 & 5

      all system wide checks OK *except* attachments 6-7

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 8-10
       ADC data items rates high - source installed?

      per FEE64 Rate spectra - attachments 11-13

      per FEE64 1.8.W spectra - 20us FSR - attachments 14-15

      per p+n FEE64 1.8.L spectra - attachment 16
       aida09 pulser peak width 60 ch FWHM

      Merger, TapeServer etc - attachments 17-19
       data file S181/R2_249 switch TapeServer to No Storage mode

      per p+n FEE64 1.8.W spectra - 20us FSR - attachment 20

20.33 DSSSD bias & leakage current OK - attachments 21

      FEE64 temperatures OK - attachment 22

      ADC data item stats - attachment 23

      per FEE64 Rate spectra - attachment 24

05.01 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      9x < 20k, max 228k aida08

      per FEE64 Rate spectra - attachment 4


08.18 DSSSD bias & leakage current - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 8


14.37 DSSSD bias & leakage current - attachment 9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 12

      Writing to disk file S181/R1 (continuing alpha background file)

14.38 End run S181/R1 and start new run file S181/R2

17.09 08.18 DSSSD bias & leakage current - attachment 13

      FEE64 temperatures OK - attachment 14

      MBS correlation scaler data - attachment 15
       data to aida01, aida02, aida03 and aida10

      ADC data item stats - attachment 16
      9x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 17

      Merger/TapeServer etc - attachments 18-19 

12.09 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      8x < 20k, max 247k aida08

      per FEE64 Rate spectra - attachment 4

12.16 all histograms & stats zero'd


21.08 DSSSD bias & leakage current - attachments 5-6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8
      8x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 9


      

We are trying to bias each system on its own and in sequence biasing each other detector system to see the interdependence.
 

HV OK. TEMP OK. Attachment 1 &2.

9.43 bPlast unbiased, BB7 unbiased. Attachments 3-6 is the noise situation with only AIDA biased (bPlast was biased but turned off after AIDA was biased.)

10/16 FEEs < 20k noise. Max noise is 250k. Downstream AIDA appears to be seeing additional noise since S100.

11:21 Returning after an hour the noise has increased significantly in many channels. Scale changed to 10000 in histograms, it appears that the problem is from a couple hot channels.

11:41 'Power' cycled AIDA turned off. bPlast turned on, on its own, then BB7. bPlast doesn't seem to notice anything.

12:21 bPlast and BB7 on. AIDA ON. Rates and histograms given by attachment 7-8. More FEEs noisy, but same level as 11:21. There appears to be the same noisy channel in all of the FEEs (?) Some grounding not properly accounted for, or a wire touching somewhere (?)

13:55h: AIDA & BB7 on. Rates and histograms given in attachments 9-10. Same level as bplast & AIDA on. Same noisy channels in FEEs. According to Nic, these correspond to the end of the ribbon cables & is normal.

13:59 AIDA, bPlast & BB7 on before lunch. Same levels of noise as in previous combinations (attachments 11-12).

So it seems we can run with the three sub-systems biased simultaneously.

DEGAS Array over Snout again

Situation similar to yesterday, most p+n rates good

15:40 reseated FEE64s adaptors of aida01, aida09, aida13 & aida16.

TEMP OK. HV OK. System wide checks all passed excl. aida07 ADC calibration, see attached.

Histograms and rates look same as previous, reseating aida01 seems to have worked. aida16 still unhappy.

re-reseated aida16. 16:19 - appears to have fixed the problem in the channel.

Pulser test + connected.

bPlast ground removed from frame.

To-do:

- Need clarification about links connected on AIDA n+n FEE64 adaptor boards. Currently, the situation is given by the attachment 8 & 9.

- Fix BB7 grounding, continuity between snout and frame observed.

- Pulser walkthrough for n+n, can do p+n when convienient.

18.30 Noted FEE64 ADC stats now significantly higher ( attachments 9-10 ) cf. earlier this afternoon (attachments 1-2)

DSSSD bias OFF. FEE64 power OFF.

08.28 DSSSD bias & leakage current - attachments 1-2

      FEE64 temps OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Stat spectra - attachment 5

      per FEE64 1.8.L spectra - attachment 6

      per FEE64 1.8.W spectra - 0us FSR - attachments 7-8

08.39 DAQ stops file S181/R1_36

      FEE64 power OFF
      DSSSD bias ON

11.12 DSSSD bias OFF

02.41 DSSSD bias & leakage current - attachment 1

      FEE64 temps OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Stat spectra - attachment 4


06.52 DSSSD bias & leakage current - attachment 5

      FEE64 temps OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 Stat spectra - attachment 8

12.13 DSSSD bias & leakage current - attachment 9

      FEE64 temps OK - attachment 10

      ADC data item stats - attachment 11

      per FEE64 Stat spectra - attachment 12


14.51 DSSSD bias & leakage current - attachments 13-14

      FEE64 temps OK - attachment 15

      ADC data item stats - attachment 16

      per FEE64 Stat spectra - attachment 17

21.49 DSSSD bias & leakage current - attachment 18

      FEE64 temps OK - attachment 19

      ADC data item stats - attachment 20

      per FEE64 Stat spectra - attachment 21


03.55 DSSSD bias & leakage current - attachment 22

      FEE64 temps OK - attachment 23

      ADC data item stats - attachment 24

      per FEE64 Stat spectra - attachment 25

13.26 DSSSD bias & leakage current - attachments 1-3

      FEE64 temps OK - attachment 4
       aida02 asic sensor u/s

      All system wide checks OK *except* WR decoder status - attachment 5

      ADC data item stats - attachment 6

      per FEE64 Rate spectra - attachments 7-8

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

      WR timestamps OK - attachment 11

13.59 Data file S181/R1
      alpha background
      BNC PB-5 disconnected?
      slow comparator 0x64

      ADC data item stats - attachment 12
      Merger, TapeServer etc - attachments 13-14

14.02 All histograms & stats zero'd



16.19 DSSSD bias & leakage current - attachment 15

      FEE64 temps OK - attachment 16

      ADC data item stats - attachment 17

      per FEE64 Stat spectra - attachment 18


19.16 DSSSD bias & leakage current - attachments 19-20

      FEE64 temps OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Stat spectra - attachment 23

      2x Check ASIC Control


21.42 DSSSD bias & leakage current - attachment 24

      FEE64 temps OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Stat spectra - attachment 27

00.48 DSSSD bias & leakage current - attachment 28

      FEE64 temps OK - attachment 29

      ADC data item stats - attachment 30

      per FEE64 Stat spectra - attachment 31

17.50 bPlas cabling, grounds etc connected, power OFF
       CC reports that bPlas power is actually ON

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ASIC settings 2024Apr26-01.29.53
       p+n FFEE64 slow comparator 0xa, n+n Ohmic FEE64 0xf

      System wide checks OK *except* aida02 WR decoder status

      ADC data item stats - attachment 3
       9/16 < 20k, max 234k (aida08)

      per FEE64 Rate spectra - attachments 4-6

      per FEE64 1.8.W spectra - 20us FSR - attachments 7-8

      WR timestamps OK - attachment 9


18.50 FEE64 power OFF

AIDA and the Snout have been remounted in S4

I had a check of the AIDA connectors and all looked good, the kapton PCB connectors were sitting comfortably above the interstage and I saw no stressed/loose connectors.
Multimeter showed no continuity between snout or inter-stage and AIDA ground

We put a single layer of wide black tape inside the end-cap to improve the light seal and minimise reflections

After mounting the snout I managed to bias both DSSDs to 100 V with no issues. I had to raise the DSSD#1 (Ch0) limit to 40 uA because it rises quite high during ramp up (then settles).

It is very hot in S4 (28 C) and the roof is closed for high-radiation experiments, so I expect it to get hotter. DSSD leakage currents were similar to before (20 and 15 uA)

There is also the new Temp and Humidity monitor, which you can view on AnyDesk at 192.168.11.119 (I think), there is a firefox window open already 

[npg@aidas-gsi MIDAS]$ ls -l
total 524
drwxrwxr-x.  2 npg npg    156 Mar 17  2021 Aida
drwxrwxr-x.  2 npg npg   4096 Jul 13  2015 bin_Linux64
drwxrwxr-x.  4 npg npg     38 Jul  3  2018 config
drwxr-xr-x.  3 npg npg     22 Jan 23  2019 config_ORIG
drwxr-xr-x.  3 npg npg     17 Jan 23  2019 Data_Acq
drwxrwxr-x. 25 npg npg   4096 Aug 19  2020 DataPackage
drwxrwxr-x.  3 npg npg     25 Aug 29  2019 DB
drwxr-xr-x.  3 npg npg     25 Jan 23  2019 DB_ORIG
lrwxrwxrwx.  1 npg npg     14 May 30  2019 Embedded -> /home/Embedded
drwxrwxr-x.  2 npg npg   4096 Mar 17  2021 ISOL_Aida
drwxrwxr-x.  2 npg npg   4096 Nov 29  2018 ISOL_Aida_PreApr9
drwxrwxr-x.  7 npg npg     87 Jul  8  2009 lib
drwxrwxr-x.  2 npg npg   4096 Sep 19  2012 lib_Linux64
drwxr-xr-x.  5 npg npg     47 Jan 23  2019 Linux
drwxrwxr-x.  6 npg npg     58 Feb 22  2018 linux-ppc_4xx
drwxr-xr-x.  6 npg npg     58 Jan 23  2019 linux-ppc_4xx_ORIG
drwxrwxr-x.  3 npg npg     73 Feb 16  2022 log
-rwxrw-r--.  1 npg npg  12862 Jul  9  2009 midasicon.ico
-rwxrw-r--.  1 npg npg    773 Jul 23  2012 midasicon.png
-rwxrwxr-x.  1 npg npg    772 Jan 20  2019 MIDASNewGen_BuildAIDA.txt
-rwxrwxr--.  1 npg npg   2532 Jan 19  2019 MIDASNewGen_BuildCore.txt
drwxrwxr-x. 14 npg npg   4096 Feb 22  2018 MIDASpackages
drwxrwxr-x.  6 npg npg    124 May 14  2022 NewMerger
-rw-rw-r--.  1 npg npg 118873 Oct 18  2019 NewMergerdiff.log
-rw-rw-r--.  1 npg npg 118971 Oct 18  2019 NewMergerdiff.log~
drwxrwxr-x.  5 npg npg     52 Jan  8  2019 NewMerger_PreApr9
drwxrwxr-x.  6 npg npg    124 Mar 12  2020 NewMerger_PreMar21
drwxrwxr-x.  7 npg npg    104 Oct 17  2019 NewMerger_PreOct19
-rw-r--r--.  1 npg npg 230517 Mar 12  2020 NewMerger.tgz
drwxr-xr-x. 14 npg npg   4096 Jan 23  2019 OLD
drwxr-xr-x.  3 npg npg     17 Jan 23  2019 SOAP
drwxrwxrwx.  4 npg npg     67 Aug 19  2021 SpecData
drwxrwxr-x.  3 npg npg     21 Nov 15  2019 TapeServer
drwxr-xr-x.  3 npg npg     21 Jan 23  2019 TapeServer_PreFeb21
drwxrwxr-x.  3 npg npg     21 Nov 15  2019 TapeServer_PreMar21
drwxr-xr-x.  2 npg npg     44 Feb  2  2021 tcl
drwxr-xr-x. 12 npg npg    152 Jan 23  2019 TclHttpd
drwxrwxr-x. 11 npg npg    155 Oct 26  2017 TclTkpackages
lrwxrwxrwx.  1 npg npg     26 May 30  2019 XilinxLinux -> /home/Embedded/XilinxLinux

[npg@aidas-gsi NewMerger]$ ls -lR
.:
total 32
-rw-rw-r--. 1 npg npg 15096 Apr 16  2019 Aida_ASICreadout.c
drwxrwxr-x. 3 npg npg    23 Oct 18  2019 Html
drwxrwxr-x. 2 npg npg   172 May 14  2022 include
drwxrwxr-x. 8 npg npg   256 May 14  2022 MergeServer
-rw-rw-r--. 1 npg npg 11048 Aug 29  2019 R3B_readout.c
-rw-rw-r--. 1 npg npg   152 Oct 17  2019 README
drwxrwxr-x. 2 npg npg    34 Apr 29  2021 tcl

./Html:
total 0
drwxrwxr-x. 6 npg npg 187 Oct 13  2020 NewMerger

./Html/NewMerger:
total 60
lrwxrwxrwx. 1 npg npg    21 Oct 18  2019 Html -> /MIDAS/NewMerger/Html
drwxrwxr-x. 2 npg npg     6 Aug 26  2019 LinkRates
-rw-rw-r--. 1 npg npg  1210 Aug 26  2019 NewMerger.js
-rw-rw-r--. 1 npg npg  9417 Aug 29  2019 NewMerger.tcl
-rw-rw-r--. 1 npg npg  9332 Aug 26  2019 NewMerger.tcl~
-rw-rw-r--. 1 npg npg 13052 Aug 27  2019 NewMerger.tml
-rw-rw-r--. 1 npg npg 13851 Aug 26  2019 NewMerger.tml~
drwxrwxr-x. 2 npg npg    82 Oct 13  2020 Options
drwxrwxr-x. 2 npg npg   102 Oct 13  2020 RunControl
drwxrwxr-x. 2 npg npg   192 Aug 29  2019 Stats

./Html/NewMerger/LinkRates:
total 0

./Html/NewMerger/Options:
total 24
-rwxrwxr-x. 1 npg npg  725 Aug 26  2019 Options.js
-rwxrwxr-x. 1 npg npg 3934 Aug 26  2019 Options.tcl
-rwxrwxr-x. 1 npg npg 7864 Aug 27  2019 Options.tml
-rw-rw-r--. 1 npg npg 7864 Aug 27  2019 Options.tml~

./Html/NewMerger/RunControl:
total 44
-rw-rw-r--. 1 npg npg  4791 Aug 26  2019 implementation.tcl
-rw-rw-r--. 1 npg npg  1643 Aug 26  2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg   539 Aug 26  2019 stats.tcl
-rw-rw-r--. 1 npg npg 12328 Aug 29  2019 sys.tcl
-rw-rw-r--. 1 npg npg 12231 Aug 26  2019 sys.tcl~

./Html/NewMerger/Stats:
total 52
-rw-rw-r--. 1 npg npg 2880 Aug 26  2019 MergeStatsALL.tcl
-rw-rw-r--. 1 npg npg 1013 Aug 26  2019 MergeStats.js
-rw-rw-r--. 1 npg npg 1534 Aug 26  2019 MergeStatsSINGLE.tcl
-rw-rw-r--. 1 npg npg 4259 Aug 26  2019 MergeStats.tcl
-rw-rw-r--. 1 npg npg 8467 Aug 27  2019 MergeStats.tml
-rw-rw-r--. 1 npg npg 8338 Aug 26  2019 MergeStats.tml~
-rw-rw-r--. 1 npg npg 1829 Aug 29  2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg 1943 Aug 29  2019 stats.defn.tcl~

./include:
total 52
-rw-rw-r--. 1 npg npg 1079 Apr 16  2019 buffer.h
-rw-rw-r--. 1 npg npg 4473 Apr 16  2019 common.h
-rw-rw-r--. 1 npg npg 1199 Jul 16  2019 merge.h
-rw-rw-r--. 1 npg npg 1273 Apr 16  2019 message.h
-rw-rw-r--. 1 npg npg 1273 Oct 17  2019 message.h~
-rw-rw-r--. 1 npg npg 5288 Aug 29  2019 netvar.h
-rw-rw-r--. 1 npg npg 5256 Apr 16  2019 netvar.h~
-rw-rw-r--. 1 npg npg  912 Apr 16  2019 services.h
-rw-rw-r--. 1 npg npg  886 Apr 18  2019 stats.h
-rw-rw-r--. 1 npg npg 2710 Apr 16  2019 transfer.h

./MergeServer:
total 60
-rw-rw-r--. 1 npg npg  2606 Apr 16  2019 alloc_data_area.c
drwxr-xr-x. 2 npg npg    69 Aug 27  2019 bin64
drwxrwxr-x. 2 npg npg   128 Apr  6  2021 bin64.BackUp
-rw-rw-r--. 1 npg npg 10510 Sep 19  2019 buffer.c
-rwxrwxr-x. 1 npg npg   305 Apr 16  2019 build
drwxrwxr-x. 3 npg npg   279 May 14  2022 link
-rw-rw-r--. 1 npg npg   494 Apr 16  2019 Makefile
drwxrwxr-x. 3 npg npg   160 Apr  6  2021 master
drwxrwxr-x. 3 npg npg   143 Apr  6  2021 merge
-rw-rw-r--. 1 npg npg  3225 Sep 19  2019 message.c
-rw-rw-r--. 1 npg npg  3199 Sep 19  2019 netvar.c
-rwxrwxr-x. 1 npg npg   397 Aug 26  2019 run
-rw-rw-r--. 1 npg npg 15309 Sep 19  2019 signal.c
drwxrwxr-x. 3 npg npg   101 Sep 19  2019 statrate
-rw-rw-r--. 1 npg npg  1548 Sep 19  2019 stats.c
-rw-rw-r--. 1 npg npg   593 Apr 16  2019 version.c

./MergeServer/bin64:
total 148
-rwxr-xr-x. 1 npg npg 51964 May 11  2021 link64
-rwxr-xr-x. 1 npg npg 33312 May 11  2021 master64
-rwxr-xr-x. 1 npg npg 36027 May 11  2021 merge64
-rwxr-xr-x. 1 npg npg 21370 May 11  2021 statrate64

./MergeServer/bin64.BackUp:
total 256
-rwxr-xr-x. 1 npg npg 51948 Mar 16  2021 link64
-rwxrwxr-x. 1 npg npg 33184 Apr  6  2021 master64
-rwxr-xr-x. 1 npg npg 33312 Mar 16  2021 master64.new
-rwxr-xr-x. 1 npg npg 36027 Mar 16  2021 merge64
-rwxr-xr-x. 1 npg npg 35368 Mar  9  2021 merge64.new
-rwxrwxr-x. 1 npg npg 35910 Mar  8  2021 merge64.orig
-rwxr-xr-x. 1 npg npg 21370 Mar 16  2021 statrate64

./MergeServer/link:
total 164
-rw-rw-r--. 1 npg npg 10216 Oct 18  2019 handle_buffer.c
-rw-rw-r--. 1 npg npg 24623 Sep 19  2019 link_server.c
-rw-rw-r--. 1 npg npg 24732 Oct 18  2019 link_server.c~
drwxrwxr-x. 2 npg npg   217 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg  7051 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg  7029 Oct 18  2019 main.c~
-rw-rw-r--. 1 npg npg 14464 Sep 19  2019 processAIDADATAbuffer.c
-rw-rw-r--. 1 npg npg 14463 Oct 18  2019 processAIDADATAbuffer.c~
-rw-rw-r--. 1 npg npg 14950 Sep 19  2019 processR3BDATAbuffer.c
-rw-rw-r--. 1 npg npg 14925 Oct 18  2019 processR3BDATAbuffer.c~
-rw-rw-r--. 1 npg npg 15077 Aug 22  2019 processR3BDATAbuffer.c_PreAug22

./MergeServer/link/Linux64:
total 176
-rw-rw-r--. 1 npg npg  3088 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  6216 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 29840 Oct 24  2019 handle_buffer.o
-rwxrwxr-x. 1 npg npg 50660 Oct 24  2019 link64
-rw-rw-r--. 1 npg npg 24464 Oct 24  2019 link_server.o
-rw-rw-r--. 1 npg npg 14864 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   795 Aug 22  2019 Makefile
-rw-rw-r--. 1 npg npg   776 Apr 16  2019 Makefile~
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 12848 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/master:
total 40
-rw-rw-r--. 1 npg npg 7850 Sep 18  2019 handle_signals.c
drwxrwxr-x. 2 npg npg  223 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 6952 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 6973 Sep 19  2019 main.c~
-rw-rw-r--. 1 npg npg 1722 Aug 29  2019 start_link.c
-rw-rw-r--. 1 npg npg 1885 Aug 22  2019 start_link.c~
-rw-rw-r--. 1 npg npg 1372 Apr 16  2019 start_merge.c
-rw-rw-r--. 1 npg npg 1345 Apr 16  2019 start_statrate.c

./MergeServer/master/Linux64:
total 120
-rw-rw-r--. 1 npg npg  3408 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  4168 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 22840 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   793 Jul 16  2019 Makefile
-rwxrwxr-x. 1 npg npg 33184 Oct 24  2019 master64
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 14616 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3408 Oct 24  2019 start_link.o
-rw-rw-r--. 1 npg npg  2904 Oct 24  2019 start_merge.o
-rw-rw-r--. 1 npg npg  2808 Oct 24  2019 start_statrate.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/merge:
total 48
drwxrwxr-x. 2 npg npg  228 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 5130 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 4579 Oct 24  2019 mergeAIDA.c
-rw-rw-r--. 1 npg npg 4119 Oct 17  2019 #merge.c#
-rw-rw-r--. 1 npg npg 4148 Oct 21  2019 merge.c
-rw-rw-r--. 1 npg npg 3703 Sep 19  2019 mergeR3B.c
-rw-rw-r--. 1 npg npg 8060 Sep 17  2019 merge_server.c
-rw-rw-r--. 1 npg npg 2394 Sep 17  2019 storage.c

./MergeServer/merge/Linux64:
total 120
-rw-rw-r--. 1 npg npg  3088 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  3872 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 10944 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   789 Jul 16  2019 Makefile
-rw-rw-r--. 1 npg npg   776 Apr 16  2019 Makefile~
-rwxrwxr-x. 1 npg npg 35910 Oct 24  2019 merge64
-rw-rw-r--. 1 npg npg  9064 Oct 24  2019 merge.o
-rw-rw-r--. 1 npg npg  8160 Oct 24  2019 merge_server.o
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 12744 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  4048 Oct 24  2019 storage.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/statrate:
total 16
drwxrwxr-x. 2 npg npg  161 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 1876 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 1717 Sep 19  2019 main.c~
-rw-rw-r--. 1 npg npg 2645 Apr 16  2019 statrate_server.c
-rw-rw-r--. 1 npg npg 2645 Apr 16  2019 statrate_server.c~

./MergeServer/statrate/Linux64:
total 80
-rw-rw-r--. 1 npg npg  8880 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   725 Apr 16  2019 Makefile
-rw-rw-r--. 1 npg npg   728 Apr 16  2019 Makefile~
-rw-rw-r--. 1 npg npg  5376 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 10144 Oct 24  2019 signal.o
-rwxrwxr-x. 1 npg npg 21370 Oct 24  2019 statrate64
-rw-rw-r--. 1 npg npg  4664 Oct 24  2019 statrate_server.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1872 Oct 24  2019 version.o

./tcl:
total 4
-rw-rw-r--. 1 npg npg 1775 Aug 26  2019 NewMergerControl.tcl

Status before dismount

14:26 DSSSD#1 119.99 V @ 17.968 uA

          DSSSD#2 119.95 V @ 11.212 uA

          Leakage currents slightly elevated from Monday.

S4 temperature at 26 °C.

14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.

Plan:

- Dismount snout.

- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.

08.11 DSSSD#1 bias -120V leakage current -17.0uA
      DSSSD#2 bias -120V leakage current -10.5uA
      S4 temperature 25 deg C

      cf. 04.00 30.4.24 S4 temperature 25 deg C

      DSSSD#1 -21.9uA
      DSSSD#2 -13.0uA

      Leakage currents currently c. 80% of 30.4.24 values at same temperature 

11.22 DSSSD#1 bias -120V leakage current -17.1uA
      DSSSD#2 bias -120V leakage current -10.6uA
      S4 temperature 25 deg C

      cf. 07.00 29.4.24 S4 temperature 25 deg C

      DSSSD#1 -21.8uA
      DSSSD#2 -12.8uA

      Leakage currents currently c. 80% of 29.4.24 values at same temperature 

08.32 DSSSD bias & leakage currents - Grafana - attachments 1-2
      FEE64 temperatures OK - attachment 3
      ADC, correlation scalers data item stats OK attachments 4-5
           11/16 <20k max aida16 174k
      per FEE64 Rate spectra - attachmnt 6


09.44 per p+n FEE64 1.8.L spectra - attachment 7
       aida09 pulser peak width 58 ch FWHM
      per FEE64 1.8.W spectra - attachments 8-10

09.47 bias cycle - off and then on
      no significant change in leakage current

09.57 FEE64 power OFF
      DSSSD bias ON

00.05 FEE64 temperatures OK
      ADC data item stats OK

09.31 FEE64 temperatures OK
      ADC data item stats OK

13.15 FEE64 temperatures OK
      ADC data item stats OK
       9/16 <20k max aida16 164k

20.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida03 no data - DAQ reset
       9/16 <20k max aida16 158k

23.15 FEE64 temperatures OK
      ADC data item stats
       aida03 no data - DAQ reset
       10/16 <20k max aida16 153k

01.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 no data
       6/16 < 20k


10.25 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 restarted
       7/16 < 20k, max 220k aida16

      all histograms zero'd

13.20 per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 59 ch FWHM
      FEE64 temperatures OK
      ADC data item stats
       aida04 no data
       5/16 < 20k, max 332k aida08 

14.37
  Moved Time Machine delayed scaler from aida04 to aida07
  aida04 no data
  Reset merger etc 
  aida05 no data
  Reset merger etc 
  All FEEs with data again

  S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C

17.26 FEE64 temperatures OK
      ADC data item stats OK

19.28 FEE64 temperatures OK
      ADC data item stats 
       aida02 no data - DAQ reset
      

04.55 FEE64 temperatures OK
      ADC data item stats OK *except* aida04 no data
      TapeServer no storage mode

09.55 FEE64 temperatures OK
      ADC data item stats OK *except* aida02 & aida04 no data
      DSSSD bias and leakage current - Grafana - attachment 1

19.28 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 no data
       noted increase in rates for aid08 and aida16
       5/16 < 20k

10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4
       9/16 FEE64s < 20k 

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachments 6-7
       aida02, aida04 inactive

11.20 DAQ restarted
      aida02 and aida04 now producing data


11.30 analysis data file R22_313
      all deadtimes <1%
      no timewarps
      HEC data item rate 560Hz (all from one channel of aida08 asic #3)
      aida03 & aida04 MBS rates c. 24kHz


14.32 FEE64 temperatures OK
      ADC data item stats OK
       all FEE64s producing data

14.43 all histograms & stats zero'd

[NH checks here]

21.26 aida04 stopped forwarding to merger
 the "quick fix" (stop merger, rerun) worked

MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability

System Wide Checks
 clocks PASSED
 ADC IGNORED
 WR:
		 Base 		Current 	Difference
aida02 fault 	 0xe4d8 : 	 0x65c0 : 	 -32536  
aida03 fault 	 0x2371 : 	 0x2380 : 	 15  
aida04 fault 	 0xa210 : 	 0xaf58 : 	 3400  
aida06 fault 	 0x3624 : 	 0x3626 : 	 2  
White Rabbit error counter test result: Passed 12, Failed 4

FPGA: 
	
			 Base 		Current 		Difference
aida02 fault 	 0xaa3b : 	 0xc455 : 	 6682  
aida04 fault 	 0x5a4 : 	 0x6ba : 	 278  
aida06 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1

Baselines reset

22.50 FEE64 temps ok
      merger ok
       all links active
      ADC data item stats ok

08.52 Beam off

09.11 New data file R22
      Monitor accumulated implanted activity in bPlas and DSSSDs

      ASIC settings 2024Apr26-01.29.53

      DSSSD bias & leakage current OK - attachments 1-3

      FEE64 temperatures OK - attachment 4

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 6

      Merger etc - attachment 7

      analysis data file R22_19 - attachment 8
       all deadtimes 0.3%, or less
       no timewarps
       HEC data item rate 390Hz

   
10.00 all histograms & stats zero'd


13.24 DSSSD bias & leakage current OK - attachments 9-10

      FEE64 temperatures OK - attachment 11

      ADC data item stats - attachment 12

      per FEE64 Rate spectra - attachment 13

      per p+n FEE64 1.8.L spectra - attachments 14-16
       common x/y axes - peak height proxy for width
       aida09 pulser peak width 58 ch FWHM (~41keV FWHM)  cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
       electronic noise today very similar to electronic noise Monday 22 April

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-19

      Merger etc - attachment 20

      analysis data file R22_60 - attachment 21
       max deadtime 1.7% (aida11), all others <1% 
       no timewarps
       HEC data item rate 710Hz

14.13 ucesb - attachment 22

14.15 aida04 not producing data

14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618

15.27 *all* FEE64s stop producing data - reason unknown

16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618


16.29 DSSSD bias & leakage current OK - attachments 23-24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27

      Merger etc - attachment 28


20.45 DSSSD bias & leakage current OK - attachments 29-30

      FEE64 temperatures OK - attachment 32

      ADC data item stats - attachment 31

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

20.54  analysis data file R22_145 - attachment 35
       max deadtime 1.5% (aida11), all others <1% 
       no timewarps
       HEC data item rate 775Hz



03.45 DSSSD bias & leakage current OK - attachments 36-37

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41

Just another manic magic Monday, ....

Taking over Betool's AIDA shift. All is good.

This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.

01:15: AIDA04 is down again. DAQ shift crew informed.

01:27: AIDA04 is back and shift crew has been informed.

01:30 Checks:

     Bias and leakage current - Attachment 1-2

     Temperatures - Attachment 3

     ADC Data item - Attachment 4

     Rate Spectra - Attachment 5

     UCESB - Attachment  6

     Timestamp checks - Attachement 7

06:00 It has run very smoothly.   

21:30 screenshots

and

23:24 screenshots

19:24 pm screenshots

and

17:18 pm screenshots

0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5)   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

6) Start DAQ

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

8) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

9) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

10) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.

16.45 aida04 stopped producing data

      DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)

       aida04 did not stop, all other FEE64s stopped

      restart Merger (workspace 3, display 2, select 'NewMerger' icon)

      SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
      GO Merger (workspace 3, display 1, browser 'NewMerger' tab)

       aida04 now stops

      disable *all* #1 data transfer 

      DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
      check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)

      DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
      enable *all* #1 data transfer
      DAQ GO (workspace 2, display 1, browser 'Run Control' tab)


      No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
      

16:00 aida04 not producing data

          power cycle all FEE64s to recover DAQ

16.20 analysis data file R21_668

         max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%

         no timewarps

         HEC data item rate 1.9kHz

Took over the shift from Marc.

10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19

          Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD

10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).

         Grafana (AIDA) is now reporting data correctly too (attachment 21).

          Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )

                             Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.

                             Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.

14.45 analysis data file R21_500 - attachment 36

         max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%

         no timewarps

         HEC data item rate 2.7kHz

Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C

0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.

After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.

I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.

I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.

We are now back in business with AIDA back in main DAQ at  01:40

01:45 full checks:

           Bias: AIDA-Graphana - Attachment 1

               Leakage current ramping up again ! S4 temperaturenot really going down either: 25C

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5

      ucesb - attachment 6

03:00 full checks

          No change to be highlighted. All pretty much the same as during previous full checks.

05:00 full checks

Expt still running very smoothly.

           Bias: AIDA-Graphana - Attachment 7-8

               Leakage current going down slightly

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
      per FEE64 Rate spectra - attachment 11

      Merger etc - attachment 12

      ucesb - attachment 13

07:15 all running smoothly.

07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.

07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.

14:50 Checks

Leakage current  is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1

It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.

18:15 Checks

Everything appears to be going smoothly 

Grafana - DSSSD bias  and leakage current - attachments 2 and 3

FEE64 temperatures  - It appears okay, nothing strange , attachment 4

ADC data item stats - attachment 5

Per FEE64 Rate spectra - attachment 6

 ucesb - attachment 7

Merger Link Data Rates  - attachment 8

18:16

The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.

18:20  it came back to work normally

............................

20:00 Checks

Nothing new to report. All is well.

20:33 Aida02 showed as not reading - attachment 10

20:34 it resumed reading and returned to normal operation.

21:08 Aida02 was  not reading - attachment 11 , had to emailed Tom to fix it .

The shift crew have restarted ucesb and AIDA  is now showing OK - attachment 12

so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.

ADC data item stats and Merger etc  appear to be going smoothly - attachments 13 and 14

...........................................

22:00 Checks

Still running smoothly.

DSSD bias and  leakage current - ok

FEE64 temperatures - ok

Per FEE64 Rate spectra - ok

ADC data item stats - ok

Ucesb - ok

.................................

23:00 Checks

 Nothing new to report. Everything is going well , and I have captured screenshots of each one

Leakage currents dropped a bit See attachment 15

.............................

23:50 Checks

All seems smooth.

06.00 DAQ continues file S100_alpha/R20_740
      FRS setting 162Eu

      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage currents tracking S4 ambient temperature

      FEE64 temperatures OK - attachment 3

      All system wide checks OK *except* 
       as reported - attachments 4-5
       *all* ADCs not calibrated (not calibrated post-synchronise ASIC clocks at most recent restart)

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME, Correlation scaler data item stats OK - attachments 7-10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L specra - attachments 12-13
       aida09 pulser peak width 55 ch FWHM (~38keV FWHM)

      per FEE64 1.8.H spectra - attachments 14-15

      Merger, TapeServer etc - attachments 16-17
       12Mb/s to disk

      ucesb - attachment 18
       spill length 3.5s, extraction 1.5s

06.30 analysis data file R20_748 - attachment 19
      max deadtime 12%
      no timewarps
      HEC data item rate 2.1kHz

06.35 new run file S100_alpha/R21

09.30 DAQ continues - file S100_alpha/R21_83

      DSSSD bias & leakage currents - Grafana - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats OK - attachments 22

      per FEE64 Rate spectra - attachment 23

      Merger etc - attachments 24

      ucesb - attachment 25
       spill length 3.5s, extraction 1.5s

09.50 per FEE64 1.8.W spectra - 20us FSR - attachments 26-27
       no significant change in electronic noise cf. start of run

      per p+n FEE64 1.8.L spectra - attachment 28
       aida09 pulser peak width 62 ch FWHM
       common x/y scale - peak heoght proxy for width


13.57 DAQ continues file R21_141

      DSSSD bias & leakage current - Grafana - attachments 29-30

      FEE64 temperatures OK - attachment 31

      ADC data item stats OK - attachments 32

      per p+n FEE64 1.8.L spectra - attachment 33
       aida09 pulser peak width 55 ch FWHM (~39keV FWHM)

      per FEE64 Rate spectra - attachment 34

      Merger etc - attachments 35

      ucesb - attachment 36
       spill length 3.5s, extraction 1.5s

14.15 analysis data file R21_144 - attachment 35
      max deadtime 16% (aida04)
      no timewarps
      HEC data item rate 2.0kHz

It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!

0.15 Checks:
      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage current dropped a bit since the evening shift ~22:00.

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

      ucesb - attachment 7

2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12

3:30 Still running smoothly. 

5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.

        DAQ restarted OK. Main DAQ informed.

This appeared in the system log:

Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

These messages seem to appear sporadically.

18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?

18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.

aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters

Thanks Nic!

22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)

22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.

Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.

Typical symptoms are:

- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
   active data links are green, inactive links are red
   N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n

0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5) Reboot problem FEE64s
   Launch a command terminal and telnet to the problem FEE64 for example

    telnet aida04

   Enter user 'root' and enter password. When logged in reboot the FEE64 using the command

    reboot

6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
    you should see two messages showing the FEE64 mounting filesystems
    wait c. three minutes for boot to complete

7) Restart DAQ

   Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
   Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

8) Restore ASIC settings

   Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
   Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
   Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
   Wait 10s

9) Check ASIC settings

   Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
   Wait for popup indicating completion of checks and dismiss

10) System wide checks

    Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']

    Select 'Check clock status'  [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
     make a note or take a screenshot of any error messages observed

11) Start DAQ

    Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
    Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

13) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

14) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

15) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.








    








   

   




   

10:00 Checks:

All seems smooth.

DSSD bias and  leakage current ( Attachment 1)

FEE64 temperatures (Attachment 2)

Per FEE64 Rate spectra (Attachment 3)

ADC data item stats -(Attachment 4)

Ucesb (Attachment 5)

10:21

Aida05 was zero not reading had to call Nic to fix it.

10:34

Now, Aida05 is running well.

.......................................................

12:00 Checks:

Everything appears to be going smoothly, and I took screenshot of them.

12:28, Aida02 showed as not reading , had to email Nic to fix it.

12:30, it resumed reading and returned to normal operation.

.........................................................

14:00 Checks:

Everything is going well , and I have captured screenshots of each one.

......................................................

16:00 Checks:

All seems smooth, and I took screenshot of them.

01:02

Aida04 was zero not reading had to call Tom to fix it.

02:00 checked all good and attached.

Its been running good after that till 7:30 again the same problem happened and Tom fixed it.

16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

09.56 DAQ continues data file S100_alpha/R18_387
      FRS switching settings

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK  - attachment 2
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 3-5

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME & Correlation Scaler data time stats - attachments 7-10

      per FEE64 Rate spectra - attachments 11

      per p+n FEE64 1.8.L spectra - attachments 12-13
       aida09 pulser peak width 57 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 14

      per FEE64 1.8.H spectra - attachments 15-16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 19-21

      BNC PB-5 settings - attachment 22

10.07 data file S100_alpha/R20
      new run for new FRS setting

10.08 all histograms, stats, merger stats and data link stats zero'd

10.10 ADC data item stats (beam OFF) - attachment 23
      11/16 < 20k, max 129k (aida08)

10.31 analysis of data file R18_349 (closed 06.00 this morning)
      max deadtime 12% (aida04)
      no timewarps
      HEC data item rate 1.7kHz

11.00 tuning 168dy FRS setting
      ucesb - attachment 24
      spill length 3.5s, extraction time 1.5s

11.30 degraders now set to implant in AIDA DSSSDs
      high rates expected 

      per FEE64 Rate spectra - attachment 25

      ADC data item stats - attachment 26

      Merger etc - attachment 27
      in spill rates to MBS up 30+Mb/s, 2.5M data item/s merged

      DSSSD bias & leakage current - attachment 28
       c. few hundred nA step in leakage current 

      analysis of data file R20_33 - attachment 29
      max deadtime 18% (aida08)
      no timewarps
      HEC data item rate 4.1kHz (c. 2.4x previous FRS setting)

      Correcting for spill cycle => 5.9kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 2.0kHz in spill or 6800 implants per spill
      SC42 L/R currently report c. 4.5kHz

      Note that we currently see a higher fraction of events in DSSSD#2 cf. previous setting so multiplicity assumption may be low
      If we assume multipicity = 6 => implant rate ~ 1.0kHz in spill or 3200 implants per spill

      we are probably somewhere in between

12.50 DSSSD bias & leakage current - Grafana - attachment 30
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 31

      ADC data item stats - attachment 32

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

14.56 analysis of data file R20_106 - attachment 35
      max deadtime c. 18% (aida04 & aida08), c. 12% (aida03), c. 11% (aida06), c. 9% (aida02 & aida13), c. 6% (aida05), all others <2%
      no timewarps
      HEC data item rate 4.2kHz


15.40 DSSSD bias & leakage current - Grafana - attachments 36-37
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41

      ucesb - attachment 42

15.57 all histograms zero'd




      

22.43 24.4.24

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

      analysis of data file R18_266 - attachment 6
       max dead time 12% (aida04)
       no timewarps
       HEC data item rate 1.7kHz

03.37 DSSSD bias & leakage current OK - attachments 7-8

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10

      per FEE64 rate spectra - attachments 11

      Merger etc - attachment 12

All seems smooth.

Screenshots attached.

All seems OK.

Screenshots attached.

09.56 DAQ continues data file S100_alpha/R18_122
      FRS setting 162Eu

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK  - attachment 3
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 4-6

      WR timestamps OK - attachment 7

      ASIC settings aida01 & aida02 - attachments 8-9

      ADC, DISC, PAUSE, RESUME, WR & Correlation Scaler data time stats - attachments 10-15

      per FEE64 Rate spectra - attachments 16-18
       unusually can observe HEC Rates esp. in DSSSD #1 

      per p+n FEE64 1.8.L spectra - attachments 19-21
       aida09 pulser peak width 61 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 22

      per FEE64 1.8.H spectra - attachments 23-24

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 27-28

      ucesb - attachments 29-30
       spill cycle reported as 3.5s, extraction 1.5s

10.36 All histograms & stats zero'd

      BNC PB-5 settings - attachment 31

      DSSSD bias & leakage current - Grafana Alerting - attachment 32
       can observe bias & leakage current in greater detail
       DSSSD #1 bias stable, c. 10s nA micro structure observed in leakage current
       DSSSD #2 bias c. 10mV variations, micro structure observed in leakage current but smaller variations cf. DSSSD#1

10.46 analysis of data file R18_131
      max dead time 12% aida04, 6% aida02, all others 0.1% or less
      no timewarps
      HEC data item rates 1.6kHz - correcting for spill cycle => 2.3kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 760Hz in spill or 2700 implants per spill
      SC42 L/R currently report c. 2kHz





   

      

23.34 23.4.24

      DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

03.41 fDSSSD bias & leakage current OK - attachment 6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8

      per FEE64 rate spectra (off and on spill) - attachments 9-10

      Merger etc - attachment 11

03.54 analysis data file R18_67 - attachment 12
      max deadtime 11% (aida04)
      HEC data item rate c. 1.5kHz
      no timewarps

0.35 fDSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data item stats - attachment 15

      per FEE64 rate spectra - attachments 16

      Merger etc - attachment 17

03.54 analysis data file R18_96 - attachment 18
      max deadtime 12% (aida04)
      HEC data item rate c. 1.6kHz
      no timewarps

All looks good.

Screenshots attached.

16.12 Analysis of data file R17_101 - attachment 1

      max deadtime c. 2% (aida02)
      no timewarps observed


16.15 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 3

      per FEE64 rate spectra (on and off spill) - attachments 4-5

      Merger etc - attachment 6


16.53 aida04 stops producing data

      DAQ STOP fails for aida04
      Restart Merger to enable aida04 to complete DAQ STOP

      aida04 rebooted and DAQ restarted OK

      data file S100_alpha/R18

All looks good.

Screenshots attached

We have observed a number of AIDA FEE64s stop producing data.

Summary of dates & (approximate) times

aida03	21.4.24	08.20
aida04	22.4.24	09.14             https://elog.ph.ed.ac.uk/DESPEC/586
aida01	22.4.24	16.00
aida02	22.4.24	22.00
aida02	23.4.24	10.21
aida02	23.4.24 11.25
aida04  23.4.24 16.53?            attachment 2
aida04/aida05 25.4.24 22.28       no messages in aida04 system console
aida04  27.8.24 c.23:00
aida04  28.8.24 c. 12-13:00       power cycle
aida04  28.8.24 c. 16.00          reset per Elog 618
aida02  28.4.24 c. 22.20          reset per Elog 618, no error messages in system console
aida04  29.4.24 01.15
aida04  30.4.24 <03.30
aida02  30.4.24 <10.00


aida02 2.5.24 c. 19.30 reset


In future, it would be useful to copy system console reports to the Elog to check whether we can identify common error messages etc.


ssh pi@nnrpi1
<password>

cd logs
cat /ttyUSBx 

The current mapping of USB port # to AIDA FEE64s - attachment 1


aida02 system console c. 19.30 2.5.24

02:05:24/14:40:51|executing generic doStop
02:05:24/19:27:28|Halt (1)
02:05:24/19:27:29|Halt (2)
02:05:24/19:27:29|Halt (3)
02:05:24/19:27:30|Halt (4)
02:05:24/19:27:30|Halt (5)
02:05:24/19:27:31|Halt (6)
02:05:24/19:27:31|Halt (7)
02:05:24/19:27:32|Halt (8)
02:05:24/19:27:32|Halt (9)
02:05:24/19:27:33|Halt (10)
02:05:24/19:27:33|Halt (11)
02:05:24/19:27:34|Halt (12)
02:05:24/19:27:34|Halt (13)
02:05:24/19:27:35|Halt (14)
02:05:24/19:27:35|Halt (15)
02:05:24/19:27:36|Halt (16)
02:05:24/19:27:36|Halt (17)
02:05:24/19:27:37|Halt (18)
02:05:24/19:27:38|Halt (19)
02:05:24/19:27:38|Action has not completed
02:05:24/19:27:39|Giving up waiting
02:05:24/19:27:39|disconnect xfer stream #0 (1)
02:05:24/19:27:39|disconnect xfer stream #0 (2)
02:05:24/19:27:39|disconnect xfer stream #0 (3)
02:05:24/19:27:40|disconnect xfer stream #0 (4)
02:05:24/19:27:40|disconnect xfer stream #0 (5)
02:05:24/19:27:41|disconnect xfer stream #0 (6)
02:05:24/19:27:41|disconnect xfer stream #0 (7)
02:05:24/19:27:42|disconnect xfer stream #0 (8)
02:05:24/19:27:42|disconnect xfer stream #0 (9)
02:05:24/19:27:43|disconnect xfer stream #0 (10)
02:05:24/19:27:43|disconnect xfer stream #0 (11)
02:05:24/19:27:44|disconnect xfer stream #0 (12)
02:05:24/19:27:44|disconnect xfer stream #0 (13)
02:05:24/19:27:45|disconnect xfer stream #0 (14)
02:05:24/19:27:46|disconnect xfer stream #0 (15)
02:05:24/19:27:46|disconnect xfer stream #0 (16)
02:05:24/19:27:47|disconnect xfer stream #0 (17)
02:05:24/19:27:47|disconnect xfer stream #0 (18)
02:05:24/19:27:48|disconnect xfer stream #0 (19)
02:05:24/19:27:48|Action has not completed
02:05:24/19:27:49|Giving up waiting
02:05:24/19:27:49|executing generic Stop_Merger
02:05:24/19:27:49|executing generic Stop_DataStorageServer
02:05:24/19:27:49|completed generic doStop
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:28:01|Transfer Error - : Connection reset by peer
02:05:24/19:28:08|send() failed: 
02:05:24/19:28:08|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
02:05:24/19:28:08|do_GetState returned z=0 and 0
02:05:24/19:28:13|executing generic doGo
02:05:24/19:28:45|Initialising communication with storage server 192.168.11.99 on port 11002
02:05:24/19:28:45|Setting Transfer Block Size 65536
02:05:24/19:28:45|Setting Transfer Mode 3
02:05:24/19:28:45|Setting Overlap Mode 0
02:05:24/19:28:45|Setting Blocking Mode 0
02:05:24/19:28:45|Setting nice 0
02:05:24/19:28:45|TCP transfer library version 3.12
02:05:24/19:28:45|TCP socket send buffer was 16384 - now 221184
02:05:24/19:28:45|TCP socket receive buffer was 87380 - now 221184
02:05:24/19:28:45|TCP socket created OK - now connecting to 192.168.11.99 port 11002
02:05:24/19:28:45|Connected to 192.168.11.99 port 11002
02:05:24/19:28:45|connect xfer stream #0 (1)
02:05:24/19:28:45|Data Acquisition Statistics counters now cleared
02:05:24/19:28:46|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
02:05:24/19:28:46|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
02:05:24/19:28:46|Go (1)
02:05:24/19:28:46|Running Go_Electronics for aida02 with TSMaster aida01
02:05:24/19:28:46|ADCs Calibrated. 
02:05:24/19:28:46|Finished Go_Electronics for AIDA
02:05:24/19:28:46|executing generic Go_Merger
02:05:24/19:28:46|executing generic Go_DataStorageServer
02:05:24/19:28:46|completed generic doGo
02:05:24/19:28:46|do_GetState returned z=0 and 1
02:05:24/19:28:46|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
02:05:24/19:28:51|Also in RDOGo_Operate: Enabled waveform readout
02:05:24/19:28:51|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0201
02:05:24/19:28:51|WV:0:Buffer info. Length = 64 : Offset = 0 
02:05:24/19:28:51|WV:1:Buffer info. Length = 62 : Offset = 2 
02:05:24/19:28:51|WV:2:Buffer info. Length = 60 : Offset = 4 
02:05:24/19:28:52|WV:3:Buffer info. Length = 58 : Offset = 6 
02:05:24/19:28:52|WV:4:Buffer info. Length = 56 : Offset = 8 
02:05:24/19:28:52|WV:5:Buffer info. Length = 54 : Offset = 10 
02:05:24/19:28:52|WV:6:Buffer info. Length = 52 : Offset = 12 
02:05:24/19:28:52|WV:7:Buffer info. Length = 50 : Offset = 14 
02:05:24/19:28:52|WV:8:Buffer info. Length = 48 : Offset = 16 
02:05:24/19:28:52|WV:9:Buffer info. Length = 46 : Offset = 18 
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 1048, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|

Everything is OK now.

Screenshots attached.

We lost aida02 at about 11:25 again. Nick fixed it about 11:50.

It is now back in the DAQ.

Nick rebooted AIDA and aida02 came back at around 10:55.

AIDA was out of the DAQ from about 10:30 until 10:55

Aida02 is down. Contacted Nick.

The rest looks stable, see screenshots.

10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14

Go > All 16 writing to merger, looks OK

aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again

Checks 08:00-08:30

Screenshots are attached and everything looks OK.

23.36 Analysis of data file R17_34 - attachment 1
      max deadtime c. 1% (aida02)
      no timewarps

23.38 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 3
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

01.03 DSSSD bias & leakage current OK - attachment 7

      FEE64 temperatures OK - attachment 8
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 9

      per FEE64 Rate spectra - attachment 10

      Merger etc - attachment 11

03.06 DSSSD bias & leakage current OK - attachment 12

      FEE64 temperatures OK - attachment 13
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 14

      per FEE64 Rate spectra - attachment 15

      Merger etc - attachment 16

05.04 DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Merger etc - attachment 21

07.29 DSSSD bias & leakage current OK - attachment 22

      FEE64 temperatures OK - attachment 23
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 24

      per FEE64 Rate spectra - attachment 25

      Merger etc - attachment 26

First checks of shift. All looks ok.

Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.

Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)

checks at 16.00 - merger has crashed - aida01 dropped out  - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09 

Reset 01, restarted DAQ, and restarted as R16.

16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.

Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms

RUN17 STARTED 16.51|

 HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.

20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift

21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps

22.48 - things apear ok. (Attachments 23-26) 

Attachment 1 - per DSSSD LEC e_p versus e_n (20keV/channel)

Attachment 2 - per DSSSD HEC m_p versus m_n 

Attachment 3 - per DSSSD HEC x versus y (I really do need to sort out the mapping!)

Attachment 4 - per DSSSD HEC e_p versus e_n (20MeV/channel)

Attachment 5 - per DSSSD per pixel HEC-HEC time (4.096us/channel)

Attachment 6 - per DSSSD implant/decay event rates (262.144us/channel)
               showing beam spill cycle

Attachment 7 - colours/contour levels for all 2D spectra

09.14 aida04 stopped producing data - system console output - attachment 1

It is not clear (to me at least) whether the cause is radiation-induced single event upsets or
the start/stop of LN2 fills.

Nor is it clear why the FEE64 stops
- WR timestamp error (data link to Merger blocks)
- AIDAExecV10 fails
- ?

rebooted aida04

DAQ RESET/SETUP etc cycle for all FEE64s

10.03 successful restart 
      current data file R15

00.15 Data link to MBS was closed by MBS - reason(s) unknown
      NH restarted MBS receiver and data transfer restarted

01.34 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      Merger, TapeServer etc - attachment 4

04.36 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      Merger, TapeServer etc - attachment 8

07.54 data storage to local disk enabled
      file aida-gsi:/TapeData/S100_alpha/R14_

09.36 DAQ continues, current data file R14_46

      168Er setting

      DSSSD bias & leakage current OK - attachments 9 & 10
      leakage current DSSSD#1 ramping and cycling with beam spills

      FEE64 temperatures OK - attachment 11

      System wide checks OK *except* WR/FPGA errors - attachments 12-13

      WR timestamps - attachment 14
       aida04 out of seq

      ADC data item stats - attachment 15
       high rates observed all FEE64s beam on, beam off rates OK

      per FEE64 Rate spectra - attachment 16

      per FEE64 1.8.L spectra - attachments 17-18

      per FEE64 1.8.W spectra - attachments 19-20

      per FEE^4 1.8.H spectra - attachments 21-22
   
      Merger, TapeServer etc - attachments 23-25
       data rates to c. 10Mb/s beam on


09.55 aida04 rebooted - https://elog.ph.ed.ac.uk/DESPEC/586

10.01 DAQ restarted
      data file R15

      ADC data item stats - attachments 26-27
       large variation in rates beam off/on

c. 10.30 beam off (for new FRS setting?)


11.25 analysis of data files R14_19 (beam ON) and R15_24 (beam OFF) - attachments 28-29

      For beam ON max deadtime c. 11% (aida04), beam OFF max deadtime c. 0.1% (aida04)

      Zero timewarps observed

      Notice large # of PAUSE/RESUME data items for aida03 & aida04 presumably due to additional overhead of processing the time machine scalers 

12.35 Beam off

      DSSSD bias & leakage current OK - attachment 30
       Can observe beam spill structure in DSSSD leakage currents - now slowly recovering with beam off

      FEE64 temperatures OK - attachment 31

      ADC data item stats OK - attachment 32

      Merger, TapeServer etc - attachment 33

12.55 New ASIC settings 2024Apr22-12.55.38
       *all* discs disabled to reduce load on aida04
       slow comparator 0x14

      BNC PB-5 settings (since yesterday evening)

      Amplitude 10.0V
      Attenuation x10
      tau_d 1ms
      Frequency 2Hz
      Tail pulse
      Polarity +


13.46 all histograms & stats zero'd

13.48 Beam off
      per FEE64 Rate spectra - attachment 34
       2Hz pulser to p+n FEE64s

      per FEE64 1.8.W spectra - attachments 35-36
       p+n FEE64 threshold = 6000 to capture pulser events
       n+n FEE64 threshold = 8000 to capture noise

      per p+n FEE64 1.8.L spectra - attachments 37-38
       common x/y-scale - peak height proxy for peak width
       aida09 pulser peak width 57 ch FWHM ~40keV FWHM ( 5 x sigma ~ 85keV )

14.28 168Eu setting (check only - setup still in progress)

01.35 DSSSD bias & leakage current OK - Grafana - attachment 1

      FEE64 temperatures OK - attachment 2
 
      ADC data item stats OK - attachment 3

      Merger OK - attachment 4

04.05 DSSSD bias & leakage current OK - Grafana - attachment 5

      FEE64 temperatures OK - attachment 6
 
      ADC data item stats OK - attachment 7

      Merger OK - attachment 8

08.20 DSSSD bias & leakage current OK - Grafana - attachment 9

      FEE64 temperatures OK - attachment 10
 
      ADC data item stats - attachment 11
       increased rates observed esp. DSSSD#2 n+n

      Merger etc - attachment 12
      rate increased x2 cf. overnight

      WR timestamp - attachment 13
       aida03 out sequence

09.25 MBS data relay restarted

      Merger data links - attachment 14
       aida03 rate zero

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16

      per FEE64 Rate spectra - attachment 17
      increased rate for DSSSD#2 n+n FEE64s in particular

      MIDASsort online analysis - attachments 18-19
      per DSSSD e_front versus e_back for LEC (20keV/channel) and HEC (20MeV/channel)
      observe light ions (LEC) c. 2MeV
      observe low rate of high energy ions (HEC) c. 600MeV
      similar energies in both DSSSDs for LEC and HEC implying both types of event have range >> 2mm Si


10.09 DAQ STOP
      aida03 does not respond - reboot aida03

      DAQ RESET/SETUP etc     

      Synchronise ASIC clocks
      All system wide checks OK *except* FPGA errors

      Data transfer disabled
      Histograms & stats zero'd
      DAQ GO

      Merger restarted 

      Data transfer enabled
      DAQ GO

      WR timestamp OK - attachments 20-21

      ADC data item stats OK - attachment 22
       rates status quo ante
  
      Merger etc OK - attachment 23


14.58 DSSSD bias & leakage current OK - Grafana - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats OK - attachment 26

      Merger etc OK - attachment 27

18.13 DSSSD bias & leakage current OK - Grafana - attachment 28

      FEE64 temperatures OK - attachment 29

      ADC data item stats OK - attachment 30

      Merger etc OK - attachment 31


20.30 Primary c, few kHz 170Er beam to S4 - upstream s/steel flange removed
      Start of stopping range calibration in the implantation stack - currently ion appear to be stopping in DSSSD#1 (upstream)


      Decay and implant events as function of time showing spill structure - attachment 32

      per DSSSD e_front versus e_back (HEC, 20MeV/channel) - attachment 33

      per DSSSD x versus y hit pattern (HEC) - attachment 34
       mapping to be configured but clear ions are mostly implanted in centre of DSSSD#1 

      2D colour/contour levels - attachment 35

      per DSSSD m_front versus m_back (HEC) - attachment 36

      per DSSSD per pixel HEC-HEC time (4.096us/channel) - attachment 37

23.44 aida02 has stopped running
      DAQ STOP
      aida02 rebooted

      Data transfer disabled

      DAQ RESET/SETUP 

      Synchronise ASIC clocks
      All system wide checks OK *except* FPGA errors

      Histograms & stats zero'd
      DAQ GO

      Merger restarted 

      Data transfer enabled
      DAQ GO







        


        

Note 40kBq 60Co source placed on bottom surface of snout centred on bPlas detectors and DSSSDs

18.50 Primary beam 750MeV/u Er to S4
      Primary beam stopped by flange (5cm s/steel in position upstream of implantation stack)

      Observe c. 2MeV dE peak in AIDA DSSSD#1 and DSSSD#2 using MIDASsort online analysis - attachment 1



21.50 Primary beam 1000MeV/u Er to S4 - attachment 2

      per DSSSD e_front versus e_back, 20keV/channel
      DSSSD #1 x & y -axis projections

10.50 Overnight stats - attachment 3
      Primary beam ?
      per DSSSD e_front versus e_back, 20keV/channel
      DSSSD #1 x & y -axis projections
      note multiple dE peaks observed in both DSSSDs

S100 ADC offsets using pulser walkthrough data from data files R12 & R13

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )




per p+n FEE64 ADC spectra (aida01-aida16 *not* aida02, aida04, aida06 & aida08) - attachments 1-12

Note common x & y-scales - peak height proxy for peak width

aida09 1.8.L (s520) pulser peak width 7 ch FWHM (~40keV FWHM)




per n+n FEE64 ADC spectra (aida02, aida04, aida06 & aida08) - attachments 13-16

Note common x & y-scales - peak height proxy for peak width

aida02 1.8.L (s72) pulser peak width 16 ch FWHM (~88keV FWHM)




ADC offset analysis

/home/npg/td/Calibration/offsets /home/npg/td/S100/R12-R13 50 1024 20 5


First pass sort variables ( ADC offsets, FEE64 configuration, LEC & HEC equal energy cuts ) - attachment 17

Note 60Co source (current activity c. 43kBq) was installed yesterday evening - assume it is positioned on the bottom side of the snout as per the other sources used this week/

04.17 DSSSD bias & leakage current OK - Grafana - attachment 1

      FEE64 temperatures OK - attachment 2
       aida02 ASIC temp known to be u/s

      ADC data item stats - attachment 3
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5


11.24 DSSSD bias & leakage current OK - Grafana - attachment 6

      FEE64 temperatures OK - attachment 7
       aida02 ASIC temp known to be u/s

      ADC data item stats - attachment 8
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs

      per FEE64 Rate spectra - attachment 9

      Merger etc - attachment 10


17.50 DSSSD bias & leakage current OK - Grafana - attachment 11

      FEE64 temperatures OK - attachment 12
       aida02 ASIC temp known to be u/s

      ALl syhstem wide checks OK *except* WR/FPGA errors - attachments 13-14

      WR timestamps OK - attachment 15

      per FEE64 Rate spectra - attachment 16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18

      Merger, TapeServer, Data Links - attachments 19-20
       no data storage

      ADC data item stats - attachment 21
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs


18.01 Grafana alerting shows detail of DSSSD leakage current and S4 temperature variations over time - attachment  22
       DEGAS LN2 fill cycle every 6h 10.00 - 16.00 - 22.00 ... etc

https://despec-vm-01.gsi.de/grafana/d/tHRVJYfGz/aida-alerting?orgId=1

12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3

      During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
      aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )

      AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
       *except* aida02 ASIC temp - known fault

      System wide checks OK *except* WR/FPGA errors - attachments 3-4

      WR timestamps OK - attachment 5

      ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
       ADC data items 10/16 < 20k, max 143k

      per FEE64 Rate spectra - attachment 11

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-13


15.10 Synchronise ASIC clocks
      Re-calibrate ALL ADCs 

      ASIC settings 2024Mar27-11.25.32

      Changes ( https://elog.ph.ed.ac.uk/CARME/499 )

       IBias LF feedback from 0xf to 0x8
       Diode link threshold from 0xbf to 0xca

      New ASIC settings saved as 2024Apr19-15.22.49

      All FEE64 slow comparators -> 0x14

      Data file S100_alpha/R12

      Pulser ealkthrough (test +)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity +
      Tail pulse
      tau_d 1ms

16.35 DAQ ends OK file S100_alpha/R12_24

      per p+n FEE64 1.8.L spectra - attachment 14
       aida09 pulser peak width 54 ch FWHM 

17.38 Data file S100_alpha/R13

      Pulser walkthrough (test -)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity -
      Tail pulse
      tau_d 1ms

18.43 DAQ ends OK file S100_alpha/R13_20

      per p+n FEE64 1.8.L spectra - attachment 15

18.55 Current status

      FEE64 power ON
      DSSSD bias ON

      All FEE64 slow comparator 0x14

      DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )


To Do

- S4 currently closed/controlled access for FRS startup

If we have access to S4 tomorrow

- switch test - to test + ( not critical )
- test AIDA interlock 
- further alpha background?



22.47 DSSSD bias & leakage current OK - attachments 16-17

      FEE64 temps OK - attachment 18
       *except* aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Data link, Tape Server & Merger - attachments 21-23

       

03.30


09.10



09.25 data file S100_alpha/R11
   
      BNC PB-5 pulser data

      amplitude 10.0V
      attenuation x10
      tau_d = 1ms
      polarity - ( initially + )
      frequency 25Hz

      ASIC settings2024Mar27-11.25.32
       slow comparator 0x64 all FEE64s


      aida02 pulser peak width 129 ch FWHM

      polsrity +

      aida09 pulser peak width 57 ch FWHM


09.40 switch to no storage

      p+n FEE64 slow comparators 0x64 -> 0xa
      n+n FEE64 slow comparators 0x64 -> 0xf

      radioactive source 152Eu 370kBq installed on the bottom side of the snout, centred tranversely, located beneath bplas-DSSSD-DSSSD-bplas position.

      ADC data item stats - attachment 35
      9/16 < 20k, max 108k

      per FEE64 Rate spectra - attachment 36

10.44 152Eu source removed

      ADC data item stats - attachment 37
      9/16 < 20k, max 102k

      per FEE64 Rate spectra - attachment 38

WR timestamps appear to be OK but readout times fort aida03, aida04 and aida09 appear to be out of sequence. Attachment 1-3.

Reseated HDMI cable of aida09 and performed complete powercycle. Check the MACB HDMI port (appears to be loose) - still no change aida09 not collecting WR timestamps. Attachment 4-5.

09.52 DAQ continues file S100_alpha/R5
       note merger not functioning correctly due to aida03, aida04 & aida09 out of WR sequence 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      WR timestamps - attachment 3
       aida03, aida04 & aida09 out of sequence

      All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 4-6

      ADC data item stats - attachment 7

      per FEE64 Stat spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-10

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-12

      Merger, TapeServer & Link Data Rates - attachments 13-15

      Grafana - attachment 16
       DSSSD bias/leakage current trends OK


19.15 alpha background file S100_alpha/R10

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temps OK - attachment 18
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 19-21

      WR timestamps OK - attachment 22

      ADC data item stats - attachment 23

      Merger, TapeServer & Meger stats - attachments 24-26
       data rate c. 2.8Mb/s

      per FEE64 1.8.W spectra - 20us FSR - attachments 27-28

      correlation scaler stats & data link stats - attachments 29-30
       data rate dominated by MACB scaler inputs aida03 & aida04 ( c. 200kHz each )


22.16 DAQ continues Ok file S100_alpha/R10_35

      Merger, TapeServer & Meger stats - attachments 31-33
       data rate c. 2.8Mb/s

      ADC data item stats - attachment 34

      FEE64 temps OK - attachment 35
      *except* aida02 ASIC temp which is known to be u/s

      DSSSD bias & leakage current OK - attachment 36

00.40 Observed zero data fron aida03, aida04, aida09
      WR timestamps offset from other FEE64s

      rebooted aida03, aida04, aida09
      no change
      see attachments 14-17



18.25 rebooted aida03, aida04, aida09, aida15

      system wide checks OK *except* aida15 ADC not calibrated, FPGA decoder errors aida02, aida03
      ADC data item stats - attachment 28
      WR timestamps OK - attachment 29

      data file S100_alpha/R5 - alpha background

      Merger and TapeServer - attachments 30-31
       c. 1Mb/s to disk

21.05 FEE64 temps OK - attachment 44
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida15 ASDC not calibrated, WR timestam/FPGA errors - attachments 41-43

      WR timestamps- attachment 40
       aida03, aida04 and aida09 differ from other timestamps again

      Merger and TapeServer - attachments 38-39
       c. 0Mb/s to disk due to WR timestamp issues

      per 1.8.W spectra - 20us FSR - attachments 36-37

      per FEE64 Rate spectra - attachments 34-35

      per FEE64 1.8.L spectra - attachments 32-33

11.45 DSSSD bias & leakage current OK - attachments 9-10

      FEE64 temps OK - attachment 8
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder/FPGA status errors - attachments 6-7
       HDMI cabling disturbed during platform move?

      WR timestamps OK - attachment 5

      ADC data item stats - attachment 4
        10/16 < 20k, max 115k

      per FEE64 Rate spectra - attachments 3

      per 1.8.W spectra - 20us FSR - attachments 1-2

11.58 BB7 preamps unplugged. Noise conditions OK. Did not change considerably.
      11/16 < 20k (aida09 has 1 hot channel), max 114k Attachments 11-12.

12:34 Detector powered down to bring to safe state to check HDMI cables of aida02 and aida03.

15:06 Reseated aida02 and aida03. No more FPGA timestamp error. See attachment 13-14.

      Rates and histograms look good for pulser walk through. Test + and test - connected. Currently, Attachment 16-17.
      Temp OK Attachment 15. Bias OK Attachment 18.

      Data links aida02 and aida08 missing (?)

16:30 All slow comparator thresholds set to 0x64 for alpha background run during platform freeze.

      Initial rates and histograms attachment 19 - 20. aida04 noisy in ASIC 1.

      Temps OK. Attachment 21.
      MBS disc and tape setup in atttachment 22-23.

9:40 started up AIDA. TEMP OK. Noise condition same as Saturday. Attachments 1-3.

9:56 BB7 premaps connected but not powered. Attahcment 4.

10:20 BB7 Preamps powered. Attachment 5.

Noise increased marginally, but still 10/16 <20kHz, max 130k.

15.15  per FEE64 Rate spectra - attachment 6

             ADC data item stats - attachment 7

           increased noise observed - 6/16 < 20k, max 300k 

17.10  per FEE64 1.8.W spectra - 20us FSR - attachments 8-9

            per FEE64 Rate spectra - attachment 10

             ADC data item stats - attachment 11

           increased noise observed - 7/16 < 20k, max 180k 

20.30  per FEE64 Rate spectra - attachment 12

             ADC data item stats - attachment 13

           status quo ante - 10/16 < 20k, max 130k 

          FEE64 power OFF

          DSSSD bias OFF

15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).

          TEMP OK.

          Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.

          9/16 <20kHz, max 176k

15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.

          No change in noise observed.

          9/16 <20kHz, max 187k

15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.

          10/16 <20kHz, max 158k.

Current bPlast grounding scheme in attachments 8-9.

15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.

           Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.

           Histograms okay, some FEE64s now have hot channels. Attachment 2.

           9/16 aidas < 20kHz - max rate 162k

18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.

TO-DO (kicking the can down the road):

- Some work to do on noise, but we might have to accept the situation as it currently is.

- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.

- Try different grounding configuration:

    - Grounding drain wires to frame. Observe AIDA conditions.

    - Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.

TO-DO for 11.04.2024

- Try bringing bPlast drain wire ground back to the PSU ground for PN 300 and R&SRMP4040.

- Recheck the downstream detector bias and ground scheme.

-

9:27 Restarted AIDA and hit go at 9:54. Situation the same as it was in the evening of 10.04.2024. 10 out of 16 FEE64s in good condition. Waveforms of aida16,

        TEMP OK - HV OK. See attachments 1-6.

10:20 We grounded the output of the R&SRMP4040 to to the drain wire ground of the bPlast ribbon cables and the output of the PN 300 PSU that is powering the booster boards. Results given by attachments 7 -11. We only saw a marginal improvement in the overall noise condition.

10:32 We connected the ground of the frame to the drain wire ground, that is also mutually connected to the ground of the R&SRMP4040 output ground and PN 300 output ground. Results given by attachments ???. We did not see any improvement in the overal noise condition.

11.00 DSSSD HV OFF

          FEE64 power OFF

          Check seating of all adaptor PCBs and drain/ground wires secured - generally OK

          Restart

          1.8.W spectra - 20us FSR - attachments 15-16

          per FEE64 Rate spectra - attachment 17

           10/16 < 20k, max 110k

          ADC data item stats - attachment 18

          Incremental improvement.

11.45 DSSSD HV OFF

          FEE64 power OFF

          test - daisy chain removed

          Restart

          1.8.W spectra - 20us FSR - attachments 20-21

          per FEE64 Rate spectra - attachment 19

           10/16 < 20k, max 210k

          ADC data item stats - attachment 22

          overall somewhat worse

12.20 DSSSD HV OFF

          FEE64 power OFF

          Re-install test - daisy chain, Tighten aida04 DSSSD ribbon cable drain wire.

          Restart

          1.8.W spectra - 20us FSR - attachments 23-24

          per FEE64 Rate spectra - attachment 25

          ADC data item stats - attachment 26

           10/16 < 20k, max 110k

          Status quo ante

         aida04 & aida08 1*W & 2*W spectra - 200us FSR - attachments 27-31

           large transients observed for aida04 1*W - cable or ASIC fault?

         aida16 **W spectra - 200us FSR - attachments 32-35

           all channels apopear to be working but mix of high/low noise channels for asics #3-4, asics #1-2 all appear high noise

14.00 DSSSD HV OFF

          FEE64 power OFF

          Disconnect ribbon cables from aida04 adaptor PCB

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 37

          per FEE64 Rate spectra - attachment 36

          ADC data item stats - attachment 38

          Implies origin of large transients observed in aida04 asic#1 is downstream of the FEE64 adaptor PCB, i.e. ribbon cable or DSSSD.

14.40 DSSSD HV OFF

          FEE64 power OFF

          re-connect ribbon cables from aida04 adaptor PCB

          1x pin J2 slightly bent - straightended with screwdriver - FFSD connector insertion OK

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 40-41

          per FEE64 Rate spectra - attachment 42-43

          ADC data item stats - attachment 39

            10x < 20k, max 120k

           aida11 asic#4 1.8.W spectra - 200us FSR - attachment

             no large transients observed

           DSSSD #1 & #2 bias from -120V to -100V

            ADC data item stats - attachment 40

            9x < 20k, max 220k (aida08)

            per FEE64 Rate spectra - attachment 41

17.15 bPlas ON -

        Current bPlast ground configuration:

         bPlas current ground configuration - drainwires of all ribbon cables excluding short side (cont. with snout) are grounded back to the bPlast R&SRMP4040 PSU, on the 29 V output. 4 V PSU to booster boards are floating, output of PN 300 is grounded to the PSU. The mesytec PSU, that also powers bPlast SiPMs at

         29V is not grounded to anything but the frame. Snout is currently light tight, I suspect internal radiation from FATIMA.

         It is noted that bPlast current draw fluctuated significantly (+/- 300 mV) as a result of thresholds set to the detector. I tried to set the thresholds HIGH to stabilise the bPlast detector as the power draw is fluctuating greatly.         

          AIDA noise very good

            ADC data item stats - attachment 42

            12x < 20k, max 83k (aida04)

            per FEE64 Rate spectra - attachment 43

To Do list

- separate 29V (low current) and 4V (high current) return paths/ground refs

- The mesytec PSU that powers the bPlast detector  is running to frame, it might be an idea to ground ribbon c

09.38 CC completed install of Bplast driver PCBs yesterday evening.
      All flat ribbon cables connected - all drain wires grounded
      PSU on but not enabled - return terminals grounded to PSU front panel ground
      SiPm bias off

      Water pressure and temperature OK
      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      ADC data item stats - attachments 5

      per FEE64 Rate spectra - attachments 6-7

      per 1.8.W spectra - 20us FSR - attachments 8-11

      ASIC settings 2024Mar27-11.25.32
       LEC slow comparator p+n FEE64s 0xa, n+n FEE64s 0xf

      BNC PB-5 pulser - attachment 12



10.45 CC returns
      bPlas ON

      ADC data item stats - attachments 13
       8x < 20k, max c. 310k

      per FEE64 Rate spectra - attachments 14

      per 1.8.W spectra - 20us FSR - attachments 15-16

      downstream DSSSD n+n and bottom left & right p+n FEE64s noisy


11.40 per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 55 ch FWHM ~38keV FWHM - no change cf. before installation of bPas *except* aida16


12.00 Photos of snout, bPlas driver PCBs, cabling, grounding and PSUs courtesy JB - attachments 18-27

      N.B outputs of PN300 PSU at base of AIDA support stand are *not* ground ref'd - attachment 23


12.20 Slow comparator -> 0x64
      Pulser OFF
      All histograms zero'd

13.17 JB: returned from lunch. Current status of AIDA modules given by attachment 29.

15.55 While bPlast thresholds were being set the noise increased substantially.

16.51 Replaced mezzanine of aida14, reinstalled and biasing detector. Resulting for noise conditions in the detector given by attachments 30-33.

17.31 With a multimeter it was found that there is continuity between the snout and the frame.

18.50 We tried to disconnect the cables of the short side of the bPlast detector and all of the grounds. This seemed to show an open line OL on the multimeter and whence connecting the bPlast grounds back excluding the short side ribbon cable grounds, the multimeter still read OL. 
      We also tried to wedge paper between the short side ribbon cable of bPlast and the snout, but this did not work, the detector still reading continuity between the frame, booster board and snout. 
      The results after booting up the detector again are given by the attachments 34-36. 
      The noise condition is appreciably better than before with 12 out of the 16 FEE64 modules with sub 20 kHz rates.
      Disconnect the BB7 preamp. ground from the frame.

18.54 We powered up bPlast. The noise condition three FEEs got worse, aida01, aida11 and 06. The problem may be associated with downstream grounding. The results are given by the attachments 37-43. 10 out of the 16 FEE64 modules showed sub 20 kHz rates.

19.07 We turned off the power supply on the base of the snout support. PN300 Attachment 23. Rates did not change - attachment 44.

19.10 We turned off the mesytec PSU that is located in the bPlast NIM crate. No change was observed in the rates - attachment 45.

19.13 We turned off the R&SRMP4040 PSU that is located above the AIDA crate. And the noise situation did not change attachment 46.

19.22 Leaving the R&SRMP4040 PSU off we turned back on the mesytec PSU and PN300 PSU, attachment 47.

Summary:

It is clear that with no continuity on the short side we are able to reduce the noise back to the scenario where bPlast was not connected. However, powering bPlast introduced substantial noise in both DSSSDs that did not go away when turning all the PSUs off. There might be some hysteresis in the system (this is just speculation).

The situation is still fair considering that 10 out of the 16 FEE64 modules are in the sub 20 kHz rate level (good noise condition).

TO-DO for 11.04.2024

- Try bringing bPlast ground back to the PSU ground.

- Recheck the downstream detector bias and ground scheme.
       

Disconnected test - pulser from aida08. Noise conditions improved slightly, probably less pulser events. (Attachments 1 & 2).

We recabled the dew point sensor and flow sensor for the water supply interlock to the FEE64s. When we tried to power up the interlock box, the PSU (charging block) LED was blinking indicating an internal power instability. We then plugged each of the sensors out and into the interlock box and found that the flow sensor was the source of the power supply issue.

The HDMI cable of aida04 appears to be broken or wobbly, but the signals still look the same after powering up again. The adaptor board was reseated.

Pulser peak of test - remeasured the peak in aida02 1.8.L and found the same result as before with 118 channels FWHM c. 95-100 keV FWHM energy resolution. (Attachment 3)

bPlast booster boards connected caused dramatic increase in the noise observed in AIDA. Grounding still needs to take place.

TO-DO for 10.04 - 11.04:

- aida04 asic #1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?

- noise optimisation of aida-bplast -- (friday if BB7 also powered).

BNC PB-5 set to remote control

Login to RPi nnrpi2

ssh -X nnrpi2 -l pi

Start PB-5 app - attachment 1

./BNC-PB-5/pb5.py

09.36 Cooling water temperature and flow OK

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      info code 4, correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 5-10

      per FEE64 Rate spectra - attachments 11

      per 1.8.W spectra - 20us FSR - attachments 12-13

12.25 DSSSD bias & leakage current OK - attachment 16

      FEE64 temps OK - attachment 15
      *except* aida02 ASIC temp which is known to be u/s

      ADC data item stats - attachments 14

      system stable

17.32 Power and detector bias cycle

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2
       *except* aida02 ASIC temp u/s

      ASIC settings 2024Mar27-11.25.32
       slow comparator p+n FEE64s 0xa (100keV), n+n FEE64s 0xf (150keV)

      BNC PB-5 pulser
      amplitude 10.0V
      attenuation x10
      frequency 25Hz
      polarity -
      tau_d 1ms
      tail pulse

      test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

      test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

      All system wide checks OK *except* aida02 & aida03 WR decoder status errors - attachment 10

      WR timestamps OK - attachment 11

      ADC data item stats - attachment 3
       12 of 16 < 20k, all < 100k

      per FEE64 Rate spectra - attachment 4

      per n+n FEE64 1.8.L spectra - attachment 5

      1.8.W spectra - 20us FSR - attachments 6-9
       preamplifier output noise generally very good

*Current ground configuration*

CAEN N1419ET LK fitted (non floating outputs)

ground cable from aida04-aida12-aida02-aida09-aida01-aida05
HV#0 aida12-aida03-aida15

ground cable from aida08-aida16-aida06-aida10-aida14-aida13
HV#1 aida16-aida07-aida11

LK3 fitted aida03, aida07

LK1 fitted aida02, aida04, aid06, aida08

test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

all DSSSD ribbon cable drain wires grounded to their resepctive AIDA adaptor PCBs

JB observed open circuit between AIDA Al snout and AIDA support frame ( as expected )

AIDA PSU cabling as reported https://elog.ph.ed.ac.uk/DESPEC/560

Snout configuration:

- all bPlast cables are disconnected and floating. 
- CC taped up all the cables and exit points in snout.
- CC, JB covered the exit of the snout with two additional layers of aluminium foil which were also taped shut.


To Do:

- repower detector, see if similar conditions persist.
- disconnect test - pulser, again see if conditions persist.

- aida04 asic#1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?




       

DRG quotes ( from Elog ) S505 FEE64 configuration as

"Configuration: Single/Narrow - 8 FEEs, 2 DSSDs

DSSD 1:
 X 0-63    =  3 (HV -ve)
 X 64-127  =  1
 Y 0-63    =  4 (HV 0v) 
 Y 64-127  =  2 
DSSD 2:
 X 0-63    =  7 (HV -ve)
 X 64-127  =  5
 Y 0-63    =  8 (HV 0v)
 Y 64-127  =  6 


Scalers:
1 - Pulser
2 - n/c
3 - Time Machine Original
4 - Time Machine Delayed
5 - n/c
6 - n/c
7 - SC41 L
8 - SC41 R"

LEC (20MeV FSR) spectra

attachment 1 m_p versus m_n DSSSD#1 & DSSSD#2
attachment 2 E_p versus E_n DSSSD#1 & DSSSD#2

HEC (20GeV FSR) spectra

attachment 3 m_p versus m_n DSSSD#1 & DSSSD#2
attachment 4 E_p versus E_n DSSSD#1 & DSSSD#2

attachment variables.dat ( ADC offsets, e_diff windows & FEE64 onfiguration ) NAMELIST I/O format

In the morning we reinstalled FEE64s aida06, aida13, aida07. The rails of the FEE64 boards were found to be incorrect (that of the rails that were used in CARME).

After replacing the rails on the abovementioned FEE64s we powercycled AIDA and found the following results in the histograms and statistics.

As can be seen in attachement 4, aida16 has a noisy waveform and we are going to investigate this further.

When setting the threshold to 0x64 (1 MeV) on the slow comparator threshold we saw that the low energy branch of aida04 was still noisy. (Attachment 6)

- Setting the slow comparator threshold to 0x32 (0.5 MeV) introduced noise into aida08, aida16 and substantially more noise in aida04. (Attachment 7)

- Setting the slow comparator threshold to 0x14 (0.2 MeV) introduced noise into aida11 and substantially more noise in aida04, aida08, aida16 and aida06. (Attachment 8)

- Setting the slow comparator threshold to 0xf (0.15 MeV) introduced more noise into the same aforementioned channels aida04, aida06, aida08, aida11, aida16 (additionally aida02), barring hot strips. (Attachment 9)

We covered the end of the snout with a black cloth and this reduced the noise in some of the channels (Attachment 10 and 11).

- We covered the end of the snout with two layers of aluminium foil - this produced the same result as covering the snout with the black cloth (Attachment 12, 13, 14 & 15).

We resumed after lunch.

- Replaced the aida16 adaptor PCB with a spare, and this produced a much more sensible rate and hitpattern in aida16, which could lead us to believe that the aida adapter card was faulty or misaligned. (Attachment 16, 17, 18 & 19).

- Placed two LK1 back on the n+n side of both DSSSDs on aida06 and aida02. The noise and rate did not change at all. (Attachment 20 & 21)

Pulser tests:

- The test + pulser was connected to the p+n sides (top and bottom) at 25 Hz.

- After checking the ASIC control 2 times over, the pulser seems to be ok for all p+n channels excluding aida01, aida13 and aida14. (Attachment 23)

- The pulser peak width of aida09 1.8.L was found to be c. 55 channels FWHM on uncalibrated scale which corresponds to 35 - 40 keV FWHM in energy resolution. (Attachment 24)

- After reseating the adaptor board PCBs we found the same result as before reseating the adaptor boards (Attachment 25).

- Connecting the test - pulser to the n+n  side we found that most of the ASICs behaved normally which ASIC 1 in aida04 was showing abnormal behaviour. (Attachment 26, 28). The test + pulser loop is still in place.

-  The pulser peak width of aida02 1.8.L was found to be c. 118 channels FWHM on uncalibrated scale which corresponds to 95 - 100 keV FWHM in energy resolution. (Attachment 27)

- We set the thresholds on the n+n side to 0xf and the p+n side to 0xa. Subsequently, we observed improved noise conditions in many of the channels. The still noisy channels are aida01, aida04, aida08. (Attachment 29, 30). This might be because we have introduced an addtional ground loop with the pusler tests, but could also mean that the ground loop that is currently being used is not sufficient or poorly implemented.

Summary:

- From the pulser test it is clear that aida14 ASICs 1&2 are not in good shape and appear to be faulty. 

- We managed to improve from 1 aida module in the sub 20 kHz rate range to 6 aida modules in the sub 20 kHz rate range by the end of the tests.

- We set the slow comparator thresholds on the n+n side to 0xf and the p+n side to 0xa. Subsequently, we observed improved noise conditions in many of the channels. The still noisy channels are aida01, aida04, aida08. (Attachment 29, 30). This might be because we have introduced an additional ground loop with the pusler tests, but could also mean that the ground loop that is currently being used is not sufficient or poorly implemented.

19.10 CAEN N1419ET LK fitted
      power cycle

attachments 1-6

FEE64 configuration per https://elog.gsi.de/despec/S100/6

Configuration: Wide - 16 FEEs, 2 DSSD

DSSD 1:
 X 0-63    =  15 
 X 63-127  =  9
 X 128-191 =  3 
 X 192-255 =  1
 X 256-319 =  12 
 X 320-385 =  5
 Y 0-63    =  4 
 Y 63-127  =  2 


DSSD 2:
 X 0-63    =  11 
 X 63-127  =  10
 X 128-191 =  7
 X 192-255 =  14
 X 256-319 =  16 
 X 320-385 =  13
 Y 0-63    =  8
 Y 63-127  =  6 

Scalers:
1 - Pulser
2 - FRS Accepted
3 - Time Machine Undelayed
4 - Time Machine Delayed
5 - SC41 L
6 - SC41 R


Thresholds (Slow / Fast / Implant)
DSSD 1 p+n : 0xA (100 keV) / 0xFF (disabled) / 0x02 (200 MeV)
DSSD 1 n+n : 0xF (150 keV) / 0xFF (disabled) / 0x02 (200 MeV)

ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:07:47|
Finished copying zImage to RAM 
07:04:24/13:07:48|
Found 0 errors checking kernel image
07:04:24/13:07:49|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
07:04:24/13:07:49|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
07:04:24/13:07:49|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
07:04:24/13:07:49|SPIconfig done now to set up the LMK3200 registers
07:04:24/13:07:49|LMK #0 : regInit[0]=0x80000000
07:04:24/13:07:49|LMK #0 : regInit[1]=0x10070600
07:04:24/13:07:49|LMK #0 : regInit[2]=0x60601
07:04:24/13:07:50|LMK #0 : regInit[3]=0x60602
07:04:24/13:07:50|LMK #0 : regInit[4]=0x60603
07:04:24/13:07:50|LMK #0 : regInit[5]=0x70624
07:04:24/13:07:50|LMK #0 : regInit[6]=0x70605
07:04:24/13:07:50|LMK #0 : regInit[7]=0x70606
07:04:24/13:07:50|LMK #0 : regInit[8]=0x70627
07:04:24/13:07:50|LMK #0 : regInit[9]=0x10000908
07:04:24/13:07:50|LMK #0 : regInit[10]=0xA0022A09
07:04:24/13:07:50|LMK #0 : regInit[11]=0x82800B
07:04:24/13:07:50|LMK #0 : regInit[12]=0x28C800D
07:04:24/13:07:50|LMK #0 : regInit[13]=0x830020E
07:04:24/13:07:50|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
07:04:24/13:07:50|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
07:04:24/13:07:50|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
07:04:24/13:07:50|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal not completed 
ADC calibrate failed 
Jumping to kernel simpleboot...
07:04:24/13:07:51|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
07:04:24/13:07:54|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
07:04:24/13:08:00|Using Xilinx Virtex440 machine description
07:04:24/13:08:01|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
07:04:24/13:08:01|Zone PFN ranges:
07:04:24/13:08:01|  DMA      0x00000000 -> 0x00007000
07:04:24/13:08:01|  Normal   0x00007000 -> 0x00007000
07:04:24/13:08:01|Movable zone start PFN for each node
07:04:24/13:08:01|early_node_map[1] active PFN ranges
07:04:24/13:08:01|    0: 0x00000000 -> 0x00007000
07:04:24/13:08:01|MMU: Allocated 1088 bytes of context maps for 255 contexts
07:04:24/13:08:01|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
07:04:24/13:08:02|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
07:04:24/13:08:02|PID hash table entries: 512 (order: 9, 2048 bytes)
07:04:24/13:08:02|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
07:04:24/13:08:02|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
07:04:24/13:08:02|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
07:04:24/13:08:02|Kernel virtual memory layout:
07:04:24/13:08:02|  * 0xffffe000..0xfffff000  : fixmap
07:04:24/13:08:02|  * 0xfde00000..0xfe000000  : consistent mem
07:04:24/13:08:02|  * 0xfde00000..0xfde00000  : early ioremap
07:04:24/13:08:02|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
07:04:24/13:08:02|NR_IRQS:512
07:04:24/13:08:02|clocksource: timebase mult[a00000] shift[22] registered
07:04:24/13:08:02|Console: colour dummy device 80x25
07:04:24/13:08:02|Mount-cache hash table entries: 512
07:04:24/13:08:02|NET: Registered protocol family 16
07:04:24/13:08:02|PCI: Probing PCI hardware
07:04:24/13:08:02|bio: create slab <bio-0> at 0
07:04:24/13:08:02|NET: Registered protocol family 2
07:04:24/13:08:02|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
07:04:24/13:08:02|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
07:04:24/13:08:03|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
07:04:24/13:08:03|TCP: Hash tables configured (established 4096 bind 4096)
07:04:24/13:08:03|TCP reno registered
07:04:24/13:08:03|NET: Registered protocol family 1
07:04:24/13:08:03|ROMFS MTD (C) 2007 Red Hat, Inc.
07:04:24/13:08:03|msgmni has been set to 214
07:04:24/13:08:03|io scheduler noop registered
07:04:24/13:08:03|io scheduler anticipatory registered
07:04:24/13:08:03|io scheduler deadline registered
07:04:24/13:08:03|io scheduler cfq registered (default)
07:04:24/13:08:03|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
07:04:24/13:08:03|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
07:04:24/13:08:03|console [ttyS0] enabled
07:04:24/13:08:03|brd: module loaded
07:04:24/13:08:03|loop: module loaded
07:04:24/13:08:03|Device Tree Probing 'ethernet'
07:04:24/13:08:03|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
07:04:24/13:08:03|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
07:04:24/13:08:03|XLlTemac: DCR address: 0x80
07:04:24/13:08:03|XLlTemac: buffer descriptor size: 32768 (0x8000)
07:04:24/13:08:03|XLlTemac: Allocating DMA descriptors with kmalloc
07:04:24/13:08:03|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
07:04:24/13:08:04|XTemac: PHY detected at address 7.
07:04:24/13:08:04|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
07:04:24/13:08:04|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04|Using buffer write method
07:04:24/13:08:04|cfi_cmdset_0001: Erase suspend on write enabled
07:04:24/13:08:04|cmdlinepart partition parsing not available
07:04:24/13:08:04|RedBoot partition parsing not available
07:04:24/13:08:04|Creating 5 MTD partitions on "fc000000.flash":
07:04:24/13:08:04|0x000000000000-0x000000500000 : "golden_firmware"
07:04:24/13:08:04|0x000000500000-0x000000800000 : "golden_kernel"
07:04:24/13:08:04|0x000000800000-0x000000d00000 : "user_firmware"
07:04:24/13:08:04|0x000000d00000-0x000000fe0000 : "user_kernel"
07:04:24/13:08:04|0x000000fe0000-0x000001000000 : "env_variables"
07:04:24/13:08:05|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
07:04:24/13:08:05|SPI: XIlinx spi: bus number now 32766
07:04:24/13:08:05|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
07:04:24/13:08:05|SPI: XIlinx spi: bus number now 32765
07:04:24/13:08:05|mice: PS/2 mouse device common for all mice
07:04:24/13:08:05|Device Tree Probing 'i2c'
07:04:24/13:08:05| #0 at 0x81600000 mapped to 0xD1030000, irq=22
07:04:24/13:08:05|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
07:04:24/13:08:05|TCP cubic registered
07:04:24/13:08:05|NET: Registered protocol family 17
07:04:24/13:08:05|RPC: Registered udp transport module.
07:04:24/13:08:05|RPC: Registered tcp transport module.
07:04:24/13:08:05|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:08:06|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:08:06|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:08:06|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:08:08|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:08:08|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:08:08|Sending DHCP requests .
07:04:24/13:08:10|eth0: XLlTemac: PHY Link carrier lost.
07:04:24/13:08:10|..... timed out!
07:04:24/13:09:39|IP-Config: Reopening network devices...
07:04:24/13:09:39|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:09:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:09:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:09:40|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:09:42|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:09:42|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:09:42|Sending DHCP requests ...... timed out!
07:04:24/13:11:15|IP-Config: Auto-configuration of network failed.
07:04:24/13:11:15|Root-NFS: No NFS server available, giving up.
07:04:24/13:11:15|VFS: Unable to mount root fs via NFS, trying floppy.
07:04:24/13:11:15|VFS: Cannot open root device "nfs" or unknown-block(2,0)
07:04:24/13:11:15|Please append a correct "root=" boot option; here are the available partitions:
07:04:24/13:11:15|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
07:04:24/13:11:15|Call Trace:
07:04:24/13:11:15|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
07:04:24/13:11:15|[c6827f10] [c00345bc] panic+0x94/0x168
07:04:24/13:11:15|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
07:04:24/13:11:15|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
07:04:24/13:11:15|[c6827fd0] [c0341220] kernel_init+0x104/0x130
07:04:24/13:11:15|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
07:04:24/13:11:15|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:14:15|
Finished copying zImage to RAM 
07:04:24/13:14:16|
Found 0 errors checking kernel image
07:04:24/13:14:17|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
07:04:24/13:14:17|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
07:04:24/13:14:17|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
07:04:24/13:14:17|SPIconfig done now to set up the LMK3200 registers
07:04:24/13:14:17|LMK #0 : regInit[0]=0x80000000
07:04:24/13:14:18|LMK #0 : regInit[1]=0x10070600
07:04:24/13:14:18|LMK #0 : regInit[2]=0x60601
07:04:24/13:14:18|LMK #0 : regInit[3]=0x60602
07:04:24/13:14:18|LMK #0 : regInit[4]=0x60603
07:04:24/13:14:18|LMK #0 : regInit[5]=0x70624
07:04:24/13:14:18|LMK #0 : regInit[6]=0x70605
07:04:24/13:14:18|LMK #0 : regInit[7]=0x70606
07:04:24/13:14:18|LMK #0 : regInit[8]=0x70627
07:04:24/13:14:18|LMK #0 : regInit[9]=0x10000908
07:04:24/13:14:18|LMK #0 : regInit[10]=0xA0022A09
07:04:24/13:14:18|LMK #0 : regInit[11]=0x82800B
07:04:24/13:14:18|LMK #0 : regInit[12]=0x28C800D
07:04:24/13:14:18|LMK #0 : regInit[13]=0x830020E
07:04:24/13:14:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
07:04:24/13:14:18|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
07:04:24/13:14:18|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
07:04:24/13:14:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
07:04:24/13:14:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
07:04:24/13:14:22|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
07:04:24/13:14:28|Using Xilinx Virtex440 machine description
07:04:24/13:14:29|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
07:04:24/13:14:29|Zone PFN ranges:
07:04:24/13:14:29|  DMA      0x00000000 -> 0x00007000
07:04:24/13:14:29|  Normal   0x00007000 -> 0x00007000
07:04:24/13:14:29|Movable zone start PFN for each node
07:04:24/13:14:29|early_node_map[1] active PFN ranges
07:04:24/13:14:29|    0: 0x00000000 -> 0x00007000
07:04:24/13:14:29|MMU: Allocated 1088 bytes of context maps for 255 contexts
07:04:24/13:14:29|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
07:04:24/13:14:29|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
07:04:24/13:14:30|PID hash table entries: 512 (order: 9, 2048 bytes)
07:04:24/13:14:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
07:04:24/13:14:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
07:04:24/13:14:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
07:04:24/13:14:30|Kernel virtual memory layout:
07:04:24/13:14:30|  * 0xffffe000..0xfffff000  : fixmap
07:04:24/13:14:30|  * 0xfde00000..0xfe000000  : consistent mem
07:04:24/13:14:30|  * 0xfde00000..0xfde00000  : early ioremap
07:04:24/13:14:30|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
07:04:24/13:14:30|NR_IRQS:512
07:04:24/13:14:30|clocksource: timebase mult[a00000] shift[22] registered
07:04:24/13:14:30|Console: colour dummy device 80x25
07:04:24/13:14:30|Mount-cache hash table entries: 512
07:04:24/13:14:30|NET: Registered protocol family 16
07:04:24/13:14:30|PCI: Probing PCI hardware
07:04:24/13:14:30|bio: create slab <bio-0> at 0
07:04:24/13:14:30|NET: Registered protocol family 2
07:04:24/13:14:30|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
07:04:24/13:14:30|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
07:04:24/13:14:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
07:04:24/13:14:31|TCP: Hash tables configured (established 4096 bind 4096)
07:04:24/13:14:31|TCP reno registered
07:04:24/13:14:31|NET: Registered protocol family 1
07:04:24/13:14:31|ROMFS MTD (C) 2007 Red Hat, Inc.
07:04:24/13:14:31|msgmni has been set to 214
07:04:24/13:14:31|io scheduler noop registered
07:04:24/13:14:31|io scheduler anticipatory registered
07:04:24/13:14:31|io scheduler deadline registered
07:04:24/13:14:31|io scheduler cfq registered (default)
07:04:24/13:14:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
07:04:24/13:14:31|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
07:04:24/13:14:31|console [ttyS0] enabled
07:04:24/13:14:31|brd: module loaded
07:04:24/13:14:31|loop: module loaded
07:04:24/13:14:31|Device Tree Probing 'ethernet'
07:04:24/13:14:31|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
07:04:24/13:14:31|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
07:04:24/13:14:31|XLlTemac: DCR address: 0x80
07:04:24/13:14:31|XLlTemac: buffer descriptor size: 32768 (0x8000)
07:04:24/13:14:31|XLlTemac: Allocating DMA descriptors with kmalloc
07:04:24/13:14:31|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
07:04:24/13:14:32|XTemac: PHY detected at address 7.
07:04:24/13:14:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
07:04:24/13:14:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32|Using buffer write method
07:04:24/13:14:32|cfi_cmdset_0001: Erase suspend on write enabled
07:04:24/13:14:32|cmdlinepart partition parsing not available
07:04:24/13:14:32|RedBoot partition parsing not available
07:04:24/13:14:32|Creating 5 MTD partitions on "fc000000.flash":
07:04:24/13:14:32|0x000000000000-0x000000500000 : "golden_firmware"
07:04:24/13:14:32|0x000000500000-0x000000800000 : "golden_kernel"
07:04:24/13:14:32|0x000000800000-0x000000d00000 : "user_firmware"
07:04:24/13:14:32|0x000000d00000-0x000000fe0000 : "user_kernel"
07:04:24/13:14:32|0x000000fe0000-0x000001000000 : "env_variables"
07:04:24/13:14:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
07:04:24/13:14:33|SPI: XIlinx spi: bus number now 32766
07:04:24/13:14:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
07:04:24/13:14:33|SPI: XIlinx spi: bus number now 32765
07:04:24/13:14:33|mice: PS/2 mouse device common for all mice
07:04:24/13:14:33|Device Tree Probing 'i2c'
07:04:24/13:14:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
07:04:24/13:14:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
07:04:24/13:14:33|TCP cubic registered
07:04:24/13:14:33|NET: Registered protocol family 17
07:04:24/13:14:33|RPC: Registered udp transport module.
07:04:24/13:14:33|RPC: Registered tcp transport module.
07:04:24/13:14:33|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:14:34|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:14:36|Sending DHCP requests ., OK
07:04:24/13:14:37|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.6
07:04:24/13:14:37|IP-Config: Complete:
07:04:24/13:14:37|     device=eth0, addr=192.168.11.6, mask=255.255.255.0, gw=255.255.255.255,
07:04:24/13:14:37|     host=aida06, domain=dl.ac.uk, nis-domain=nuclear.physics,
07:04:24/13:14:37|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06
07:04:24/13:14:37|Looking up port of RPC 100003/2 on 192.168.11.99
07:04:24/13:14:37|Looking up port of RPC 100005/1 on 192.168.11.99
07:04:24/13:14:37|VFS: Mounted root (nfs filesystem) on device 0:12.
07:04:24/13:14:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
07:04:24/13:14:38|Starting sysinit...
07:04:24/13:14:38|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
07:04:24/13:14:39|		Press 'I' to enter interactive startup.
07:04:24/13:14:39|Setting clock  (utc): Sun Apr  7 12:14:39 BST 2024 [  OK  ]
07:04:24/13:14:39|Building the cache [  OK  ]
07:04:24/13:14:39|Setting hostname aida06:  [  OK  ]
07:04:24/13:14:41|Mounting local filesystems:  [  OK  ]
07:04:24/13:14:42|Enabling /etc/fstab swaps:  [  OK  ]
07:04:24/13:14:45|Finishing sysinit...
INIT: Entering runlevel: 3
07:04:24/13:14:48|Entering non-interactive startup
07:04:24/13:14:48|FATAL: Module ipv6 not found.
07:04:24/13:14:49|Bringing up loopback interface:  [  OK  ]
07:04:24/13:14:52|FATAL: Module ipv6 not found.
07:04:24/13:14:53|Starting system logger: [  OK  ]
07:04:24/13:14:54|Starting kernel logger: [  OK  ]
07:04:24/13:14:54|Starting rpcbind: [  OK  ]
07:04:24/13:14:56|Mounting NFS filesystems:  [  OK  ]
07:04:24/13:14:56|Mounting other filesystems:  [  OK  ]
07:04:24/13:14:56|Starting xinetd: [  OK  ]
07:04:24/13:14:57|Starting midas:  Starting MIDAS Data Acquisition for aida06
07:04:24/13:14:58|xaida: device parameters: base=0x81000000 size=0x200000
07:04:24/13:15:03|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
07:04:24/13:15:03|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
07:04:24/13:15:03|xaida: driver assigned major number 254
07:04:24/13:15:04|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
07:04:24/13:15:09|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
07:04:24/13:15:09|AIDAMEM: aidamem: driver assigned major number 253
07:04:24/13:15:09|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
07:04:24/13:15:16|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
07:04:24/13:15:16|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
07:04:24/13:15:16|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
07:04:24/13:15:16|Computer Name = aida06; Temp Directory = /tmp/tcl361
07:04:24/13:15:21|
07:04:24/13:15:25|AIDA Data Acquisition Program Release 10.0.Jul  6 2022_15:10:57 starting
07:04:24/13:15:25|
07:04:24/13:15:25|Built without pthreads
07:04:24/13:15:25|
07:04:24/13:15:25|Creating NetVars
07:04:24/13:15:25|Output buffer length = 65504; format option = 4; transfer option = 3
07:04:24/13:15:25|EB transfer option = 3
07:04:24/13:15:26|NetVars creaxaida: open:
07:04:24/13:15:26|ted and initialised
07:04:24/13:15:26|StatisticsAIDAMEM: aidamem_open:
07:04:24/13:15:26| thread starting
07:04:24/13:15:26|Data Acquisition task has PID 375
07:04:24/13:15:26|Statistics thread created
07:04:24/13:15:26|Stat/Rate creation thread starting
07:04:24/13:15:26|Stat/Rate creation thread created
07:04:24/13:15:26|Hit/Rate creation thread starting
07:04:24/13:15:26|Hit/Rate creation thread created
07:04:24/13:15:26|AIDA Heartbeat thread starting
07:04:24/13:15:26|Heartbeat thread created
07:04:24/13:15:26|Installing signal handlers
07:04:24/13:15:26|Done
07:04:24/13:15:26|ModuleNum = 0
07:04:24/13:15:26|Aida Initialise complete. AidaExecV10.0: Build Jul  6 2022_15:10:57. HDL version : 03350706 
07:04:24/13:15:26|package limit is not available: can't find package limit
07:04:24/13:15:26|Running with default file descriptor limit
07:04:24/13:15:26|Spectra table initialised
07:04:24/13:15:26|AIDA Data Acquisition now all ready to start
07:04:24/13:15:26|SIGBUS, SIGSEGV and SIGPIPE traps setup
07:04:24/13:15:26|package setuid is not available: can't find package setuid
07:04:24/13:15:27|Running as user 0 group 0
07:04:24/13:15:27|[  OK  ]
07:04:24/13:15:36|
07:04:24/13:15:36|DENX ELDK version 4.2 build 2008-04-01
07:04:24/13:15:36|Linux 2.6.31 on a ppc
07:04:24/13:15:36|
07:04:24/13:15:36|aida06 login: /debug user

13.00 Cooling water temperature and flow OK - attachment 1

      Test of AIDA 2x MSL type BB18(DS)-1000 24cm x 8cm DSSSDs 'as is' ( to be defined later in this Elog )

      DSSSD bias & leakage current OK - attachment 2
      Ambient temp 24.7 deg C, d.p. 7.1 deg C, RH 32.4%
      Leakage current c. 6.6uA => 4nA/cm2/100um ( very good )

      FEE64 temps OK - attachment 3
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder status - attachment 4

      WR timestamps OK - attachment 5

      WR (info code 4 & 5), correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 6-12

      per FEE64 Rate spectra - attachments 13-15

      per 1.8.W spectra - 20us FSR - attachments 16-1710129


14.45 FEE64 config check - per https://elog.ph.ed.ac.uk/DESPEC/562


FEE64 # PSU cable #           MAC
1       1                     ?
2       2                     ?
3       3                     ?
4       4                     41 ee 71
5       5                     ?
6       6                     ?
7       7                     f6 5a
8       8                     41 d7 cd
9       ?                     ?
10      10                    41 d0 0e
11      11                    41 ee 0f
12      ?                     ?
13      13                    ?
14      ?                     0d 15
15      15                    ?
16      16                    f6 ed

? = no line of sight

- AIDA FEE64 PSU cabling - see attachment 18

Currently

PSU #1 1-3, 2-4, 9-5, 15-12
PSU #2 14-7, 6-8, 10-13, 11-16

Should be changed to

PSU #1 1-3, 2-4, 9-15, 5-12
PSU #2 14-7, 6-8, 10-11, 13-16


- AIDA snout mount to support frame - attachments 19 & 21
LHS ( looking downstream ) incorrect, RHS OK

- LKs
LK1 fitted aida02, aida04, aida06, aida08
LK3 fitted aida03, aida07, aida01, aida14


Significant gaps in snout ( i.e. possible light leaks ) where bPlas cabling exits snout - attachments 20 & 22

bPlas driver PCBs removed 
bPlas ribbon cables, drain wires and Lemo 00.250 cabling disconnected - some cabling/drain wires touching snout etc


15.15 Adjust bPlas ribbon cables/drain wires/Lemo cables to ensure that none are touching snout or AIDA support frame - significant improvement

      DSSSD bias & leakage current OK - attachment 23

      FEE64 temps OK - attachment 24

      ADC data item stats - attachments 25

      per FEE64 Rate spectra - attachments 26
       6x FEE64s < 20k. all FEE64s < 120k ( cf. 4x FEE64s < 20k, 5x FEE64s > 100k, 1x FEE64 320k earlier - attachment 12 )

      per 1.8.W spectra - 20us FSR - attachments 27-28

17.15

LHS top hat washers
LK1, LK3
PSU power cables
tighten gnd screws
attachments 29-34

Note that some of the newly installed FEE64s have the incorrect rails fitted - FEE64s may ground to AIDA support stand

17.55 CAEN N1419ET LK removed -> floating outputs

18.15 +LK1 restored
attachments 35-40

18.40 +pwwer cycle
attachments 41-46

Changed /MIDAS/Linux/startup/NewMerger

Change parameters -i and -l in master64 to 16 for 16 FEEs
Update NewMerger Options LinksAvailable to 16, LinksInUse to 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Fix NetVar RunOptions 1 (was 0)

Restart Merger HTTPd, Tape, Merger, MBS Spy

Reset/Setup/Go 

16 Links green and status going, all good?

Bias DSSSDs and turn data transfer on

Merger connected, shows rate and updates... no rate in Tape Server?

.. Oops forget to turn on Output to data storage in merger!

Rate in tape server and to MBS: 7 MB/s

Merger, Tape and MBS working with 16 FEEs

The WR error counter for aida02 seems to consantly rise

Tried reseating cable on both ends, no change

However clock status passed, aida02 has a correct WR timestamp and no FIFO/PLL errors seen

Edit to add: aida02 has the faulty ASIC temperature readout as well, related or coincidence?

When booting up AIDA aida06 usually crashes the first time, it fails to get IP from DHCP
After 180 seconds it reboots and seems to connect fine

Log file attached, key part (to me) is this:

27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|

S505 ADC offsets using pulser walkthrough data from data file R1

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

Mounted on frame:

DSSD 1 (Upstream) :     3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22

Current mapping has been redone for better management.

Going to try optimising noise now.

DHCP updated

new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)

New layouts: /home/npg/LayOut/GSI_Triple_S100

New layout.txt

Firmware of aida11 updated from 0xea40704 to 0x3350706

Temps GOOD fig 5

Rates fig 6, 7

Check adapter alignment aida14 and aida16

bPlas left/right cables are not insulated and shorting to the snout

Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor

From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!

White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)

Turn off bPlas
We see the noise drop a lot

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly 

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

Sensor ( 4 pins )

Red  +24V
Blue   0V
Yellow } contact 
Green  } closure in ( short these two wires together for logic 1 )

N.B. yellow and green wires of unused inputs must be connected together


Output ( 3 pins ) to USB-controlled ac mains relay

Red    NC ( normally closed - with respect to COM with no power applied, single pole double throw relay out )
Blue   NO ( normally open - with respect to COM with no power applied, single pole double throw relay out )
Green COM

Taken 4 FEE64s from CRYRING (the 4 easiest to access)
  41:d8:2b
  41:f6:5a
  41:ee:71
  41:d0:0e

In addition a FEE without rails marked only 20 was found in my old office for 5
- This probably came in a shipment to/for repairs

DSSSD#1 unbiased

DESPEC Platform moved out of beam and gamma platform moved from AIDA
JB works on grounding the original 8 DSSSD#1 FEEs

Connected grounding from ribbon cable drain to adapter PCB
Ground loop connected

AIDA rates unchanged bad still fig 1
Waves fig 2 (p+n 6000-9000)
Waves fig 3 (n+n 8000-10000)
Statistics fig 4

09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39

Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Anydesk address now restored to 832827869

Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference

> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.

The following messages are in the system log very often:

Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk)

Turn on AIDA for Dry Run demonstrations and so on

All system wide checks, temp, bias OK

Noise situation is dreadful (but has not been optimised). Deterioriation since first mounted, suspect cabling issues with bPlast and BB7.

 Note thresholds at 0x32 (!!!) to not brutalise the DAQs during testing

 aida08 seems OK

Server running to MBS totally fine

18:00

Carole grounded some of the Bplast and this reduced the rates in AIDA, although they are a bit fluctuatey. Due to position constraints she couldn't ground it all
Also AIDA ribbon cables are not grounded yet
The indication is these fixes should make a lot of difference to the situation

AIDA is now powered off for the end of day

AIDA has 8 FEEs and 1 DSSSD 

Aida08 (HDMI#12) had no WR again, I moved it to a different MACB and now it gets WR
The MACB (currently with jsut HDMI#10) seems issues, check/replace the upstream HDMI and thent eh MACB (after dry run!)
 
DSSSD#1 biased to -120 V, leakage current 5.6 uA (fig 1)
 (also on Grafana)

Temps OK fig2

System Checks (fig3-5)
 Clocks OK
 aida07 fails calibration, others OK
 WR OK (aida04 0x10, seems OK)
 FPGA OK

Did "Synchronise ASIC clocks" to align ASIC clocks

(Notes for scalers: SC41L HDMI 5, SCI41R HDMI 9)

18:12 FRS is taking beam, AIDA is powered off and unbiased

In preparation for the dry run the following *temporary* changes to the FEE numbering have been prepared
These should be reverted after the dry run to ensure cable->fee agreement again

AIDA09 => AIDA06
AIDA11 => AIDA07
AIDA12 => AIDA08

This will allow the merger to run with 8 FEEs for 1 DSSD

dhcpd.conf updated

The 2023Oct19-13.46.30 should work with this numbering (check tomorrow)

As should layout GSI_triple_test_renumber 

AFTER dry run:

Revert DHCP and prepare for full 16 FEEs 

Make new ASIC settings key for 16 FEEs and prepare the aidaXX folders

Prepare a new Layout.mlf set

The behaviour of the DSSSD leakage current at low voltages and during biases is unusual and varies depending on how the adapter boards are connected
To summarise the behaviour I have observed

Minimum bias configuration:
 4 adapter boards, one n+n (LK1), three p+n (-ve bias), ground from n+n to one p+n
 Voltage (and leakage current) unstable at low voltages, seems to settle at around -60 V
 Drops can include 0 leakage current

Full adapter configuration:
 8 adapter boards, ground ring complete
 Same as minimum, but the drops seem to be much smaller (and not to 0 leakage current)
 -60V again seems to be the turnover to a stable leakage current 

In both cases the leakage current during ramping appears basically the same as when settled

Full into FEEs
 8 adapter boards, fully connected to 8 FEEs
 The leakage current is *much* higher during ramp,up to 17 uA near the end. No fluctuations
 Once ramping has finished the current quickly drops back down and settles at the nominal leakage current
 This has been observed in October/December too, it is not new (https://elog.ph.ed.ac.uk/AIDA/910)
 During power up of the FEEs the current sometimes drops briefly (when the ASICs get programmed, I believe)


I think it is related to the ground (more or less current flowing through the HV supply instead of alternate paths?)
It should be kept in mind when testing new detectors to not worry about the detector at low voltages

new Downstream DSSD2: 3208-2/3208-5/3208-8

Covered with black cloth.

Wafer 3 (beam left) shows breakdown issues at 90V... curious

We replace with november beam triple: 3208-3/3208-21/3208-22

We must speak to Micron about why so many 3208 wafers seem to have issues at >90V
It seems unlikely to be other issues

Upstream bPlast mounted

new Upstream DSSD: 3208-2/3208-5/3208-8

Covered with black cloth.

V-I behaviour found to be nominal.

Dismounted Snout and biased DSSSD1 channels

leftmost waifer working

middle waifer reaches 90V then current ramps up

rightmost waifer reaches 80V then current ramps up

DSSSD2 had fingerprint near connections, as shown in the attached image some had been squashed.

DSSD#1 undergoes a breakdown at 90V, two of the three wafers show this
- The adapter PCBs themselves have no breakdown at 100V, indicating the   issue is internal to the snout

If we are lucky it may be a loose or misaligned kapton connector inside the snout. If not we will have to remove DSSD#2 and inspect DSSD#1 for damage/lint/etc. If we see nothing it should be replaced

DSSD#2 biases perfectly fine, but the leakage current is unstable with biasing -ve to p+n and gnd (return) to n+n. Leakage current is stable biasing +ve to n+n and gnd to p+n. (https://elog.ph.ed.ac.uk/DESPEC/240311_093933/Downstream_positive_bias_vs._Current_(uA).png very nice curve)

We see the same fluctuations just daisy chaining 3 p+n PCBs togerher with the bias (no DSSSD or gnd links). The fluctuations were reduced by connecting all 8 adapter boards of the DSSD together.
When biasing just *one* PCB in this way, the current is stable
The fluctuations happened changing the HV channel, adapter board PCB and LEMO cables: it doesn't seem to be a defect with a specific thing.

The test today confirmed the HV cables (the only ones used) were correctly isolated from everything (OL = "infinite" resistance to ground)

We saw unstable current indications using the Mesytec MHV-4 to apply voltage to the PCB as well.

The behaviour is odd but doesn't seem to be related to the DSSD, which I suspect is OK. One idea is to try connecting the 3 adapter boards to FEEs and repeating the test, as this introduces another (substantial) path to gnd. Maybe this eliminates the current instability? It's about the only difference left from December.

-

For the next steps I believe the snout must be dismounted to inspect DSSD#1. It would be best to coordinate this with replacing the broken(?) bPlas. If we are lucky we may not have to remove the DSSDs but it is likely we have to remove both DSSDs to swap out #1

After replacing the DSSD(s) we should cover the snout with a clean black bin bag and bias it on the MH table to confirm both detectors work. This saves the effort of carrying it to S4 just to find another problem.
If both detectors make it to 120V we can mount it again more confidently

Once we have two biased detectors we can rearrange some FEEs to get 8 for DSSD#1 to do the noise tests. I suggest we wire them up for the numbering plan in https://elog.ph.ed.ac.uk/DESPEC/532 but in the DHCP renumber 9,10,15,16 to 5,6,7,8 temporarily to allow data to be sent to MBS for the dry run (merger limitation)

When the remaining FEEs are recovered from UK+CRYRING we can instrument DSSD#2 and renumber the FEEs to match the cables

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

 - When the p+n ribbon cables were disconnected for both the upstream and downstream AIDA, the pins and ribbon cables did not appear to have any blemishes.

- Both sets of adaptor PCBs were biased to 100 V and only had an apparent fluctuation in current at c. 0.01 uA. At low voltages (~10V) the current cycled between 0 and 1 uA.

2) Check that all ribbon cables are properly seated in the adaptor PCBs

- When inspected all ribbon cables seemed to be properly seated. A decent press was applied to confirm the seating. The PCBs themselves were inspected and no noticeable damage was observed.

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

- Done for the p+n side, can decide if this is needed for the n+n side.

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

-  Will have to decide further.

5) From email discussion: Check the SHV connector is grounded if one unplugs the cable from the back of the HV module. If yes: We are touching a ground somewhere. If no: it's via the HV (fine).

- When HV#0 was disconnected from the HV module, we used a multimeter on the SHV connecter and frame and read zero resistance 0L on the meter. While when completing the circuit for HV#1 (still) plugged into the HV module we read a resistance of > 1.7 Ohm.

- The HV is touching ground somewhere. This should not be a problem however as the snout is isolated, and this was observed by connecting the SHV to the snout and reading 0L.

6) Properly cover snout with black cloth and bias upstream.

- Snout was covered with black cloth and a black bag. The downstream detector p+n side was biased at 50V and charging/discharging was observed. This is probably due to a short connection somewhere.

Summary: from the p+n side test the voltage-current break down appears to come from inside the snout.

Proposed FEE numbering and wiring plan for upcoming experiment S100 (2x Wide DSSSDs)

Image designed in draw.io, source attached 

FEE numbering is as S450, minimises cable movement from S505/Narrow AIDA
But means merger is not working with 1 DSSD (until all FEEs installed)

Wiring of adapter boards as from noise tests and what should work for DSSD bias
LK3 on middle bottom adapter to ground DSSD
LK1 on one n+n adapter to ground n+n side bias 
p+n has -ve voltage (w.r.t. ground) bias applied via lower adapter boards
ground loop grounds all adapter boards, except 2 p+n adapter boards which are grounded by the bias lemo shield instead

MACB layout also included, with expected NIM logic signals for the aida scalers:

1: Pulser/Sync clock (send to all subsystems, "trigger 3")0

3/4: Time Machine 
5/6: SC41L/R 

All other FEEs have their scaler available
(Scaler should be in left LEMO on MACB, right is output (AIDA->NIM/unused), bottom 4 are triggers from AIDA (unused)

Test circuit will not be used in experiment due to noise, but can be temporarily set up for pulser walkthrough

Revision 2 correct as of 27 March 2023

Bias tests of AIDA on individual wafers and in parallel. Spreadsheet can be found in attachment 1, and graphic results can be seen in Attachment 2. Summary: Upstream detector cannot be biased in parallel nor individually. Downstream detector can be biased with positively and each wafer can individually be biased negatively. In parallel, it was not possible to bias the downstream detector negatively as indicated by the tests below.

Test downstream DSSSD with positive polarity bias

Configuration as follows:

CAEN N1419ET ch #3 connected to LHS FEE64 adaptor PCB ( looking upstream ) - LK1 *not* fitted LK1 fitted to 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cables from/to GND terminals of 3x ( top )

FEE64 adaptor PCBs *and* LHS FEE64 adaptor PCB 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything, LK3 fitted Total 5x adaptor PCBs installed

No other LKs fitted

Nominal V-I curve, stable leakage current. Attachment 3.

Following this success we attempted to repeat test using negative polarity bias

Configuration as follows:

CAEN N1419ET ch #1 connected to ( top, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cable from/to GND terminals of LHS and ( top, left )

FEE64 adaptor PCBs 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything,

LK3 fitted Total 5x adaptor PCBs installed No other LKs fitted With detector bias

-20V we continue to observe the leakage current cycling between 0 and ~2uA with a frequency ~1Hz ( as before )

Copy configuration used for upstream DSSSD test ( which was successful albeit there was detector breakdown at bias voltages > c. 90V )

CAEN N1419ET ch #1 connected to ( bottom, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( bottom )

FEE64 adaptor PCBs Ground cable jumpered from/to GND terminals LHS, ( left, bottom ), RHS and all 3x top FEE64 adaptor PCBs ( bottom, middle )

FEE64 adaptor PCB LK3 fitted Total 8x adaptor PCBs installed

No other LKs fitted With detector bias -20V we observe leakage current of ~2-3uA.

Current unstable - variations 10-100nA over periods of several seconds Although the leakage current is unstable this is an improvement over previous tests with negative bias. The duplication of upstream and downstream configurations suggests that for some unknown reason it is necessary to connect all 8x FEE64 adaptor PCBs whereas our expectation was that only 4x were necessary.

Summary: Upstream DSSSD Si wafers 1 & 2 breakdown for bias > c. 90V Si wafer 3 OK to 120V Positive bias - not tested Negative bias OK - leakage current stable to c. 90V Downstream DSSSD Si wafers 1, 2 & 3 OK to 120V Positive bias OK Negative bias - leakage current unstable

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

2) Check that all ribbon cables are properly seated in the adaptor PCBs

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

- Voltage-Current test of newly installed AIDA snout. Breakdown observed at around ~92 V. Probably caused my light leakage into the snout, will investigate further.

- Tried to cover the snout with a black cloth. This did not change the breakdown behaviour.

Snout assembly

DSSSDs

Upstream   3208-10/3208-18/3208-20
Downstream 3131-5/3131-10/3131-12

p+n junction side bias continuity checked OK for *all* wafers
 all c. 22 Ohm as expected

n+n Ohmic side bias continuity checked OK for both DSSSDs
 upstream 73 Ohm, downstream 92 Ohm

n+n Ohmic side bias wafer #2 
 positioned at top of snout assembly ( lower stage snout side marked 'T' )

p+n junction side faces downstream


Distances

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.0cm
ref - upstream DSSSD 11.0cm
ref - downstream DSSSD 12.0cm
ref - middle of downstream bPlas 14.0cm

Measurements consistent to +/-1mm between LHS and RHS of snout assembly as viewed from top side 


13.00 At this position the Kapton PCBs bulged in/out and there was some concern that when the upper stage
snout was installed it would the Kapton PCBs into the 'active' area. It was decided to move the 2x bPlas
and 2x DSSSDs c. 5mm downstream.

Jeroen and Phillip cut sections from 4x PEEK 6mm spacers so that they would clip onto 3mm dia support rods.

We raised the assembly fro the inter stage and inserted the modified spacers between the inter stage and 
the other spacers above. The Kapton PCBs now run fairly straight to the DSSSD connectors.

Distances re-measured as

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.6cm
ref - upstream DSSSD 11.5cm
ref - downstream DSSSD 12.4cm
ref - middle of downstream bPlas 14.5cm

Old ribbon cable assemblies - attachments 1  2

Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors

p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm

Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm



Manufacture of new ribbon cable assemblies - attachment 3

8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables


17.52 from Helena

"These are the detector positions within the snout:

Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame

Upstream AIDA @ 513 mm

Downstream AIDA @ 523 mm

This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is 
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by 
10mm.

The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position."

12.30 FEE64 41:d7:cd ASIC mezzanine u/s, no data, ASIC temperature c. 20 deg C ( ambient ) low
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida08

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps - attachment 1
       aida02 with new ASIC mezzanine continues to ramp to c. 512 deg C - initially reads 0, next refresh c. 70 and third refresh c. 512 deg C
       aida02 Virtex and PSU temps OK, aida02 cooling plate ambient to touch

       aida08 ASIC temp c. 45 deg C as expected, Virtex and PSU temps OK

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 13 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11

17.00 FEE64 41:a0:71 ASIC #2 u/s
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida04

      Unable to boot nnrpi2 - manual/local control of BNC PB-5 and FEE64 PSU - no interlock

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps OK - attachment 1

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 15 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11

      MIDAS configuration - attachment 12

      DHCP config - attachment 13



18.30 FEE64 41:f6:b7 ASIC #1 u/s
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida02

      All system wide checks OK

      FEE64 temps - attachment 14
       aida02 ASIC temperature c. 512 deg C !
       PSU/Virtex temps OK
       Mezzanine ambient to touch - assume faulty sensor/poor connection?

      aida02 ASIC settings - attachment 15

      WR timestamp OK - attachment 16

      ADC data item stats OK - attachment 4

      aida04 Rate spectrum - attachment 18

      aida02 1.8.W spectra - 20us FSR - attachment 19-20

      aida02 *.*.L spectra - attachments 21-24
       1.8.L pulser peak width 16 ch FWHM


19.30 Power OFF



     

First pass analysis of data file /TapeData/DEC23/R9_6

 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  234532726 (   99567.1 Hz)
        Other data format:   27387282 (   11626.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         53 (       0.0 Hz)
                               RESUME:         53 (       0.0 Hz)
                              SYNC100:      40775 (      17.3 Hz)
                              WR48-63:      40775 (      17.3 Hz)
                           FEE64 disc:    2928025 (    1243.0 Hz)
                             MBS info:   24377601 (   10349.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     287073 (     121.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2355.524 s

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   59013868     372005          0          0         20         20       9888       9888     281563      70626          0      55467
  1   31445079    1098714          0          0          2          2       5017       5017     243009     845667          0      38876
  2   22895499   11841669          0          0          1          1       5294       5294     482655   11348424          0      42748
  3   18228498   11705354          0          0          0          0       3913       3913     416619   11280909          0      78526
  4   25500772    1237178          0          0          3          3       4320       4320     396557     831975          0      28918
  5          4          0          0          0          0          0          0          0          0          0          0          0
  6   15933921     394210          0          0          0          0       2560       2560     389090          0          0      11489
  7   61515085     738152          0          0         27         27       9783       9783     718532          0          0      31049

FEE64 configuration https://elog.ph.ed.ac.uk/AIDA/935


Note

- aida06 not connected to DSSSD ( cabling broken  - will be replaced later this week )

- LEC ( 20MeV FSR ) data ADC offset corrected 
   ADC offsets available for 489 of 512 channels and are included in the analysis - 23 strips for which no ADC offsets could be calculated ( usually because there was 
no 
pulser data ) are not included in analysis 
- LEC ( 20MeV FSR ) front-back strip energy difference cut +/- 50 channels ( c. +/- 280keV )
- HEC ( 2GeV FSR ) front-back strip energy difference cut +/-200 channels ( c. +/- 1120MeV )

- LEC ADC data: 13 < channel < 188 ( c. 73-1053keV )
- HEC ADC data: > 13 channels ( c. 73MeV )
- FEE64 hardware thresholds: LEC c. 100keV, HEC c. 200MeV 




Attachments 1 & 2 - per pixel HEC-LEC event time difference spectra - 4.096us/channel

Attachments 3 & 4 - per pixel HEC-LEC event time difference spectra - 65.536us/channel

Attachment 5 - DSSSD x-y hit pattern: HEC-LEC event time difference < 4.3s
 Majority of events associated with 'hot' p+n junction strips - few plausible decay candidate events - as expected.
 z-scale - semi-logarithmic - scattered events are single counts ( blue )
 1 count => rate ~ 1/2355s ~ 0.0004Hz, the great majority of pixels have zero counts

Attachment 6 - LEC ( decay ) and HEC ( implant ) events - 262.144us/channel ( 65536 channels = 17.2s ) 

HEC implants channels 12000-25000 = 216 counts => HEC rate ( in spill ) 63.4Hz
LEC decays channels 12000-25000 = 3865 counts => LEC rate ( in spill ) 1134.1Hz
LEC decays channels 25000-38000 = 2790 counts => LEC rate ( inter spill ) 818.7Hz

Attachment 7 - per pixel HEC-HEC event time difference spectra - 4.096us/channel

Attachment 8 - per pixel LEC-LEC event time difference spectra - 4.096us/channel

Attachment 9 - variables.dat
This is a ( Fortran ) Namelist I/O data file containing of the ADC offsets, FEE64 configuration, LEC and HEC energy difference windows 

The ADC offset channel number is calculated as

channel = channel_ident + ( module * 64 ) where module = 0-7 corresponding to AIDA FEE64s aida01-aida08

and is used as follows

ADC data = INT ( RSHIFT( ABS( 32768 - data( i ) ), 3 ) - offset( i ) + 0.5 )

where data(i) is the ADC data item for channel i, offset(i) is the ADC offset for channel i

An ADC offset of -9999 means there was no pulser data for this channel in data files R5 and R11.

               

Attachment 1 - 4 socket input connectors ( from sensor to interlock box )

Attachment 2 & 2 - 24V dc relays and internal wiring

Attachment 4 - 3 socket input connector ( from interlock box to USB-controlled ac mains relay )

DSSSD MSL type BB18 24cm x 8cm 3208-3/3208-21/3208-22

FEE64 configuration see https://elog.ph.ed.ac.uk/AIDA/872 attachment 2

Bias -150V -6.590uA ambient temperature +24.7 deg C d.p. +13.7 deg C RH 50.3%

BNC PB-5
amplitude 10.0V
attenuation x10
decay time 1ms
tail pulse
frequency 25Hz

PB-5 output direct to p+n junction side FEE64 aida01 or aida12, or n+n Ohmic side FEE64 aida02


aida01 1.8.L pulser peak width  61 ch FWHM ~  46keV FWHM => 5s threshold 98keV 
aida12 1.8.L pulser peak width  56 ch FWHM ~  42keV FWHM => 5s threshold 89keV
aida02 1.8.L pulser peak width 102 ch FWHM ~  77keV FWHM => 5s threshold 163keV


slow comparator  0xa 100keV ( all p+n junction FEE64s ) 
                 0xf 150keV ( all n+n Ohmic FEE64s )

per FEE64 Rate spectra - attachment 1
 p+n FEE64s ( aida010, aida01, aida09, aida12, aida03, aida11 ) rates dominated by hot channels, other channels typically <1Hz ( 25Hz pulser to aida12 )
 n+n FEE64s ( aida02, aida04 ) rates ~ 10-20Hz/channel
 Note aida06 and aida08 are not connected to anything and should be ignored


ADC data item stats - attachment 2


For further information see https://elog.ph.ed.ac.uk/AIDA/906 and https://elog.ph.ed.ac.uk/AIDA/907


To Do

- repair/replace Honeywell HSS-DPS dew point sensor
   USB-controlled ac mains relay interlock currently overrriden
   do NOT operate AIDA unattended

- aida04 asic #1 u/s - replace ASIC mezzanine

- electrically isolated test signal distribution box req'd

- aida10 asic #4
   v. high rates observed and large signal transients 
   cause unclear ... ASIC/adaptor PCB/cabling/Si wafer ?

- extended background alpha run to check all DSSSD bond wires
   pulser OFF
   slow comparator 0x64

- bPlas + 2x triple DSSSD + bPlas stack assembly and test

- all up, in beam test

The photographs show the cooling water controls and temperature/pressure gauges outside S4 and the connections used by AIDA within S4.

The AIDA MBS FDR will be x86l-119 from now on, not x86l-94

the MBS relay and startup scripts will be changed for this

There was a power failure in the morning of 24.08.2023 in the Rhein-Main area affecting GSI

The Aida workstation (aida-3) has been restarted, it is unknown if the Pis in S4 rebooted as well (there is a UPS)

29.8.23 TD Both RPi systems rebooted four days ago.

Firefox browser proxy setting changed to 'Auto-detect proxy settings for this network' ( Firefox -> Edit -> Settings -> Network Settings -> Settings )

When using the VETAR connected to the HDMI port of the root MACB the settings for all the MACB in the system 0x3.

This setting takes the Clock and Data line from the input HDMI and outputs it via teh HDMI output ports.

When using the Emulator connected via the SMA connectors on the back of the root MACB then the setting for the root MACB should be 0xD and all others should be 0x3.

Attached is the .jed file for programming the MACB and the .vhd source file to help with understanding of the settings.

The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible

Investigating AIDA noise with a TA041 differential probe and oscilloscope 

AC Mains (DESPEC platform AC, L-N)
Probe attentuation = 1:100

Fig 1: Main AC waveform [X: 5ms/div, Y: 100 V/div]
Fig 2: Zoomed in at peak (20 V FSR, any less and the waveform clipped) [X: 10us/div, Y: 20V/div]
Fig 3: Longer time base and FFT of 0-5 MHz. No significant frequency harmonics noticed [X: 5ms/div, Y:20 V, FFT X: 500 kHz/div, Y: 10 dBm/div]

No significant noise or distortion present, fully within any AC specification. 
Note that at the moment there is almost no load on AC
Equipment on on DESPEC rack: AIDA NIM, AIDA Raspberry Pis, bPlas PC (+ WR) + 2x DESPEC NIM crates
 No autofill, VME crate or detectors
All big machines at GSI (SIS, FRS) off (suspect pumps are on)

Ion catcher not on (I think under repair)

-

FEE PSU studies
Probe connected to 5V exposed power pin on FEE64 (+v) and to grounding crimp on FEE64 (-v)
No adapter board connected
Attentuation = 1:10


Fig 4: FFT when FEEs are *off* - essentially probe+scope noise [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 5: FFT when FEEs are *on* - notice 1.4 MHz peak in FFT, also seen on ADC waveform readout before (fig 6) [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]

Fig 7: 500 ns/div 5V output on FEE, single FEE on the PSU [X: 500 ns/div, Y: 100 mV/div]

Note average max voltage is 5.31 V (power on) and ~ 70 mV "peak to peak" -might be from probe/scope?

Also see voltage changes with FEE power draw:

Power on  : 5.45 V (different scale to above)
SETUP ran : 5.51 V 
FADCs off : 5.86 V
ACQ Go:   : Unchanged; ASIC threshold 0xa: Unchanged

Also check situation on a fully loaded PSU (8 fees connected and powered on)

Power on:   5.29 V (fig 8)
SETUP ran:  5.36 V (fig 9)
FADCs off:  5.64 V (fig 10)

All X: 500 ns/div, Y: 100 mV/div

Both cases observe voltage rises as current draw drops (as expected for voltage drop along a cable)

Noise on 'scope seems to get slightly worse with reduced current (and higher voltage)

No sign of strong 100 kHz noise as seen in ADC traces beforehand

Todo:

- Check -6V and 7V rails
- Check 5V and noise when front-end card is added and pulser/HV connected
- Check between two FEE64 grounds 
- Check direct out of PSU vs ground to see if 1.4 MHz appears on PSU side or FEE64 side

-
11.10.22 Updates

Attachement 11 - 5V PSU on upper PSU with no FEEs attached whatsoever. No 1.4 MHz (on FFT) but clear low frequency beats from switching - presumably low/no load behaviour

Attachement 12 - 5V PSU on aida12 with 8 FEEs on PSU. Longer time base to allow lower frequencies in FFT. 1.4 MHz switching spikes visible but nothing around 100 kHz region

Attachments 13-16: 5V PSU on aida12 at 20 mV/div vertical and 1, 0.5, 2, 5 us/div horizontal respectively

12.10.22 Updates

Attachment 17: -6V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: -6.21 V

Attachment 19: 7V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: 7.46V

Measurement between AIDA12 ground and Reference ground/copper bar
+ve (red probe) attached to copper bar at ground point (not strong connection at present)
-ve (black probe) attached to ground crimp on aida12 (connected to cooling plate)
aida12 no adapter board connected: connections are PSU, Ethernet, HDMI and TTY only

Attachment 21: 5 us/div 100 mv/div waveform, big oscillations present. Not seen before FEEs turned on (8 FEES, 1-7+12)
Attachment 22: 10 ms/div for FFT, sharp peak at exactly 100 kHz observed...


Attachment 23: Between 5V PSU (+ve) and 19" rack (-ve) with  no FEEs connected to PSU
 See strong 100 kHz oscillations too, note that voltage isn't 5V as PSU is floating w.r.t. ground
 Looks to be common mode noise (on both 5V and Return of PSU)

Attachment 24: Same as 21 but using thick crocodile clips on probe to ground and aida12. Noise is attenuated but still present

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use

Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 

The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system

In no particular order

1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)

   observe +/- 6V, 12V, 24V lines with/without load 

   try new CAEN NIM bin and/or NIM bin of different type

2) Measure actual voltages at FEE64 power connector input

   OH suggests fab of power adaptor for safe observation - contact EW

3) rev B adaptor PCB

   invert 125 way ERNI - check for mech conflicts
   paired HV input (avoid Lemo-00 T pieces)
   consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
   straight jumpers
   shrouded Samtec headers - consider mech issues/consequences of using eject clips too
   re-visit HV filtering & separate trace ground

4) isolation transformer

   as practical matter may be necessary to operate all platform from isolation transformer
   consider hire of appropriate unit

   need method to measure isolation - will require permit to work or equiv

5) investigate S4 area ac mains

   NH discussing with GSI electricians

6) Systematic measurement of AIDA PSU noise

   Spec linear AIDA FEE64 PSU

7) Redesign of snout
   Return to 1mm welded box Al for lower stage of snout  for added rigidity

8) Revisit calculation of cable lengths. Particularly for the triple

9) 



   

All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 

11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?

Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after

00:07 Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_896 - attachment 1
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6 & 7

00:15 Check ASIC control all FEE64s, all ASICs


02:03

analysis of file S505/R5_914 - attachment 8
 zero timewarps
 deadtime all FEE64s << 1% 

per FEE64 1.8.H spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 94 ch FWHM

per FEE64 stat & rate spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today


03:41

analysis of file S505/R5_926 - attachment 19
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24


07:00

analysis of file S505/R5_954 - attachment 25
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

07:45 Spoke to David and the beam has been gone since about 05:30
      Reason for the loss of beam is a vacuum issue before the FRS
      They are waiting for the experts

08:31 Statistics ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage currrents ok - attachment 3
      ASIC clock check ok
	
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2
	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  

      Currently on file R5_771
      Analysis of file R5_770 (No beam) - attachment 4
      Around 0.25% deadtime on AIDA02 rest even less

09:02 Current free HDD space 965 GB
      Current tape server rate 4979 kB/s
      Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours

11:41 Statistics ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7
      ASIC clock check ok
	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
      Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently?

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change

09:51 Taken over from Tom following the night shift
      Experiment is still running smoothly
      Compression of the files is complete up to the start of R5.
      Have started compression of files in R5 which should run up to R5_499
      Currently on R5_528

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      System wide checks:
          		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc594 : 	 87  
      aida08 fault 	 0xf1be : 	 0xf271 : 	 179  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 	Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Current merger rate is 2E6-4E6 events per second
      Current tapeserver rate is 5800 kB/s
      Free HDD space is 1.1 TB
      At current rates will last 54 hours which should see out the experiment
      Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
      
      Analysis of R5_528 - attachment 4
      Deadtime of AIDA02 currently around 15%

11:56 Statistics ok - attachment 5
      Temps ok - attachment 6
      Bias and leakage currents ok - attachment 7
      System wide checks:
      Clocks all ok 
      
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59c : 	 95  
      aida08 fault 	 0xf1be : 	 0xf27a : 	 188  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of file R5_458 - attachment 8
      Deadtime in AIDA02 only 11.5% in this file
      TapeServer rate still 5.5 MB/s

14:36 Has been no beam for the last while or so.
      With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
      Stats- attachment 9
      Temp - attachment 10
      Bias and leakage currents ok - attachment 11
      System wide checks:
      Clock ok
      	
        		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59d : 	 96  
      aida08 fault 	 0xf1be : 	 0xf27c : 	 190  
      White Rabbit error counter test result: Passed 6, Failed 2
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3c : 	 18  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

13:30 Beam taken to change an ion source

14:57 Beam back

16:21 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14

      System wide checks - ASIC clocks ok 
	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5a3 : 	 102  
      aida08 fault 	 0xf1be : 	 0xf285 : 	 199  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3e : 	 20  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Currently on file R5_592
      Analysis of R5_591 - Attachment 15
      Deadtime of AIDA02 sitting at 17%

[NH taking over for OH so he can get home]

18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
        Statitics ok - attachement 16
        Temps ok - attachement 17
        Bias & leakage ok - attachement 18

        System wide checks - 
        Clocks OK
        ADC Calibration N/A 
	WR Decoder -
        		 Base 		Current 	Difference
        aida07 fault 	 0xc53d : 	 0xc5ac : 	 111  
        aida08 fault 	 0xf1be : 	 0xf28c : 	 206  
        White Rabbit error counter test result: Passed 6, Failed 2
        FPGA -
         Base 		Current 		Difference
        aida07 fault 	 0x2a : 	 0x3f : 	 21  
        PLL OK

        Currently on file R5_621
        Analysis of R5_620 - Attachement 19
        aida02 deadtime only 5%, all others negligible 

20:20 Stats ok - attachment 20
      Temps ok - attachment 21
      Bias and leakage currents ok - attachment 22
      System wide checks:
      Clock ok
    	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ad : 	 112  
      aida08 fault 	 0xf1be : 	 0xf293 : 	 213  
      White Rabbit error counter test result: Passed 6, Failed 2

     	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3f : 	 21  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R6_637 - attachment 23

22:13 Statistics ok - attachment 24
      Temperatures ok - attachment 25
      Bias and leakage currents ok - attachment 26
      ASIC clock check ok

	
		         Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ba : 	 125  
      aida08 fault 	 0xf1be : 	 0xf2a2 : 	 228  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x40 : 	 22  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_658 - attachment 27

23.43 Check ASIC control
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

All system wide checks OK *except* WR & FPGA errors - attachments 1 & 2

DSSSD bias & leakage currents OK - attachment 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6

analysis of file S505/R5_415 - attachment 7
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 1%



02:36

analysis of file S505/R5_449 - attachment 8
 zero timewarps
 max deadtime aida02 c. 15%, all other FEE64s < 1%

per FEE64 rate & stat spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 99 ch FWHM

per FEE64 1.8.H spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


05:38

analysis of file S505/R5_483 - attachment 19
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24



07:27

analysis of file S505/R5_502 - attachment 25
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30

Took over the night shift from Magda

18:00 : attachments 1-3

20:00 : attachments 4-6

22:00 : attachments 7-9

00:00 : attachments 10-12

Took over the night shift from Tom.

7:00 The beam is back.

8:00 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

14:00 : attachments 10-12

16:00 : attachments 13-15

23:36 Check ASIC contorol
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_154 - attachment 1
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

stats - attachments 2-6

per FEE64 1.8.H spectra - attachments 7 & 8

per p+n FEE64 1.8.L spectra - attachment 9
 aida01 pulser peak width 101 ch FWHM

per FEE64 stat & rate spectra - attachments 10 & 11

adc data item stats - attachment 12

FEE64 temps OK - attachment 13

DSSSD bias & leakage currents OK - attachments 14 & 15

00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17


03:03

analysis of file S505/R5_189 - attachment 18
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20

adc data item stats - attachment 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage currents OK - attachment 23

03:30

per FEE64 1.8.H spectra - attcahments 24 & 25


06:09 no beam 

analysis of file S505/R5_221 - attachment 26
 zero timewarps
 all dead times << 1%

all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28

adc data item stats - attachment 29

FEE64 temps OK - attachment 30

DSSSD bias & leakage currents OK - attachment 31

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good

16:20
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc525 : 	 39  
aida08 fault 	 0xf0e9 : 	 0xf166 : 	 125  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
                 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x23 : 	 9  

Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)


18:05
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc527 : 	 41  
aida08 fault 	 0xf0e9 : 	 0xf177 : 	 142  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x24 : 	 10  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)


20:22
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf17f : 	 150  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x20 : 	 15  
aida08 fault 	 0x1a : 	 0x25 : 	 11  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)

21:56
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf186 : 	 157  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x26 : 	 12  

Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)

23:50
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc52e : 	 48  
aida08 fault 	 0xf0e9 : 	 0xf18c : 	 163  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
	
			 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x27 : 	 13  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15)

07:56 Taken over from AM
      Current White Rabbit Error status:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51a : 	 28  
      aida08 fault 	 0xf0e9 : 	 0xf132 : 	 73  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA error status:
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link

      Statistics ok - attachment 1
      FEE temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      Analysis of file R3_93 - attachment 4
      Still deadtime in aida02 of ~15%
      Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
      
      Currently 2.2 TB of space on /media/SecondDrive
      Current data rate of 6.23MB/s
      At current data rate this should last 4 days

08:27 Currently no beam
      Problem with Alvarez 3 now
      Tomorrow will have a 1 hour break due to changing of the source

09:36 Beam back

09:59 System wide checks
		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51b : 	 29  
      aida08 fault 	 0xf0e9 : 	 0xf138 : 	 79  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  

      Stats ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7

11:14 FRS group is changing from Hg setting back to Pt

11:36 Back on Pt R3_129

14:14 Stats ok - attachment 8
      Temperature ok - attachment 9
      Bias and leakage currents ok - attachment 10
      System wide checks:
      WR:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc523 : 	 37  
      aida08 fault 	 0xf0e9 : 	 0xf156 : 	 109  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x1b : 	 10  
      aida08 fault 	 0x1a : 	 0x20 : 	 6

      Analysis of R3_156 - attachment 11  

00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
      Attempted to recover restarting merger but this caused 01 to drop out.

00:10 DAQ recovered and AIDA01 recovered with a reboot


01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good

03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good

05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good

07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good

16:00 Take over from Magda mrning shift

Beam was off of sometime for tunning to Mercurey setting.

17:00 attached 1-3

19:00 attached 4-6

21:00 attached 7-9

23:30 10-12

Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

Took over the night shift from Tom.

8:10 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.

12:19 OH Disabled all discriminators to reduce the data rate.

13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10

14:00 : attachments 11-13

16:00 : attachments 14-16

00.28 all histograms & stats zero'd
      baseline counters

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

BNC PB-5 pulser
 to p+n FEE64s only
  amplitude 1.0V
  attenuator x1
  decay time 1ms
  polarity +
  frequency 2Hz

all AD9252 waveform ADCs disabled

Tape Server -> no storage mode

00.36 check ASIC control all FEE64s all ASICs

      all system wide checks OK

00.41 BNC PB-5 pulser rate increased from 2Hz to 22Hz per request by HA/NH - to ensure DTAS correlation checker has sufficient rate of identified pulser events 

per p+n FEE64 1.8.L spectra - attachment 1
 aida01 pulser peak width 88 ch FWHM

per FEE64 rate spectra - attachment 2

adc, pause and correlation scaler data items - push, aida01 and aida02 stats - attachments 3-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage current OK - attachments 10 & 11

02.28 aida05 zero adc data items, unable to halt aida05 - all other FEE64s OK

      restart NewMerger -> now able to halt aida05

      DAQ go OK, merge rate c. 1.4M data items/s

per FEE64 1.8.H spectra - attachments 12 & 13
 approx flat distribution to c. 10GeV


adc data items stats OK - attachments 14

FEE64 temps OK - attachment 15

DSSSD bias & leakage current OK - attachments 16 & 17

02.55 Some problems with GO4 data analysis for PID (FRS PID looks OK). 204Pt setting. Will commence writing data to disk file S505/R2_0

04.12 per FEE64 1.8.H spectra - attachments 18 & 19
      c. 9GeV peak in upstream DSSSDs (aida01, 2, 3 & 4) indicating (high energy) heavy fragments are stopping
      Low energy events near c. 200MeV lower mass, high energy, low energy loss - fission fragments?

04.20 per FEE64 stats spectra - attachment 20
      indicates implant x-y distribution - off centre (more significantly in x than in y) but DSSSDs appear to see most of distribution
      remember raw distro will have significant fraction of fission fragments - may be misleading - need to gate on high energy events to see x-y distro of ions of 
interest

adc data items stats OK - attachments 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage current OK - attachments 23



06.10 per FEE64 rate spectra - attachment 24

per FEE64 1.8.H spectra - attachments 25 & 26

per FEE64 stats spectra - attachment 27
     
adc data items stats OK - attachments 28

FEE64 temps OK - attachment 29

system wide checks OK *except* WR & FPGA errors - see attachments 30 & 31

DSSSD bias & leakage current OK - attachments 32

07.52 analysis of file S505/R2_47 - attachment 33
      all FEE64 dead times < 1% *except* aida02 c. 7%

11:19 Stats ok 0xa on all FEE - attachment 1
      Current pulser settings - attachment 2
      FEE Temperatures ok - attachment 3
      Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
      Layout 1 after recovering channels - attachment 4
      New stats (Note increased rate on 3 and 7) attachment 5
      Bias and leakage currents ok - attachment 6

      S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.

14:05 Have had it confirmed that the flange beam dump is in position in S4

15:43 Stats ok -attachment 7
      Temp ok - attachment 8
      Bias and Leakage current ok - attachment 9


17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
      As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
      During the experiment this will be changed to SC1 trigger to solve this.

19:49: Stats ok - attachment 10
       temperatures ok - attachment 11
       Bias - Leakage current ok - attachment 12

10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11

aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour

Yesterday the sum/inverter was found to be a considerable source of noise in the system.

Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace

Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow

Put inverter back in CAEN crate in S4 and confirm similar behavior.

Move CAEN crate to new mains port, same behaviour

Move CAEN crate to AC voltage stabiliser, same behaviour

Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears  - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears

Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.

MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd

CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?

MACBs and inverter checked on another crate - no signal

Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected

Tested adding bias filters - No change

15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
      We get ok stats even with waveforms on at 0xa- attachment 5

      Waveforms still have the 100kHz oscillation in them though attachments 6 and 7

      Without waveforms we get similar performance to before the switch to the new mains - attachment 8

15:56 Move CAEN NIM crate across to this mains network
      300+kHz on all FEEs with waveforms on. Can recover waveform off performance though

16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.

17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9

01.20 

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

All waveform ADCs disabled

BNC PB-5 pulser
 to p+n FEE64s only

all system wide checks OK *except* ADC calibration, WR decoder & FPGA timestamp errors - see attachments 1 & 2

DSSSD bias & leakage current OK - see attachments 3 & 4

FEE64 temps OK - see attachment 5

statistics OK - ADC data, pause, resume & correlation scaler data items
                data push, data flush, aida01, aida02 - see attachments 6-13

per FEE64 1.8.L spectra - see attachment 14
 aida01 pulser peak width 73 ch FWHM

Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23

All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 

Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change

Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 

10.30 : Turn off AIDA FEE64s 

        Prepare a ground cable for snout - M10 terminal on one end for copper bar. Will affix with Cu tape for now (no free screws)

Plan for today:

Remove extraneous ground cables
Check ground of all FEE64s esp. 05 and 07 - include on FEE64 side.
 14:42 - Done all grounds seem snug, extra ground cables removed. Try power on and check

Try powering just DSSD1 FEE64s
 - No change (aida01 125k, aida02,03,04 - 400k)

Try grounding snout
 14:59 Connection to snout very bad as tape does not fix well to thick copper in braid. nO difference either way.

Probe PSU output with Battery DSO if available
 - See figure 2

Try shuffling FEE64 power allocation
 - 15:32 - Now as 1 X 3 X 2 X 4 X
                  5 X 7 X 6 X 8 X
(Also FEEs on own pair)
   No difference to rates

Multimeter voltages for last socket of PSU1 (paired with aida04):
  1 : 6.02 V
  2:  6.02 V
  3:  6.02 V
  4: -6.44 V
  5:  7.74 V

Similar on PSU2

Voltage between right (GND) pin 5 and 1 (pin 5 is COM) is -0.4 V, similar if pin 4 is COM. Pin1-3 are 0 V when paired
  

Spectra layouts for 2x DSSSDs can be found in directory ~/LayOut/2xDSSSDs

Attachment 1 - DSSSD bias & leakage current - OK

Attachment 2 - ADC data item stats

Attachment 3 - FEE64 temps - OK

Attachments 4-7 - aida01, 02, 03 & 04 2.*.W spectra

Noise observed for asics #1 & #2 of aida02 consistent with no cable attached - confirmed by visual inspection - ribbon cable disconnected from Samtec header of adaptor PCB

To Do list

1) re-connect ribbon cable to aida02 adaptor PCB
2) good quality ground to snout - use screws at inter-stage
3) install heavy duty grounds
4) ... 


Attachment 8 - adc data item stats - slow comparator 0x64

Attachment 9 - per FEE64 rate spectra - slow comparator 0x64

Attachment 10 - per FEE64 rate spectra - slow comparator 0xa

-

13:30 - Attach heavy duty ground to AIDA 

Attachment 11 - per FEE64 rate spectra - slow comparator 0xa

Try power cycle as system ground changed significantly now

Try to reattach aida04 TTY connector - still wrong

Archive old tty logs (ttyUSB*) to zip and delete them, makes easier to track startup 

Attachment 12 - per FEE64 rate spectra - slow comparator 0xa

No major difference, aida01 quieter than before

aida05 has some missing channels - loose ERNI?

Attachements 13 & 14 - Waveforms

Temporarily raise to 0x64 and test merger for DTAS synchro test later... all works good to MBS 
Change NETVAR MERGE.LinksAvailable to 8

Notice one copper screw touching aluminium, add some kapton tape behind it to isolate it. No difference

Voltage diff between aida02,aida06 < 1 mV
Voltage diff between copper bars < 0.5 mV (but not 0)

A comment from DTAS people that they observed a large increase in noise recently - perhaps from accelerator
May indicate current power/gronud oddities?

Attachment 15 - Rates at 0x1f (= 310 keV)

DSSSD#1 reasonable (aida04 a bit bad), but mostly just pulser (25 Hz)

DSSSD#2 much worse esp. p+n sides (aida05, aida07)

ERNI connectors pushed in and feel fully in

2x MSL type BB18(DS)-1000 installed


Adaptor PCB LK config

all FEE64s LK2, LK4 fitted
n+n FEE64s LK1 fitted
bottom p+n FEE64s LK3 fitted


AIDA PSU config

top 1-3 2-4
bottom 5-7 6-8


DSSSD bias -> p+n FEE64 core, n+n FEE64 braid


Heavy duty grounds not connected to FEE64 adaptor PCBs


ASIC settings 2021Apr29-13-16-00
 slow comparator 0xa

BNC PB-5 settings
 amplitude 1.000V
 attenuation x1
 tau_d 1ms
 frequency 25Hz
 polarity +


Attachment 1 - DSSSD bias & Leakage current - OK

Attachment 2 - FEE64 temps OK

Attachment 3 - adc data item stats

Attachments 4-9 - system wide checks OK *except* aida07 fails ADC calibration

Attachments 10 & 11 - 1.8.W spectra 20us FSR

Attachments 12 & 13 - 1.8.L spectra - aida03 pulser peak width 155 ch FWHM

Attachment 14 - per FEE64 rate spectra



19.15 Attachments 15-18 - 2*W spectra for aida01, 02, 03 &04

Attachment 19 - per FEE64 rate spectra

Attachment 20 - adc data item stats

DSSD stack will be as follows

1      3208-7  1019       125              upstream
2      3208-13 1020       120              downstream

First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance

3208-7: 
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

3208-13:
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

Conclusion: All bias bond wires are intact on both DSSDs

Visual inspection is both DSSDs look good with no damage or bond wire issues

Installed in snout: full stack

3 mm bPlas 1
17 mm Air
1 mm AIDA 1  (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2

The n+n FEEs were all removed to adjust the platform back to the single configuration

After some confusion we found the orientation of the frame to position the single detectors and moved back

Faulty AIDA cards aida05 and aida14 were removed to be returned backi (ASIC damage)

The cables for aida10 and aida12 were moved back to the left-right side and the n+n cards were installed and reocnnected

p+n cards moved to bring aida11 and aida09 above aida03/aida07 and aida01/aida05. Also move a different card to replace old aida05

While doing this it was noticed the HDMI for aida13 was very loose and actually fell off (I guess the cable was pulled incorrectly)
It has been removed and will eb sent back to be resoldered 

The copper grounding bars and ITEM frame were removed due to conflicting with the single snout layout
The bolts are slightly too short to hold the frame where it won't conflict so we loko for longer bolts to reattach the system
Some new ground cables will be made to reach the extra n+n FEEs

Aida13-Aida16 removed from the PSU 

fig1: layout of snout frame in "single" mode (alu block configuration)


MAC Addresses

aida01 - ba-8a
aida05 - 0d:15
aida09 - ee-01

aida04 - f6-b7
aida08 - ba-2b
aida12 - ba:89 

aida03 - d8-21
aida07 - 41:b4:0c
aida11 - f6-5a

aida02 - ba:22
aida06 - ee:72
aida10 - cf-ac [mistakenly swapped with aida12]

Second switch will stay in S4 and has now been screwed in properly below the switch 1 and the hdmi cables entrance

DHCP *not* updated

Relay is powered *off* so FEEs cannot be turned on, alL HV channels disabled

Tomorrow is a holiday so no work will be happening @ GSI

09:00 - Remove bias of DSSDs  and pwoer of FEEs

Remove snout from S4 so DEGAS frame can be mechnically removed

Snout now in FRS Messhuette 

17.40 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK

15.11 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK

16.31 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK

17.20 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps - OK

18.01 DAQ stopped

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachment 2 - DSSSD bias & leakage current

Attachment 3 - FEE64 temps - OK

Attachment 4 - ADC data item stats

      file no storage mode, no MBS data transfer

      ADC control register 0x0 

      std disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xa

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - ADC data item stats

Attachment 4 - FEE64 temps - OK

16.35 zero all histograms
      zero statistics
      zero NewMerger statistics

      ASIC check control - all FEE64s, all ASICs

      baseline counters
      system wide checks - all OK *except* aida07 WR error 0x10

Attachment 5 - per FEE64 rate spectra

Attachmnents 6 & 7 - per FEE64 1.8.W spectra 20us FSR

17.35 zero all histograms
      zero statistics
      zero NewMerger statistics

      ASIC check control - all FEE64s, all ASICs

      file no storage mode, no MBS data transfer

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0x80

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2-10 - NewMerger stats

Attachments 11-13 - iptraf, TapeServer, NewMerger

Attachment 14 - per Fee64 rate spectra

Attachment 15 - lost activity  monitor

Attachment 16 - ASIC settings aida01

Attachments 17-22 - adc, pause, resume & correlation scaler data items, push, flush, aida01 & aida06 stats

Attachment 23 - FEE64 temperatures OK

Attachment 24 - DSSSD bias & leakage current

08:00 Still have beam for the time being.
      Currently on file R4_2049

      FEE Stats - attachment 1
      FEE Temps ok - attachment 2
      Bias and leakage currents - attachment 3
      System wide checks all ok *except aida09 fails clock check with value 6 which is ok*

      Plan for the morning is once beam is lost is Ge group will enter and place source on the snout to perform efficiency measurement.
      Once they have left the area and are taking data a pulser walkthrough will be performed.
      Patrick can then have access to the system

08:16 Beam taken away
      Data taken stopped on file R4_2055
      Will perform a pulser walkthrough now

      Pulser walkthrough started on R5
      1V->0.1V in 0.1V steps
      Pulser setting - attachment 4
      p+n pulser peaks - attachment 5
      n+n pulser "peaks" - attachment 6

01:00:     Attachments 1-3, System checks good

03:00:     Attachments 4-6, System checks good

05:00:     Attachments 7-9, System checks good

07:00:     Attachments 10-12, System checks good

16:00 Took over the shift from Magda.

17:30 Updates: Attached (1-3)

19:30 Updates: Attached (4-6)

21:30 Updates: Attached (7-9)

23:00 Updates: Attached (10-12)

08:00 Took over the night shift from Tom (Attachments 1-3)

10:00: Attachement 4-6

around 10:30: Beam went off (Attachment 7)

11:00: Beam is back

12:00 Attachments 8-10

14:00 Attachements 11-13

15:50 Attachments 14-17

00.01 DAQ continues
      file S450/R4_1319

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Disk space OK - /media/SecondDrive

[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  389M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  4.1T  2.8T  61% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

00.03 all histograms zero'd
      system wide checks counter baseline


00.08 check ASIC control - all FEE64s all ASICs

Attachments 1 & 2 - DSSSD bias & Leakage current - OK
                    grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps OK

Attachments 4-9 - adc, pause, resume & correlation scaler data items, push, flush stats

Attachments 10-16 - TapeServer, NewMerger, NewMerger stats

Attachment 17 - ucesb

04.08

Attachment 18 - DSSSD bias & Leakage current - OK

Attachment 19 - FEE64 temps OK

Attachment 20 - adc data item stats

Attachments 21 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

06.36

Attachment 22 - DSSSD bias & Leakage current - OK

Attachment 23 - FEE64 temps OK

Attachment 24 - adc data item stats

Attachments 25 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

Status at 16:45 (CET)

The expreiment continues to run smoothly.

AIDA Stats look ok - Attachment 1

AIDA Temperature ok - Attachment 2

AIDA Leakage current: ok - Attachment 3

System wide check:

Clock: 13 passed, 1 failed (aida09)

ADC Calibration: 9 passed, 5 fialed (aida2,6,9,10,13)

White rabbit decoder: 14 passed, 0 failed.

FPGA timestamp: 14 passed, 0 failed.

Status at 18:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 4

AIDA Temperature ok - Attachment 5

AIDA Leakage current: increase slightly - Attachment 6

System wide check: No changes, same as above

Status at 20:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 7

AIDA Temperature ok - Attachment 8

AIDA Leakage current: increase slightly - Attachment 9

System wide check: No changes, same as above

Status at 22:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 10

AIDA Temperature ok - Attachment 11

AIDA Leakage current: increase slightly - Attachment 12

System wide check: No changes, same as above

08:40 FEE Temperatures ok - attachment 1
      FEE statistics - Attachment 2
      Bias and leakage currents ok - attachment 3
      ASIC check ok
      All system wide checks ok - Except aida09 fails clock check with bit 6

11:00 FEE statistics - attachment 4
      FEE temperatures all ok - attachment 5
      BIAS and leakage currents - attachment 6
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1030
      13.1 MB/s to disk
      4.8 million items per second
      
13:13 FEE statistics - attachment 7
      FEE Temperatures all ok - attahcment 8
      Bias and leakage currents ok - attachment 9
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1078
      13.1 MB/s to disk
      4.8 million items per second

15:35 Analysis of R4_1128 - attachment 10

Pb beam c. 1e+9/spill

00.02 zero all histograms
      system wide checks - baseline counters

00.08 ASIC check control - all FEE64s, all ASICs

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2 & 3 - ucesb - AIDA DSSSD #1 c. 10-20Hz peak rate on spill, c. 0Hz off spill

Attachment 4 - DSSSD bias & Leakage current - OK

Attachment 5 - FEE64 temps OK

Attachment 6-13 - adc, pause, resume & correlation scaler data items, push, flush, aida01, aida06

Attachments 14-18 - system wide checks - all OK *except* aida09 clock fail status 6

Attachments 19-25 - iptraf, TapeServer, NewMerger, NewMerger stats



00.44 file S450/R4_792

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

00.51 disk space OK - c. 3' per data file

[npg@aidas-gsi S450]$ ls -l /
total 40
lrwxrwxrwx.   1 root root     7 Oct 18  2021 bin -> usr/bin
dr-xr-xr-x.   5 root root  4096 May 12 18:31 boot
drwxr-xr-x.  23 root root  4100 May 12 18:31 dev
drwxr-xr-x. 145 root root  8192 May 12 18:31 etc
drwxr-xr-x.   4 root root    56 Jan 28 14:56 home
lrwxrwxrwx.   1 root root     7 Oct 18  2021 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 Oct 18  2021 lib64 -> usr/lib64
drwxr-xr-x.   5 root root   113 Apr 28 11:58 media
lrwxrwxrwx.   1 root root    45 Oct 18  2021 MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119
drwxr-xr-x.   2 root root     6 Apr 11  2018 mnt
drwxr-xr-x.   3 root root    16 Oct 18  2021 opt
dr-xr-xr-x. 464 root root     0 May 12 18:30 proc
dr-xr-x---.  10 root root  4096 May 13 11:47 root
drwxr-xr-x.  44 root root  1360 May 14 06:34 run
lrwxrwxrwx.   1 root root     8 Oct 18  2021 sbin -> usr/sbin
drwxr-xr-x.   2 root root     6 Apr 11  2018 srv
dr-xr-xr-x.  13 root root     0 May 12 18:31 sys
lrwxrwxrwx.   1 root root    28 Apr 28 14:19 TapeData -> /media/SecondDrive/TapeData/
drwxrwxrwt.  40 root root 12288 May 15 00:49 tmp
drwxr-xr-x.  13 root root   155 Oct 18  2021 usr
drwxr-xr-x.  21 root root  4096 Oct 18  2021 var
[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  373M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  3.1T  3.8T  46% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

Attachment 26 - analysis data file R4_792

Attachments 27-28 - per FEE64 rate & stat spectra - shows distro HEC events

Attachments 29-30 - per FEE64 1.8.L spectra 
 pulser peak widths aida01 134 ch FWHM, aida04 393 ch FWHM

Attachments 31-32 - per FEE64 1.8.H spectra

Attachments 33-34 - aida02 & aida04 1.*.H spectra

04.13

Attachment 35 - DSSSD bias & Leakage current - OK

Attachment 36 - FEE64 temps OK

Attachment 37 - adc data item stats

Attachments 38 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6


07.05

Attachment 39 - DSSSD bias & Leakage current - OK

Attachment 40 - FEE64 temps OK

Attachment 41 - adc data item stats

Attachments 42 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

Attachments 43-48 - aida02 & aida04 & aida06 & aida08 & 1.*.H spectra

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

14:00

12:00

10:00

08:00 Took over the shift from Philippos
      In system wide checks aida09 fails ASIC clock check but it is bit 6 which is ok

4:13

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Screenshot 1: statistics

Screenshot 2: temperatures

Screenshot 3: scalers

Screenshot 4: Bias and leakage current

6:36

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Screenshot 5: statistics

Screenshot 6: temperatures

Screenshot 7: scalers

Screenshot 8: Bias and leakage current

AIDA stats ok

Leakage current ok

Temperatires ok

grafana ok

ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.

System wide check done and same results as earlier:

aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Took over remotely from CB  about an hour ago

At 0h40 (CET):
      Stats still ok - attach 1
      Leakage still OK - Attach 2
      Temps - attach 3
      Grafana OK
      UCESB rates ok - attach 5
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before

08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)

The rate is very low in all aida, not sure .

This was because the statistics page had been changed to transfer buffers. The rates were ok

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).

This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

All seems ok  (apart for my virgin network connection usually quite reliable but not so tonight !?!).

Statistics seems ok.

Temperatures appears stable

Leakage current ok.

System Wide Checks:

Clock check: Passed= 14: - Failed=: 0

ADC calibration check:

FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES

White rabbit decode status: Passed=14 ; Failed=0

FPGA timestamp check: Passed=14 ; Failed=0

Memory information: Page loading failed with the error message below:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida*

Attachment 1 - top aida06

Attachment 2 - top aida01

Attachment 3 - top aida-gsi (AIDA server)

Attachment 4 - TapeServer

Attachment 5 - NewMerger stats block 6 (aida06)

Attachment 6 - NewMerger stats block 1 (aida01)

Attachment 7 - NewMerger stats block 0 (all FEE64s)

Attachment 8 - NewMerger options

Attachment 9 - NewMerger link stats

Attachment 10 - NewMerger

Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats

Attacgment 15 - Network traffic

08.10 AIDA DAQ status
    
      14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
      AD9222 OFF - waveforms disabed
      disc - all disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23

      BNC PB-5
      amplitude 1.000V
      attenuation x1
      tau_d 1ms (tail pulse)
      polarity +
      frequency 22Hz

08.14 all histograms zero'd
2
08.15 system wide check counter baseline

08.17 check ASIC control - all FEE64s all ASICs

Attachment 1 - grafana DSSSD bias & leakage current - OK

Attachment 2 - per FEE64 stat spectra

Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips

Attachments 5 & 6 - per FEE64 1.8.L spectra
 pulser peak width aida01 124ch FWHM, aida04 293ch FWHM

Attachment 7 - per FEE64 rate spectra

Attachments 8-12 - system wide checks - all OK

Attachments 13-16 - statistics - adc, pause, resume & correlation data

Attachment 17 - FEE64 temps OK

Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends

Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages

08.44 DAQ continues - file NOTAPE/R18 - no storage mode

08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc

08.53 no storage mode enabled

Attachment 21 - analysis of file NOTAPE/R18_20

Attachment 22- analysis of file NOTAPE/R19_19
               AIDA12 slow comp up to 0xd improved deadtime drastically for FEE

10:54 Beam on detector. Take another run to test deadtime - attachment 23
      Deadtimes ranging from 10-40% in n+n FEEs

Attachment 23 & 24 aida06 aida02 1.*.H spectra


14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
        Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime)

01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
      That is now complete.
      Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc

      R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
      Will lower 2 and 4 to 200 and raise 6 and 8 to 300

      R17 2 good, 4 shakey 6 and 8 poor - attachment 2

      R18 2 and 4 at 220 and 6 and 7 at 350
      2 and 8 low deadtime, 4 and 6 could be better. - attachment 3


      Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
      Statistics at this rate - attachment 4




02:18 Statistics - attachment 7
      Temperatures - attachment 6
      Bias & leakage current - attachment 5

      Clock Status - OK
      White Rabbit - OK
      The other 3 checks currently aren't working




04:11 Statistics - attachment 10
      Temperatures - attachment 9
      Bias & leakage current - attachment 8

      Clock Status - OK
      White Rabbit - OK




06:05 Statistics - attachment 13
      Temperatures - attachment 12
      Bias & leakage current - attachment 11

      Clock Status - OK
      White Rabbit - OK

20:19 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

22:12 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
      It is currently sending data pause requests and data push requests but nothing else - attachment 1
      telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
      FEE appears to have hung. - attachment 2
      Still passes all system wide checks.
      When performing a check load can see the tclsh process take up CPU usage

09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
      AIDA14 recovered but ASIC2 still missing
    
      Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
      Given access would change the adapter board on AIDA05.
      Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
      Would retain the option of a speedybacktrack

      Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
      A threshold of 0x4 reduced the rate in the HEC channels considerably
      A threshold of 0x6 removes entirely

      Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned

09:59 Had access so replaced the adapter card on AIDA05
      Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
      This could also have been the cause


10:09 Restarted DAQ and there is no change to the behavior of AIDA05
      Still see events in HEC
      Again raising threshold to 0x5 in ASIC1 largely removes them. 

13.10 Attachments 3-5
      DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
      ASIC check load
      ASIC slow comparator all FEE64s all ASICS 0xa
      ADC data item stats

13.17 Attachments 6-9
      DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
      ADC data item stats
      NewMerger/Tape Server

18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
      It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2

      To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
      After powering on it was found there was no improvement and possible worse noise than before

      We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14

8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam

Reseat aida05 ribbon cable on adapter board while in...

No difference to rates, aida05 still shows hot asic01
aida14 seems seated OK, still asic02 basically empty

Noise level still high, threshold set to 0x20 (320 keV) which has decent rate

Start forwarding to MBS with 0x20 thresholds

10.12 slow comparator 0x20 -> 0xa

Attachments 1-5

Attachment 6 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 7 FEE64 temps OK

Attachment 8 DSSSD bias & leakage currents OK


13.30 system checks

      N.B. aida05 ASIC temp low and pause/resume data rate high

Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK

Attachments 10-15 spectra

Attacahments 16-19 system wide checks OK *except* WR error counter

Attachment 20 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 21 FEE64 temps OK

Attachment 22 DSSSD bias & leakage currents OK

Attachments 23-29 newMerger stats

20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable

22:26 - No implantation test although we had some brief beam through AIDA.
      - FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow
        Will keep DAQ on 0x64 to eliminate deadtime issues for implantation

09.32 Re-start AIDA

Attachments 1 & 2 1.8.W spectra 20us FSR

Attachments 3 & 4 1.8.L spectra
 aida10 98ch FWHM
 aida06 384 ch FWHM

Attachment 5 per FEE64 rate spectra

Attachments 6-13 system wide checks - all OK

Attachment 14 ADC data item stats
 n+n FEE64s c. 250k -> c. 370k
 aida01, 7 & 12 c. 200-400k 

Attachment 15 FEE64 temps OK

Attachment 16 DSSD bias & leakage current OK

Attachment 17 per FEE64 sys console status summary



10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
      ASIC check load

Attachments 18 ADC data item stats
 no change to n+n FEE64s, very high p+n FEE64s now at lower rates

Attachments 19 & 20 1.8.W spectra 20us FSR


18.27 all ADC data rates very high
      aida05, asic #1 is again producing HEC data
      aida14, asic #2 producing very little data

Attachments 23 & 24 1.8.W spectra 20us FSR

Attachments 25 & 26 1.8.L spectra
 aida10 165ch FWHM

Attachment 27 per FEE64 rate spectra

Attachment 28 ADC data item stats

Attachment 29 FEE64 temps OK

Attachment 30 DSSD bias & leakage current OK

Attachments 21 & 22 1.8.L spectra
 aida10 65ch FWHM
 aida06 394ch FWHM

Attachment 31 Rate with threesholds at 0x64

14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change. 


14.30 Replace FEE64 aida05
      old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
      old dhcpd.conf file backed up

AIDA FEE64 layout with respect to the beam.

DSSSD bias and FEE64 power OFF during Ge install



Photographs of AIDA (triple) + DEGAS setup as of Sunday pm.

Attachments 1-4 side/front/oblique views - note AIDA snout droops and twists

Attachments 5 & 6 - position of the AIDA stand vertical adjustment threads (brass) 

The asymmetry upstream-downstream pre-dates this setup. To lift the end of the snout clear of the of lower DEGAS detectors all 3x 1/4 turns clockwise (viewed from above) were applied to each of the four threads.

Adaptor PCB link settings

n+n FEE64s

aida02, 4, 6 & 8 - LK1 fitted, HV Lemo-00 not connected - left/right wafer n+n side bias ring -> gnd

p+n FEE64s

   top - aida09, 1, 10 & aida13, 5, 14 - LK2 & LK4 fitted - front/back FP -> gnd
bottom - aida11, 3, 12 & aida15, 7, 16 - LK2 & LK4 fitted - front/back FP -> gnd

   top - aida01, 5 - LK3 fitted - middle wafer n+n side bias ring -> gnd
bottom - aida03, 7 - LK3 fitted - DSSSD PCB gnd -> gnd

AIDA PSU

   top 1-2, 3-4, 9-10, 11-12
bottom 5-6, 7-8, 13-14, 15-16


ASIC settings 2021Apr29-13-16-00
     slow comparator 0x64 -> 0xa

BNC PB-5 settings
    amplitude 1.000V
    attenuation x1
    tail pulse
    tau_d 1ms
    polarity +
    frequency 25Hz

Attachments 1 & 2 per FEE64 1.8.L spectra 
 pulser peak widths aida01 121 ch FWHM, aida02 517 ch FWHM

Attachments 3 & 4 per FEE64 1.8.W spectra 20us FSR

Attachment 5 per FEE64 rate spectra

Attachment 6 - grafana DSSSD bias/leakage current for previous 7 days showing effect of day/night temp variations

Attachments 7 & 8 - merger & tape server c. 14Mb/s to disk - no errors reported by servers

Attachments 9-12 - system wide checks - all OK

Attachment 13 ADC data item stats

Attachment 14 FEE64 temps OK

Attachment 15 DSSSD bias & leakage currents OK

16.15 aida05 zero ADC data items stats
      merger restart, DAQ STOP/Go cycle to recover


      activity is S4 area near AIDA DSSSDs as install of DEGAS detectors commences

19.50 DEGAS install continues

      DSSSD bias and FEE64 power OFF
 

10:30 DAQ appears to have stopped

      Attachment 2 - DSSSD bias and leakage current OK

      Attachments 1, 3-5 - zero rate adc data items all FEE64s - note aida08 adc data item counter zero

      RESET/SETUP/GO aida08 *only*
      DAQ STOP *all* FEE64s


      Attachment 6 - FEE64 temperatures OK

      Attachments 7-23 - system wide checks - all OK

10:48 DAQ GO *all* FEE64s
      all spectra zero'd

      attachment 29 - ADC data items
       n+n FEE64s aida02, 4, 6 & 8 c. 250k
       p+n FEE64s aida07, 9 & 12 c. 100-150k, all other FEE64s < 100k

      attachments 26 & 27 1.8.L spectra
       aida01 peak width 112ch FWHM
       aida02 peak width 485 ch FWHM

      attachments 24 & 25 1.8.W spectra 20us FSR
      
      attachment 30 - aida05 1*H spectra - aida05 asic#1 is producing HEC data

      attachment 31 - aida05 also producing disc data

      attachments 32 & 33 merger and tape server OK - 14MB/s to disk - no data to MBS currently? 


18.36 Disconnect ribbon cabling from adaptor PCB - continue to observe HEC events from aida05 asic#1
      Disconnect test inputs from adaptor PCB - continue to observe HEC events from aida05 asic#1

      Conclude aida05 asic#1 has probably failed in operation and should be replaced. Other FEE64s are currently in CRYRING hall and measurement is in progress - ML says we should be able to access Monday.

bPlas has been mostly cabled in now

Reseat LAN cable (AIDA Switch -> AIDA-3) and reachieve 1 gbit/s speed as needed.

CHeck DAQ with MBS, working well (16 MB/s). Performance seems better than before on MBS side too

Note implant counts in aida05, observable in rate spectra (fig 1)
Also a few early channels in aida10 seem empty

ASIC check-loads didn't fix
Try powercycle... also didn't fix

Tomorrow will inspect adapter board and ribbon cable, perhaps it is loose.

Absolute rates otherwise seem OK (fig 2) aida01 and aida07 are both very hot.
aida01 is usually quieter but aida07 has always been observed running fast

The snout endcap has been sealed with new copper tape over the red tape

Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive



-

the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s

-

Preliminary alpha background analysis (over weekend, 90 hours statistics)

Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Apr 28 19:50:20 2022
Last event : Mon May  2 14:04:03 2022
Duration   : 324823 seconds
--------------------------------------
MBS Events   (M):     912899 (2 Hz)
AIDA Words   (W): 7472078760 (-3441 Hz)
ADC Words    (A): 2742871865 (-4778 Hz)
-- Decays    (D): 2728717223 (-4821 Hz)
-- Implants  (I):   14154642 (43 Hz)
Info Words   (F): 4729206895 (1 kHz)
-- SYNC48    (S):    2093328 (6 Hz)
-- SYNC63    (s):    2093328 (6 Hz)
-- PAUSE     (P):    3823161 (11 Hz)
-- RESUME    (R):    3823161 (11 Hz)
-- Discrim   (D): 1390520565 (4 kHz)
-- Scaler    (C): 3326853352 (3 kHz)
-- Unknown   (U):          0 (0 Hz)
Timewarps    (T):         20 (0 Hz)
Dead Time    (X): 5024.43 s (1.54682%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
   1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
   2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
   3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
   4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
   5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
   6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
   7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
   8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
   9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
  10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
  11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
  12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
  13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
  14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
  15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
  16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9

PAUSE/RESUME stats are dominated by 2x FEE64s - aida07 and aida16

These FEE64s also have the highest rates of discriminator data?

> The snout endcap has been sealed with new copper tape over the red tape
> 
> Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
> It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive
> 
> 
> 
> -
> 
> the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
> When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s
> 
> -
> 
> Preliminary alpha background analysis (over weekend, 90 hours statistics)
> 
> Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04
> 
> ======================================
> AIDA Unpacker Analysis
> --------------------------------------
> First event: Thu Apr 28 19:50:20 2022
> Last event : Mon May  2 14:04:03 2022
> Duration   : 324823 seconds
> --------------------------------------
> MBS Events   (M):     912899 (2 Hz)
> AIDA Words   (W): 7472078760 (-3441 Hz)
> ADC Words    (A): 2742871865 (-4778 Hz)
> -- Decays    (D): 2728717223 (-4821 Hz)
> -- Implants  (I):   14154642 (43 Hz)
> Info Words   (F): 4729206895 (1 kHz)
> -- SYNC48    (S):    2093328 (6 Hz)
> -- SYNC63    (s):    2093328 (6 Hz)
> -- PAUSE     (P):    3823161 (11 Hz)
> -- RESUME    (R):    3823161 (11 Hz)
> -- Discrim   (D): 1390520565 (4 kHz)
> -- Scaler    (C): 3326853352 (3 kHz)
> -- Unknown   (U):          0 (0 Hz)
> Timewarps    (T):         20 (0 Hz)
> Dead Time    (X): 5024.43 s (1.54682%)
> ======================================
> ======================================
> Individual FEE Overview
> ======================================
>  FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
>    1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
>    2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
>    3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
>    4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
>    5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
>    6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
>    7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
>    8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
>    9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
>   10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
>   11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
>   12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
>   13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
>   14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
>   15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
>   16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9

Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 3)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- IGNORE FOR TIME BEING Options database monitor (Top middle terminal screen 2, workspace 3)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 5)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 3)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 3)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 3)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 3)

- Perform the five system wide checks (Web browser tab screen 1, workspace 3)
     - Ignore the system wide checks for ADC calibration (ADCs are turned off so they will all fail)
     - Check memory information also fails.
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 5)

Runs are in /TapeData/BGAPR22 located on 8TB disk "SecondDrive" (N.B. 2A experiment is using ThirdDrive)

Old runs and test files were removed from disk. Left is an unknown S481 folder (fibre implant test?) and Patrick
chown to npg:npg as uid changed from old SL6 to CentOS 7

Bias DSSDs, leakage current higher than before (maybe the sheet isn't covering the snout)
Pulser set to OFF
ASIC Thresholds 0x64, rates mostly 0 (good)
Edge strips on aida02 and aida06 show non zero, and lots in aida04 asic 1 are high, will recheck connector tomorrow in hope

Connecting to merger and rate is 60k items/sec with random jumps to 400k items/sec
Start xfer to tape server: Rate approx 100 KB/s (360 MB per hour, 2.88 GB per 8 hours)

19:51 - Will leave overnight running to collect alpha

Fri 29 Apr

14:38 - Run paused for MBS tests with FRS

16:50 - Resume alpha collection over weekend
Pulser OFF, 0x64 thresholds

Remount snout in S4 (photos to come) 
Alignment with DEGAS OK but a bit short...  DEGAS platform will be brought 31 mm closer to AIDA by rail adjustment

Can mount bPlas booster boards on old ITEM frames and also partly on AIDA snout (isolated)

-

Reconnect all adapter boards and grounds, bias AIDA...

HV OK. We still have a light leak at the snout endcap. "Fixed" with sheet for now

HV leakage currents figure 1

Start AIDA DAQ and check rates.. all seem OK. aida07 is very noisy other p+n good
Notice some issues in aida04, check and ribbon cable slightly loose on one side of adapter
No change to other FEEs.
Aida07 was inspected but nothing obviously wrong

Rates figure 2
Rate spectra figure 3
Wavesform figures 4 (p+n), 5 (n+n)
Pulser peak withs
 p+n figure 6, width ~  70 channels (50 keV)
 n+n figure 7, width ~ 570 channels (400 keV)

n.b. double peak in aida14 is an artifact from an uncleared histogram

DSSDs debiased to allow work on bPlas cabling without tripping DSSDs

09:59 Noticed that the right hand ribbon cable for aida07 was offset vertically by 1. i.e. Only 1 of the 2 rows of pins in. Unbiassed detectors and reconnected.
      Rebiassed detectors - leakage current on DSSD2 increased to 9.96uA - Attachment 1

      Power cycling FEEs

      Following restart the FEEs show some improvement in rates - attachment 2
      Rate spectra - attachment 3
      Still issues with aida11 in the rate spectra (Likely a misaligned ERNI connector)



      Pulser on rate spectra - attachment 4
      Pulser off rate spectra - attachment 5
      We can see that with the pulser on there are events in the channels in aida01 and aida11
      With the pulser off these channels disappear.
      ERNI connector is then likely aligned the issue is more likely the DSSD ribbon cable is misaligned.

      Taking bias off to investigate.


10:39 Could not observe anything out of alignement about either the connectors or the adaptor boards
      Note that after pushing on the connector AIDA01 had largely recovered

13:30 Top of snout taken off while mounted in S4. It was observed that one of the ribbon cables for aida11 was off by 1 row.
      This was corrected in situ and the snout was put back together and made light tight again.
      Oddly leakage current still the same - DSSD2 has come down though - attachment 6
      FEEs powercycled for new tests

      Half the channels in aida11 had appeared again
      Took apart once more and found the other header also off by one row which has been corrected

      FEEs powercycled for new test

      All of aida11 now showing - attachment 7
      Statistics at 0xa shown in attachment 8
      All p+n for DSSD1 are good (aida01, 03, 09, 10, 11 and 12)
      p+n for DSSD2 are so so (aida05, 07, 13, 14, 15 and 16)
      All n+n 2, 4, 6 and 8 are poor still


14:50 p+n waveforms - attachment 9
      n+n waveforms - attachment 10

      Current jumper configuration
      LK1 (Bias ground) on FEEs 6 and 8
      LK2&4 on all p+n
      LK3 on aida03 and aida07 (Bottom middle)

15:00 Investigated whether there was a light leak. Placed black material over snout.
      Leakage on DSSD2 dropped to 5.6 - attachment 11
      There was a light leak
      Stats have decreased slightly - attachment 12

16:00 Move raspberry pis atop 5mm alminimum plate to keep away from FEE64 PSUs
      No change

16:30 Remove pulser network from all FEEs 
      Rates get noticably worse in many FEEs... why?
      Stats - attachment 13

      Possible ideas include braid touching things it shouldn't (Adapter PCBs?) 

Re-assembled triple snout re-installed on the AIDA stand - DSSSD ribbon cables and grounds re-connected.
FEE64 adaptor PCB LKs per https://elog.ph.ed.ac.uk/DESPEC/409

Attachments 1 & 2 1.8.L spectra
  p+n FEE64 aida10 pulser peak width 97 ch FWHM
  n+n FEE64 aida04 pulser peak width 550 ch FWHM

Attachment 3 rate spectra
  missing channels aida01, aida07 & aida11 - misaligned ERNI and/or Samtec connectors?

Attachments 4 & 5 1.8.W spectra 

Attachment 6 ADC data item rates
 aida10 OK, all others c. 300-400k

Attachment 7 FEE64 temperatures OK

Attachment 8 DSSSD bias & leakage currents OK

DSSSD bias OFF. DAQ STOPped

Attachment 6 FEE64 temps OK

Attachment 1-5 system wide checks OK 

OH/NH disassembling current AIDA triple snout and DSSSD stack.

Length of ribbon cables (including connector) outside snout as follows

p+n junction side 9cm & 15cm
n+n ohmic side 10.5cm & 16cm 


Update
p+n junction side: 6.5cm, 10.5 cm (used  7.5, 11.5 for 1cm slack)
n+n ohmic side:  11.0 cm, 15.0 cm (used 12.0, 16.0 for 1cm slack)
Measurement from SAMTEC connector to exit from the rubber seals in AIDA snout.

DSSSD bias OFF. DAQ STOPped

Attachment 1 FEE64 temps OK

Attachment 2 system wide checks OK *except* WR decoder (attachment 2) and FPGA ts errors no longer works

Attachment 6 System wide check - 'Synchronise ASIC clocks'

Attachment 5 System wide check - check ADC calibration - all 16x FEE64s fail

Attachment 4 System wide check - WR decoder status - aida09 WR status 0x10 clears

Repeat Synchronsie ASIC clocks

Attachment 3 System wide check - WR decoder status - aida05 WR status 0x10 clears
             

AIDA adaptor PCB config - all LKs removed *except*

aida04 & aida08 LK1

aida02 & aida07 LK2, LK3 & LK4

aida09, aida01, aida10, aida11, aida15, aida12 & aida16 LK2 & LK4



10.05 check all ASICs all FEE64s
      zero histograms

Attachment 1 per FEE64 rate spectra

Attachment 2 per FEE64 p+n 1.8.L spectra - aida10 peak width 79 ch FWHM

Attachment 3 per FEE64 p+n 1.8.W spectra 20us FSR

Attachment 4 per FEE64 n+n 1.8.W spectra 20us FSR

Attachments 5-8 - system wide checks - all OK *except*

  FPGA timestamp errors no longer works
  aida05 & aida09 report WR decoder error 0x10

Attachment 9 ADC data item stats

Attachment 10 FEE64 temps OK

Attachment 11 DSSSD bias & leakage current OK

Attachment 12 per FEE64 n+n 1.8.L spectra - aida08 peak width 369 ch FWHM


14.25 DAQ STOP
      DSSSD bias OFF

      AIDA triple snout removed from AIDA stand and moved to table by aida-gsi (Messeheute)

all system wide checks OK *except* aida10 failed ADC calibration

Attachment 1 ADC data item stats
Attachment 2 FEE64 temps OK
Attachment 3 DSSSD bias & leakage currents OK





DSSSD bias to left (looking upstream) wafer *only*
      FEE64s 10 & 12, 14 & 16

Attachments 4 & 5 1.8.W spectra 20us FSR

Attachment 6 ADC data item stats

Attachment 7 DSSSD bias & leakage current




ASIC check all FEE64s

DSSSD bias to middle (looking upstream) wafer *only*
      FEE64s 1 & 3, 5 & 7

Attachments 8 & 9 1.8.W spectra 20us FSR

Attachment 10 ADC data item stats

Attachment 11 DSSSD bias & leakage current





ASIC check all FEE64s

DSSSD bias to right (looking upstream) wafer *only*
      FEE64s 9 & 11, 13 & 15

Attachments 12 & 13 1.8.W spectra 20us FSR

Attachment 14 ADC data item stats

Attachment 15 DSSSD bias & leakage current



Conclusions

ADC data item stats -> 300-400k

n+n 1.8.W spectra show large signal excursions

p+n 1.8.W spectra for biased Si wafers OK - unbiased Si wafers show large signal excursions


DSSSD#1 leakage currents 1.850+2.220+1.990=6.06uA    cf. 5.095uA with all 3 wafers biased 
DSSSD#2 leakage currents 1.510+1.580+2.350=5.44uA    cf. 5.235uA 


Right wafer (looking upstream)
             
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -1.135 -1.475 -1.645 -1.750 -1.805
DSSSD#2 I_l(uA) -1.625 -2.020 -2.205 -2.320 -2.360

Middle wafer (looking upstream)
          
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -1.540 -1.870 -2.035 -2.150 -2.210
DSSSD#2 I_l(uA) -0.985 -1.285 -1.465 -1.575 -1.615

Left wafer (looking upstream)
          
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -0.735 -0.915 -1.005 -1.080 -1.110
DSSSD#2 I_l(uA) -0.865 -1.205 -1.400 -1.510 -1.560

Bias versus leakage cuurent characteristic of each wafer of each DSSSD indicates diode behaviour.

Note LK1 not fitted - presumably LK3 of mid/top p+n FEE64 adaptor PCBs (aida01 and aida05) connecting n+n bias to gnd is sufficient to establish circuit.




15.50 Re-connect bias to all 3x Si wafers p+n junction side bias rings

ASIC check all FEE64s

Attachments 16 & 17 1.8.W spectra 20us FSR

Attachment 18 ADC data item stats

Attachment 19 DSSSD bias & leakage current

Conclusion - status quo ante restored



16.40 Remove aida01 & aida05 LK3, add aida04 & aida08 LK1

ASIC check all FEE64s

Attachments 20 & 21 1.8.W spectra 20us FSR

Attachment 22 ADC data item stats

Attachment 23 DSSSD bias & leakage current

Conclusion - no change



17.00 Add Thorn-Lab HV filters DSSSD#1 & #2


ASIC check all FEE64s

Attachments 24 & 25 1.8.W spectra 20us FSR

Attachment 26 ADC data item stats

Attachment 27 DSSSD bias & leakage current

Conclusion - aida09-aida12 no change, aida13-aida16 -> 300-400k
             I do not know why there's a difference between DSSSD#1 & #2 but more important point (probably) no improvement aida09-aida12
             Thorn Lab HV filters removed - status quo ante restored


18.02 Remove LK2 & LK4 all top p+n adaptor PCBs (aida13, 5, 14, 9, 1, & 10)

ASIC check all FEE64s

Attachments 28 & 29 1.8.W spectra 20us FSR

Attachment 30 ADC data item stats

Attachment 31 DSSSD bias & leakage current

Conclusion - no change cf. 16.40 test



19.10 Re-order power cabling at AIDA FEE64 PSUs from ...

Top    1-3 2-4 9-10 11-12
Bottom 5-7 6-8 13-14 15-16

to ...

Top    1-2 3-4 9-10 11-12
Bottom 5-6 7-8 13-14 15-16


ASIC check all FEE64s

Attachments 32 & 33 1.8.W spectra 20us FSR

Attachment 34 ADC data item stats

Attachment 35 DSSSD bias & leakage current

Conclusion - all p+n FEE64 rates c. 100k, or less, n+n FEE64 rates c. 400k

FEE64 adaptor PCB config

LK2-LK3-LK4 fitted for all bottom p+n FEE64s (aida11, 3, 12, 15, 7, 16)

LK1 fitted n+n FEE64s aida06 & aida02 *only*

11.10 Found aida08 not running - merger/TapeServer stalled

Restart required power cycle of all FEE64s
*all* ADC calibrations failed during system wide checks it was necessary to use FADC align & control (act on all FEE64 modules selected) to calibrate - first pass 
12/16 calibrated, successive individual re-calibrations calibrated remaining FEE64s

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x2f

BNC PB-5
amplitude 1.0V
attenuation x1
polarity +
tail pulse
tau_d 1ms
rate 25Hz

Attachments 1-4 1.8.W n+n spectra 20us, 200us, 2ms, 20ms FSR
Attachments 5-8 1.8.W p+n spectra 20ms, 2ms, 200us, 20us FSR

Attachment 9 per FEE64 1.8.L spectra n+n strips
 aida04 pulser peak width 387 ch FWHM
Attachment 10 per FEE64 1.8.L spectra p+n strips
 aida10 pulser peak width 93 ch FWHM

Attachment 11 per FEE64 Rate spectra

Attachments 12-15 all system wide checks OK
Attachment 16 ADC data item stats
Attachment 17 FEE64 temps OK
Attachment 18 DSSSD bias & leakage currents OK

15.45 check of ASIC settings cf. https://www2.ph.ed.ac.uk/~td/AIDA/Information/Default%20AIDA%20ASIC%20Parameters.txt

Differences from default AIDA ASIC parameters
--------------------------------------------- 

                                            Input Polarity
Offset Title                                Positive   Negative            Notes

12     slow comparator threshold            0x2f->0xa                     100keV nominal
 
14     fast comparator threshold HEC        0xf->0x02                     200MeV nominal
15     fast comparator threshold LEC/MEC    0xf->0xff                     > LEC/MEC nominal FSR

18     preAmp reference                     0xb2       0x20->0x30

21     Ibias LF feedback                    0x08       0xb->0xf

26     diode link threshold                 0xca->0xbf 0x11->0x23         decrease switch threshold

(front) p+n junction strips - positive input polarity
(back) n+n ohmic strips - negative input polarity

All looks OK - variations from default settings are those used at RIKEN

16:45 FEE64 adaptor PCB config

    LK1 LK2 LK3 LK4
p+n n/c o   *   o
n+n o   o   n/c o

o   = jumper fitted
n/c = jumper not fitted

*   = top, middle FEE64s - n+n bias ring ground for Si wafer #2
      bottom, middle FEE64s - DSSSD PCB gnd

DSSSD#1 & #2 bias via standard coaxial cable to top p+n FEE64 adaptor PCBs

17.05 aida09 or aida13 cable braid not secured to ground - cable tag dia too large - now sandwiched between correctly sized cable tags 
      no obvious difference to adc data item rates


17.20 LK1 removed for all n+n PCBs
      attachment 19 - adc data item stats

18.55 LK2/LK4 removed for all n+n PCBs
      attachment 20 - adc data item stats

19.10 power cycle FEE64s
      attachment 21 - adc data item stats

11.15 - S4 is currently closed no access

If access possible (should be) plan is

Re-attach ribbon cable braids
Re-attach DSSSD middle wafer rear g/r (LK3)
Re-attach DSSSD ground (LK2?)

See difference in rates, expectation is braid should restore most p+n rates

Once status quo ante achieved explore ASIC settings to identify preamp feedback and shaping time options to stabilise preamp output and optimise signal:noise.

aida12 Peak width =    84.73
aida02 Peak width =   526.68
Attachment 1 - Stats
Attachment 2 - Per FEE Rates
Attachments 3,4 - Waveforms
Attachments 5,6 - Pulser Peaks
Attachment 7 - HV Bias. Leakage current recovered and does not seem to be ramping up

16.00 - Attach LK3 jumpers to aidas 1, 3, 5, 7 (DSSD ground and middle rear g/r)

Attachment 8 - Stats
Attachment 9 - Per FEE Rates
Attachment 10, 11 - Waveforms
Attachment 12, 13 - Pulser Peaks

aida12 Peak width =    78.51
aida02 Peak width =   480.90

No major change, FEES 1-8 are all noisy and FEES 9-16 are quiet
aida01, aida05 waveforms very noticably bad (100 kHz), aida03, aida07 waveforms seem OK


16.45 - Attach LK2, LK4 jumpers to all FEEs (field plane grounding)
No significant changes

Attachment 14 - Stats
Attachment 15 - Per FEE Rates
Attachment 16, 17 - Pulser Peaks
Attachment 18, 19 - Waveforms

aida12 Peak width =    71.28
aida02 Peak width =   458.18

Remove LK3 jumpers 

aida12 Peak width =    71.43 
aida02 Peak width =   496.05


Attachment 20 - Stats
Attachment 21 - Per FEE Rates
Attachment 22, 23 - Pulser Peaks
Attachment 24, 25 - Waveforms

Re-add jumpers - attachments 26-28

aida12 Peak width =    87.42
aida02 Peak width =   572.24

17.45 Directly bias from the p+n FEEs using SHV->BNC->Lemo - attachments 29-33

aida12 Peak width =    87.87
aida02 Peak width =   504.41

The correlation between high 'ADC data item' rate and the noise observed on the corresponding 1.8.W spectrum is not clear *except* for aida01 & aida05 - see also attachments 34-35

ADC data item stats - attachment 36

DSSSD bias & leakage current OK - attachment 37

FEE64 temperatures OK - attachment 38

all system wide checks OK *except aida11 clock fail status 6

11.25 System wide checks OK except aida06 clock fail status 6 - attachments 1-13

      ADC data item stats - attachment 14

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage currents - attachment 16
  
      DSSSD#1 OK. DSSSD#2 leakage current has increased from c.5uA to c. 11uA overnight.


11.43 per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - attachments 17-24

      per FEE64 1.8.L spectra - attachments 25-26
          aida12 1.8.L pulser peak width 143 ch FWHM
          aida02 1.8.L pulser peak width 439 ch FWHM

      Per FEE64 rate spectra - attachment 27

12.00 Cycled DSSSD#2 bias from -100V to 0 to -100V leakage current -7.7uA but ramping slowly (c. 5nA/10s ... ?)

13.26 DSSSD#2 bias -100V leakage current -8.54uA

15.46 - Noticed that DSSSD#1 braid no longer has filter, DSSSD#1 core still does. DSSSD#2 is unfiltered on both.

      - Swap FEE64 power connectors #3 with #4, and #6 with #7, this means both n+n FEE64s are on the same paired channel

      No significant change to stats

      - Notice braid was only connected to FEEs 4 and 8, which meant beam-left bias guard ring 0v was floating. Use T to bring bias braid over to FEES 2 and 6 as well.
        Middle bias guard ring 0v is still floating

      No significant change to stats, amplitude of 100 kHz waveform in n+n waves seems attenuated though

     aida12 1.8.L pulser peak width 76 ch FWHM
     aida02 1.8.L pulser peak width 438 ch FWHM
 
     Attachment 28 - HV leakage current
     Attachment 29 - FEE ADC stats
     Attachment 30 - p+n side waveforms
     Attachment 31 - n+n side waveforms
     Attachment 32 - p+n pulser peak widths
     Attachment 33 - n+n pulser peak widths

15.55 - Reattach filter to DSSSD#1 braid

     No major change, possible improvement in waveforms 
     Attachment 34 - Stats
     Attachment 35,36 - waveforms

16.46 - Reattach LK1 jumper to FEES 2, 4, 6, 8

     Waveforms look OK, p+n rate slightly worse?
     Attachment 37 - Stats
     Attachment 38, 39 - waveforms
     Attachment 40 - Per FEE Rates

Tests performed

• After leaving overnight noticed the pulser was not fully inserted into aida09
  • Little change in rates observed in 9 but otherwise nothing else
• Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
  • No change observed in rates
• Also tested covering the snout in light tight material
  • No Changes
• Unplugged the system monitor from 6 and 8, no change
• Connected the snout to ground with copper tape
  • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
  • Removed the ground and powercycled and rates restored to where previously
• Changed the bias supply jumper inside to have the bias line connected to the rack ground
  • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
    • Recrimped and inserted again
  • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
  • Pulser widths however larger than yesterday

p+n width

n+n width

• Connected up entire pulser circuit to n+n FEEs

Switching bias supply back

• because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
• DAQ was then left overnight as had to leave to catch the bus
• Rates are observed to be back at their higher values - attachment 2
• n+n 1.8.L spectra - attachment 3
• p+n 1.8.L spectra - attachment 4

p+n widths

n+n widths

• The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

n+n

• Don't see a pulser peak in 4 or 2
• Will replace cable between 4 and 2 and 2 and 6
• Statistics at 0x64 with pulser on - atachment 5
• Now see peaks - attachment 6
• Statistics at 0xa - attachment 7

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

• Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
• 0x64 stats - attachment 7
• 0xa statistics - attachment 8
• p+n 1.8.L - attachment 9
• p+n waveform - attachment 10
• n+n waveform - attachment 11
  • Same high frequency component in both
  • 35 channel spacing corresponds to a frequency of ~1.4MHz

Next tests

• Returned braid to FEE 4 and 8
  • Recovered earlier rates and performance
• Re-added the ground cable to the snout
  • Much worse noise performance across all FEEs
  • p+n rates in 200k
• Removed the ground from the snout
  • Got really good rates originally - attachment 12
  • Noticed 1 and 5 failed ADC calibration
  • Calibrated them and there rates increased massively - attachment 13
• Notice that the pulser peak in FEE2 and 6 keeps dropping out
  • As the cables have been changed multiple times thought it could be that an adaptor board was broken
  • 4 seemed the most likely suspect with its high rates and triple peaks
  • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
  • Pulser peaks were still muissing
  • Issue could be then with adaptor board 2 possibly

Conclusions

• Rates on the FEEs are in a better place now than they were at the start of the week
  • p+n in particular have shown a large amount of improvement
• There are still however issues with the n+n FEEs
• I think there is likely an issue with one of the adaptor boards
  • Possible aida02 at the pulser connection point
  • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
  • Would recomend swapping out aida02 as a test next week
• Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
  • Nic would like to use something other than clingfilm in future
  • One suggestion would be mylar which could be held in place with strips of double sided tape
  • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
• Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
• I am at a slight loss of what else to try

• I have left the daq running at 0x64 overnight to check for alphas
• Histograms and statistics zeroed

Grounding implemented

• Each FEE given direct connection to heavy duty earth bar via adaptor board lemo ground
• Earth bars are then connected to platform ground on AIDA frame
• Photo of grounding - attachment 1
• Improvements not as large as expected from previous experience though
  • Particularly in the n+n side where widths still appear awful
• While implementing the grounding it was noticed the ground lemo socket on FEE7 was damaged and no longer gripped onto a lemo cable
  • It was noted that it would need changing at some point but continued with it for the time being
• Jumper configuration was n+n LK1-4 all on, p+n LK2-4 all on

p+n widths

• Compared to the width with no grounding the FEEs for the first DSSD (First 6) range from no change to slightly worse
• 1.8.L spectra - attachment 3

n+n widths

• Widths still very large
• Width for 8 is around 500
• 1.8.L attachment 2

Additional info

• Statistics - attachment 4

Jumper optimisation

• Went to our previously found optimal jumper configuration for the trips
• n+n LK1 only
• p+n LK 2 and 4 on all FEE
• p+n LK3 on non isolated cables (PCB ground on lower middle cable and n+n side bias on top middle cable)
• While changing the jumpers we also swapped out the damaged adaptor board on aida07 for a new one
• Saw an improvement for most FEEs

• Unfortunately saw no improvement for the n+n
• We event tested having the pulser only going into a single FEE (aida08)
• Still have a width of 622 for that FEE
• To compare FEEs we have layout 1 - attachment 5
• Layouot 1 on a common timescale - attachment 6
  • Note that the rate is typically dominated by a single strip or a couple of strips within a FEE
  • Many channels are just showing the pulser rate 25Hz
• p+n  1.8.L attachment 7
• p+n waveforms attachment 8
• n+n waveforms attachment 9
• Testing the noise on the n+n strips we observe noise with slow comparator set to 200 (2MeV)
• Meanwhile the p+n strips are silent at 0x64 for the most part.
• DAQ left running overnight with 0x64 on ASIC settings to see if we observe any alphas

Plan for the day

• Test daisy chaining the n+n across to the other n+n strips
  • Currently the n+n braid is only plugged into either aida06 or aida08 (depending on DSSD)
  • All of the jumpers are however grounded
  • Will still test
  • May investigate also trying this on LK3 on adaptors for aida01 and aida03
• Observe the output of the new sum inverter on the scope to check for variation in amplitude

Pulser spectra

• Changes to the nn pulser circuit yesterday have improved the peak widths on the p+n side
• p+n peak widths averact around 100 channels
• n+n around 1000 at 0xa threshold
• n+n 1.8.L spectra - attachment 1
• p+n 1.8.L spectra attachment 2

p+n peak widths

n+n peak widths

Daily checks

• Fee temperatures ok - attachment 3
• Bias and leakage currents ok - attachment 4
• Stats spectra with current grounding - attachment 5
  • Current grounding is just a cable going between each adaptor board and its respective FEE
  • Interfee grounding done by power supply and also pulser circuit
• Rates for each of the asics - attachment 6
• p+n waveforms - attachment 7
• n+n waveforms - attachment 8
  • Lots of noise evident on them

Plans for the day

• Connect FEEs up to the copper earth bars installed
• Powercycle and check noise levels
• If n+n remains significantly higher investigate the n+n pulser circuit further
• Investigate the effects of different jumper connetions At present n+n have all LK jumpers on and p+n have lk2-4

An interesting find with the new sum inverter is that with the pulser at default settings the pulser peak on the n+n side is above the range of the ADC.
We have checked signals from the sum inverter and pulser and both are equal in magnitude.
We therefor determine it is over range as the charge is spread over 4 FEEs vs 12 on the p+n side.
This possibly wasn't an issue with the previous inverter if it produced a lower amplitude.

For tests going forward p+n voltage will be 2V. n+n voltage will be 0.5V - Attachment 1
Going to look into a 20dB attenuator

Seems there were issues with the lemo cables in the n+n pulser chain

DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
Temperature menu is still responsive and all FEEs report a temperature
Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
Merger shows no data arriving as expected - Attachment 3
Merger data buffers for each FEE - attachment 4
Merger time errors - attachment 5

DAQ has not yet been restarted in case Patrick wanted to look.

> DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
> Temperature menu is still responsive and all FEEs report a temperature
> Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
> Merger shows no data arriving as expected - Attachment 3
> Merger data buffers for each FEE - attachment 4
> Merger time errors - attachment 5
> 
> DAQ has not yet been restarted in case Patrick wanted to look.

Looking at the first attachment..... aida09 has rebooted. So it is not setup at all.

This has caused the NewMerger to hang .... which surprises me .... as I thought it was designed to spot a bad FEE and continue with the rest.

Yes I will have a look today as I'm in Daresbury.

More later ......

Required packages not included as default

• csh (For running MIDAS scripts)
• xinetd (Runs the rdate server for the FEEs)
• Telnet (Communication with the FEEs)
• dhcp

Configuration changes

• FEEs uses NFS2 which is no longer enabled by default
  • Must edit config in /etc/nfs.conf to uncomment the following lines:
    • [nfsd]
    • vers2=y
• /etc/xinetd.d/time-stream must be edited by uncommenting

  • disable        = no

• Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

  • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

  • Likewise can check status with `systemctl status $SERVICE_NAME`

• Also need to check the firewall status of the ethernet port connected to the FEEs.

  • Easiest solution is to set the port to be trusted

Optional extras

• For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
  • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
  • Can also add screenshot to this - Attachment 2
• Gnome tweaks also allows the number of workspaces to be changed
• Also recommend setting workspaces to span displays

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 

Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24

 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6

During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.

- The adaptor boards have no connection to the FEE ground over the ERNI connector
   - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
    - Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
    - With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
    - The cost of this is the rate of FEE8 goes up to 300k
    - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
   

> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.

Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.

Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.

launcher1
TclHttpd @8015 for AIDA
/MIDAS/Linux/startup/HTTPD

launcher2
TclHttpd @8115 for AIDA
/MIDAS/Linux/startup/HTTPD@8115

launcher3
TapeServer
/MIDAS/Linux/startup/TapeServer

launcher4
New Merger for AIDA
/MIDAS/Linux/startup/NewMerger

Launcher5
MBS Relay
/MIDAS/Linux/startup/datarelaymbs

Proxy for firefox and anydesk

Contents of an email from NH to OH and TD about tests performed on an additional bias supply:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

so I looked at the other supply in the lab (in a NIM crate there, cable direct to DSO)

Still see the same noise wave - also it looked a lot worse at 50V but that may be a worse NIM crate or something else - the current also wasn't 0 (calibration?)

More unusually the signals remain even if the NIM crate is off (DSO_Lab_NoP) and when the SHV is removed from the HV completely (DSO_Lab_NoCable)

Investigate noise situation of AIDA (no DSSSDs connected)

Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
- aida16 very noisy ASICS 1&2, normal ASICS 3&4
- back-side aidas (02,04,06,08) worse resolution than front-side
- high-frequency noise still noticeable in waveforms

Figs 7-8: Remove pulser in (from NIM rack) only
- no noticable change

Figs 9-10: Remove all pulser cables from adapter boards
- looks improved

Figs 11-12: HV Braid->GND Jumpers removed

Fig 13-14: All HV cables removed
- aida06, aida14 isn't updating
- very noticable improvement

Fig 15-16: Intermediate HV cables re-added
- signal worse in FEE64s with HV cables
- aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good

Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.

Pulser widths now:
- aida14 (-ve amplitude) = 13.5 channels
- aida08 (+ve amplitude) = 40 channels (?)

11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
      Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
      Switch Cooknell SA1 Sum Amp output from negative to positive

      Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14

      This probably indicates
      - noisy Cooknell SA1
      - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1

> Investigate noise situation of AIDA (no DSSSDs connected)
> 
> Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> - back-side aidas (02,04,06,08) worse resolution than front-side
> - high-frequency noise still noticeable in waveforms
> 
> Figs 7-8: Remove pulser in (from NIM rack) only
> - no noticable change
> 
> Figs 9-10: Remove all pulser cables from adapter boards
> - looks improved
> 
> Figs 11-12: HV Braid->GND Jumpers removed
> 
> Fig 13-14: All HV cables removed
> - aida06, aida14 isn't updating
> - very noticable improvement
> 
> Fig 15-16: Intermediate HV cables re-added
> - signal worse in FEE64s with HV cables
> - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> 
> Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> 
> Pulser widths now:
> - aida14 (-ve amplitude) = 13.5 channels
> - aida08 (+ve amplitude) = 40 channels (?)

> 
> 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
>       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
>       Switch Cooknell SA1 Sum Amp output from negative to positive
> 
>       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> 
>       This probably indicates
>       - noisy Cooknell SA1
>       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> 
> > Investigate noise situation of AIDA (no DSSSDs connected)
> > 
> > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > - back-side aidas (02,04,06,08) worse resolution than front-side
> > - high-frequency noise still noticeable in waveforms
> > 
> > Figs 7-8: Remove pulser in (from NIM rack) only
> > - no noticable change
> > 
> > Figs 9-10: Remove all pulser cables from adapter boards
> > - looks improved
> > 
> > Figs 11-12: HV Braid->GND Jumpers removed
> > 
> > Fig 13-14: All HV cables removed
> > - aida06, aida14 isn't updating
> > - very noticable improvement
> > 
> > Fig 15-16: Intermediate HV cables re-added
> > - signal worse in FEE64s with HV cables
> > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > 
> > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > 
> > Pulser widths now:
> > - aida14 (-ve amplitude) = 13.5 channels
> > - aida08 (+ve amplitude) = 40 channels (?)

Removed the T-piece between pulser and summing amplifier
pulser FWHM still ~30 channels - not the T-piece, probably summing amp

Checked the noise of HV and LV on battery DSO
- LV - no noticeable noise on any channel (< 5 mV rms)
- HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
       independent of bias voltage 
       See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
                      AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2

Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 
 Check on one DSSSD, if see improvement can migrate to adapter boards in new revision

Things to do:
TD:
 Inline filters
NH:
 Check same behaviour other N1419ET HV supplies
 Check line voltages from CAEN NIM bin

> > 
> > 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
> >       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
> >       Switch Cooknell SA1 Sum Amp output from negative to positive
> > 
> >       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> > 
> >       This probably indicates
> >       - noisy Cooknell SA1
> >       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> > 
> > > Investigate noise situation of AIDA (no DSSSDs connected)
> > > 
> > > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > > - back-side aidas (02,04,06,08) worse resolution than front-side
> > > - high-frequency noise still noticeable in waveforms
> > > 
> > > Figs 7-8: Remove pulser in (from NIM rack) only
> > > - no noticable change
> > > 
> > > Figs 9-10: Remove all pulser cables from adapter boards
> > > - looks improved
> > > 
> > > Figs 11-12: HV Braid->GND Jumpers removed
> > > 
> > > Fig 13-14: All HV cables removed
> > > - aida06, aida14 isn't updating
> > > - very noticable improvement
> > > 
> > > Fig 15-16: Intermediate HV cables re-added
> > > - signal worse in FEE64s with HV cables
> > > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > > 
> > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > > 
> > > Pulser widths now:
> > > - aida14 (-ve amplitude) = 13.5 channels
> > > - aida08 (+ve amplitude) = 40 channels (?)
> 
> Removed the T-piece between pulser and summing amplifier
> pulser FWHM still ~30 channels - not the T-piece, probably summing amp
> 
> Checked the noise of HV and LV on battery DSO
> - LV - no noticeable noise on any channel (< 5 mV rms)
> - HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
>        independent of bias voltage 
>        See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
>                       AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2
> 
> Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
> 100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
> Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 

1nF/3k => 333kHz low pass rolloff

10nF/1M => 100Hz low pass rolloff

>  Check on one DSSSD, if see improvement can migrate to adapter boards in new revision
> 
> Things to do:
> TD:
>  Inline filters
> NH:
>  Check same behaviour other N1419ET HV supplies
>  Check line voltages from CAEN NIM bin

AIDA Unbiased and powered down
All FEEs were running before stoppage. None had rebooted.
Aida05 did not respond until merger was restarted, all other DAQs responded fine.

Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup
AIDA will remain in Snout in lab for time being.

FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now

The following runs are now copied to lustre
If necessary they can be removed from AIDA HDDs for space.
There are also spare HDDs that can (and should) be swapped for the current ones

/lustre/gamma/DESPEC_S452_FILES/AIDA
S452
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58

/lustre/gamma/DESPEC_S460_FILES/AIDA
S460
R1
R2
R4
R5
R7
R9
R11
R13
R14
R16
R20
R21
R23
R24
R25
R26
R27
R30
R31
R32
R33
R34
R35
R36
R38
R40
R45
R46
R48
R50
R51
R52

/lustre/gamma/DESPEC_S496_FILES/AIDA
S496
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58

11.21 - DAQ continues OK (no storage mode)

11.21 - System checks ok *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d95 : 	 89  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16


			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x9560 : 	 38240  
aida13 fault 	 0x0 : 	 0x15b9 : 	 5561  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     12     14      2      4      4      2      2      2      4      6   : 37636
aida02 :     40      7     15      1      3      4      3      2      2      4      6   : 37800
aida03 :     39      7      9      6      5      3      1      3      2      4      6   : 37860
aida04 :     27      7      5      3      3      4      2      3      2      4      6   : 37908
aida05 :     33      5      5      3      6      4      3      3      2      4      6   : 38364
aida06 :     45     12      3      4      4      2      2      3      4      3      6   : 37828
aida07 :     31     11      5      4      4      2      3      2      2      4      6   : 37540
aida08 :     31      6      8      5      3      4      3      2      2      4      6   : 37772
aida09 :     45      4      9      3      2      3      3      2      2      4      6   : 37572
aida10 :     37     11     10      2      3      4      3      3      3      3      6   : 37260
aida11 :     40      9     11      3      3      4      3      3      1      4      6   : 37304
aida12 :     44     11      6      5      3      4      3      3      1      4      6   : 37320
aida13 :     41     11      9      3      5      3      3      3      1      4      6   : 37292
aida14 :     31     12      4      6      5      3      3      3      3      3      6   : 37276
aida15 :     17      9      7      1      2      2      3      2      1      4      7   : 40348
aida16 :     41     12      5      4      5      3      3      3      1      4      6   : 37268


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


11.23 - figures
    Grafana temp & leakage - attachment 1
    Weather cooler this week in Germany, expect S4 to cool down eventually
    Lost activity monitor - attachment 2
    1.8.W spectra - 20us FSR - attachments 3 & 4
    aida05 looks very noisy (?) 
    ADC data items - attachment 5
    FEE64 temperatures OK - attachment 6
    DSSSD bias & leakage currents - attachment 7
    Merger/Tape Server/Merger statistics - attachment 8
    Looks some new time seq error since Thursday

Experiment is runnign in S4 so no access possible this week

Data is being archived to lustre (GSI PB file storage) 

12.15 DAQ continues OK - no storage mode

12.15 System checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d6 : 	 13014  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     31     11      9      4      4      4      2      2      2      4      6   : 37604
aida02 :     30     10     10      3      3      4      3      2      2      4      6   : 37768
aida03 :     35     11      8      6      5      3      2      2      3      3      6   : 36580
aida04 :     33      9      4      4      3      4      2      3      3      3      6   : 36940
aida05 :     41      9      7      3      4      4      2      2      2      4      6   : 37564
aida06 :     38      6      5      3      2      3      2      3      4      3      6   : 37752
aida07 :     25     12      6      5      4      2      3      2      2      4      6   : 37572
aida08 :     32      7     10      5      4      3      3      2      2      4      6   : 37752
aida09 :     41      8      6      4      2      3      3      2      2      4      6   : 37572
aida10 :     31     10     11      5      5      3      3      3      3      3      6   : 37340
aida11 :     29      7     12      3      4      4      3      3      1      4      6   : 37324
aida12 :     42      8      7      5      3      4      3      3      1      4      6   : 37304
aida13 :     43     10      4      5      5      3      3      3      1      4      6   : 37276
aida14 :     40     15      4      5      4      4      3      3      3      3      6   : 37368
aida15 :     13      5      8      2      1      2      3      2      1      4      7   : 40284
aida16 :     39      9      6      6      5      3      3      3      1      4      6   : 37316

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

12.15 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

19.04 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8960 : 	 28  
aida02 fault 	 0x9b36 : 	 0x9b52 : 	 28  
aida03 fault 	 0xf26b : 	 0xf287 : 	 28  
aida04 fault 	 0x57a0 : 	 0x57bc : 	 28  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d4 : 	 13012  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     53     10      4      4      4      4      2      2      3      3      6   : 36580
aida02 :     28      7     10      4      3      4      3      2      2      4      6   : 37768
aida03 :     41     10      7      6      5      3      1      3      3      3      6   : 36836
aida04 :     45      7      2      4      3      4      2      3      3      3      6   : 36940
aida05 :     35      8      8      4      4      4      2      2      2      4      6   : 37580
aida06 :     45      6      6      3      3      3      2      3      4      3      6   : 37860
aida07 :     37     10      6      4      5      2      3      2      2      4      6   : 37636
aida08 :     32      5     10      4      4      4      3      2      2      4      6   : 37832
aida09 :     24      5      8      7      1      3      3      2      2      4      6   : 37544
aida10 :     29      9     12      5      4      3      3      3      3      3      6   : 37276
aida11 :     33      7      7      6      3      4      3      3      1      4      6   : 37292
aida12 :     36     11      6      6      3      4      3      3      1      4      6   : 37320
aida13 :     37      7      6      6      5      3      3      3      1      4      6   : 37292
aida14 :     25      9      7      6      5      3      3      3      3      3      6   : 37276
aida15 :      3      3      4      3      0      3      3      2      1      4      7   : 40260
aida16 :     35      7      6      6      4      4      3      3      1      4      6   : 37348

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19.07 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

13.35 DAQ continues OK - no storage mode

      All system wide checks OK *except*
	
		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8949 : 	 5  
aida02 fault 	 0x9b36 : 	 0x9b3b : 	 5  
aida03 fault 	 0xf26b : 	 0xf270 : 	 5  
aida04 fault 	 0x57a0 : 	 0x57a5 : 	 5  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
aida15 fault 	 0x4e4 : 	 0x1267 : 	 3459  
White Rabbit error counter test result: Passed 7, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x3e01 : 	 15873  
aida13 fault 	 0x0 : 	 0x32d2 : 	 13010  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     37     10      8      5      4      4      2      2      2      4      6   : 37636
aida02 :     37     10     10      3      4      4      3      2      2      4      6   : 37860
aida03 :     41      8      7      7      5      3      1      3      2      4      6   : 37876
aida04 :     33      9      4      4      2      5      2      3      3      3      6   : 37004
aida05 :     37      9      9      3      5      4      2      2      2      4      6   : 37644
aida06 :     39      9      6      5      4      2      2      3      4      3      6   : 37860
aida07 :     29     12      6      5      4      2      3      2      2      4      6   : 37588
aida08 :     25      7      9      5      4      3      3      2      2      4      6   : 37708
aida09 :     34      5      9      6      2      3      3      2      2      4      6   : 37632
aida10 :     34     12     11      5      4      3      3      3      3      3      6   : 37304
aida11 :     38      8      7      6      3      4      3      3      1      4      6   : 37320
aida12 :     34     10      8      6      3      4      3      3      1      4      6   : 37336
aida13 :     33     11      5      6      4      4      3      3      1      4      6   : 37356
aida14 :     31     10      5      6      5      3      3      3      3      3      6   : 37276
aida15 :     14      6      7      4      1      2      3      2      1      4      7   : 40344
aida16 :     33      8      6      7      5      3      3      3      1      4      6   : 37316

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

13.40 All histograms, stats & merger stats zero'd

18.15 DAQ continues file S496/R32_946
      Hot HEC channels multiple FEE64s from c. 06.00 this morning
      ASIC check fixes

      Out of disk space on /TapeData filesystem - switch to no storage mode

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8949 : 	 5  
aida02 fault 	 0x9b36 : 	 0x9b3b : 	 5  
aida03 fault 	 0xf26b : 	 0xf270 : 	 5  
aida04 fault 	 0x57a0 : 	 0x57a5 : 	 5  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
aida15 fault 	 0x4e4 : 	 0x1267 : 	 3459  
White Rabbit error counter test result: Passed 7, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x3e01 : 	 15873  
aida13 fault 	 0x0 : 	 0x32d2 : 	 13010  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26     16      4      6      4      4      2      2      2      4      6   : 37608
aida02 :     29     11      9      3      3      4      3      2      2      4      6   : 37756
aida03 :     32      9      6      8      5      3      1      3      3      3      6   : 36840
aida04 :     32      8      3      4      3      4      2      3      3      3      6   : 36912
aida05 :     36      8      6      4      5      4      2      2      2      4      6   : 37616
aida06 :     25     13      4      3      2      3      2      3      4      3      6   : 37740
aida07 :     28     13      4      5      5      2      3      2      2      4      6   : 37624
aida08 :     29      5      9      4      3      4      3      2      2      4      6   : 37740
aida09 :     41      8      4      7      1      3      3      2      2      4      6   : 37572
aida10 :     45     11      7      5      5      3      3      3      3      3      6   : 37340
aida11 :     33     13      5      6      3      4      3      3      1      4      6   : 37308
aida12 :     39      8      6      6      4      4      3      3      1      4      6   : 37372
aida13 :     28     14      3      6      4      4      3      3      1      4      6   : 37328
aida14 :     30     11      3      6      4      4      3      3      3      3      6   : 37312
aida15 :      3      3      4      3      0      3      3      2      1      4      7   : 40260
aida16 :     28      9      5      6      5      3      3      3      1      4      6   : 37256

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

04/11:03:40|get_ASICBlk : Blk=1 : bytes = 196096 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500240 : 1 => 0F07C000 
04/11:03:41|Last Data : Databuffer  273286648 : 0 => 80500240 : 1 => 0AFF8000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:42|Machine check in kernel mode.
12/15:30:08|I-Cache Parity Error
12/15:30:08|Oops: Machine check, sig: 7 [#1]
12/15:30:08|PREEMPT Xilinx Virtex440
12/15:30:08|Modules linked in: aidamem xdriver xh_spidev_register
12/15:30:08|NIP: c0055c54 LR: c00558fc CTR: c000ae5c
12/15:30:08|REGS: c6ff9f10 TRAP: 0214   Not tainted  (2.6.31)
12/15:30:08|MSR: 00021000 <ME,CE>  CR: 82000028  XER: 20000000
12/15:30:08|TASK = c6ac44e0[375] 'AidaExecV9' THREAD: c6338000
12/15:30:08|GPR00: 00000000 c6339cd0 c6ac44e0 ffffffff ffe79600 ffffffff ffe79600 ffffffff 
12/15:30:08|GPR08: 00000000 0000001f ffffffe0 00000000 00000000 1005520c 1dcd6500 00000000 
12/15:30:08|GPR16: 00000000 1009df98 100f0000 c0390000 000007a1 7fe9e580 ffffffff ffffffff 
12/15:30:08|GPR24: c0390000 00000001 00000000 001366ca c036f0d0 ffffffe1 ffffffff ffec9936 
12/15:30:08|NIP [c0055c54] update_wall_time+0x40c/0xd3c
12/15:30:09|LR [c00558fc] update_wall_time+0xb4/0xd3c
12/15:30:09|Call Trace:
12/15:30:09|[c6339cd0] [c0055f44] update_wall_time+0x6fc/0xd3c (unreliable)
12/15:30:09|[c6339d90] [c0040b84] do_timer+0x38/0x4c
12/15:30:09|[c6339da0] [c005a088] tick_periodic+0xbc/0xe8
12/15:30:09|[c6339db0] [c005a0d4] tick_handle_periodic+0x20/0x120
12/15:30:09|[c6339df0] [c000af70] timer_interrupt+0xa4/0x10c
12/15:30:09|[c6339e10] [c000e9c4] ret_from_except+0x0/0x18
12/15:30:09|[c6339ed0] [d10751dc] aidamem_read+0x28/0x108 [aidamem]
12/15:30:09|[c6339ef0] [c0097e80] vfs_read+0xb4/0x188
12/15:30:09|[c6339f10] [c00982f8] sys_read+0x4c/0x90
12/15:30:09|[c6339f40] [c000e364] ret_from_syscall+0x0/0x3c
12/15:30:09|Instruction dump:
12/15:30:09|39600000 354bffe0 418008b0 7cc35030 38800000 2f8a0000 419c08bc 7ffe5030 
12/15:30:09|3aa00000 93c10048 92a1004c 83c10048 <83e1004c> 7d095830 4800056c 419e06bc 
12/15:30:09|Kernel panic - not syncing: Fatal exception in interrupt
12/15:30:09|Call Trace:
12/15:30:09|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
12/15:33:10|
Found 0 errors checking kernel image
12/15:33:11|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
12/15:33:11|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
12/15:33:11|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
12/15:33:11|SPIconfig done now to set up the LMK3200 registers
12/15:33:12|LMK #0 : regInit[0]=0x80000000
12/15:33:12|LMK #0 : regInit[1]=0x10070600
12/15:33:12|LMK #0 : regInit[2]=0x60601
12/15:33:12|LMK #0 : regInit[3]=0x60602
12/15:33:12|LMK #0 : regInit[4]=0x60603
12/15:33:12|LMK #0 : regInit[5]=0x70624
12/15:33:12|LMK #0 : regInit[6]=0x70605
12/15:33:12|LMK #0 : regInit[7]=0x70606
12/15:33:12|LMK #0 : regInit[8]=0x70627
12/15:33:12|LMK #0 : regInit[9]=0x10000908
12/15:33:12|LMK #0 : regInit[10]=0xA0022A09
12/15:33:12|LMK #0 : regInit[11]=0x82800B
12/15:33:12|LMK #0 : regInit[12]=0x28C800D
12/15:33:12|LMK #0 : regInit[13]=0x830020E
12/15:33:12|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
12/15:33:12|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
12/15:33:12|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
12/15:33:13|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
ADCs calibrated
12/15:33:13|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
12/15:33:13|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
12/15:33:13|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
12/15:33:16|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 
12/15:33:23|Using Xilinx Virtex440 machine description
12/15:33:23|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
12/15:33:23|Zone PFN ranges:
12/15:33:23|  DMA      0x00000000 -> 0x00007000
12/15:33:23|  Normal   0x00007000 -> 0x00007000
12/15:33:23|Movable zone start PFN for each node
12/15:33:23|early_node_map[1] active PFN ranges
12/15:33:23|    0: 0x00000000 -> 0x00007000
12/15:33:23|MMU: Allocated 1088 bytes of context maps for 255 contexts
12/15:33:23|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
12/15:33:24|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
12/15:33:24|PID hash table entries: 512 (order: 9, 2048 bytes)
12/15:33:24|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
12/15:33:24|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
12/15:33:24|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
12/15:33:24|Kernel virtual memory layout:
12/15:33:24|  * 0xffffe000..0xfffff000  : fixmap
12/15:33:24|  * 0xfde00000..0xfe000000  : consistent mem
12/15:33:24|  * 0xfde00000..0xfde00000  : early ioremap
12/15:33:24|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
12/15:33:24|NR_IRQS:512
12/15:33:24|clocksource: timebase mult[a00000] shift[22] registered
12/15:33:24|Console: colour dummy device 80x25
12/15:33:24|Mount-cache hash table entries: 512
12/15:33:24|NET: Registered protocol family 16
12/15:33:24|PCI: Probing PCI hardware
12/15:33:24|bio: create slab <bio-0> at 0
12/15:33:24|NET: Registered protocol family 2
12/15:33:24|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
12/15:33:24|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
12/15:33:25|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
12/15:33:25|TCP: Hash tables configured (established 4096 bind 4096)
12/15:33:25|TCP reno registered
12/15:33:25|NET: Registered protocol family 1
12/15:33:25|ROMFS MTD (C) 2007 Red Hat, Inc.
12/15:33:25|msgmni has been set to 214
12/15:33:25|io scheduler noop registered
12/15:33:25|io scheduler anticipatory registered
12/15:33:25|io scheduler deadline registered
12/15:33:25|io scheduler cfq registered (default)
12/15:33:25|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
12/15:33:25|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
12/15:33:25|console [ttyS0] enabled
12/15:33:25|brd: module loaded
12/15:33:25|loop: module loaded
12/15:33:25|Device Tree Probing 'ethernet'
12/15:33:25|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
12/15:33:25|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
12/15:33:25|XLlTemac: DCR address: 0x80
12/15:33:25|XLlTemac: buffer descriptor size: 32768 (0x8000)
12/15:33:25|XLlTemac: Allocating DMA descriptors with kmalloc
12/15:33:25|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
12/15:33:26|XTemac: PHY detected at address 7.
12/15:33:26|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
12/15:33:26|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26|Using buffer write method
12/15:33:26|cfi_cmdset_0001: Erase suspend on write enabled
12/15:33:26|cmdlinepart partition parsing not available
12/15:33:26|RedBoot partition parsing not available

14.11 DAQ continues file S496/R32_697
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8948 : 	 4  
aida02 fault 	 0x9b36 : 	 0x9b3a : 	 4  
aida03 fault 	 0xf26b : 	 0xf26f : 	 4  
aida04 fault 	 0x57a0 : 	 0x57a4 : 	 4  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x21be : 	 8638  
aida13 fault 	 0x0 : 	 0x2294 : 	 8852  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     10      9      2      5      4      2      2      2      4      6   : 37604
aida02 :     46     12      7      2      4      4      3      2      2      4      6   : 37832
aida03 :     34     10      9      5      4      3      1      3      3      3      6   : 36744
aida04 :     35     12     10      1      3      4      2      3      3      3      6   : 36972
aida05 :     37      9      8      3      4      4      2      2      2      4      6   : 37564
aida06 :     34     10      5      5      4      2      2      3      2      4      6   : 37832
aida07 :     31     11      1      7      4      2      3      2      2      4      6   : 37572
aida08 :     40      7      4      5      3      4      3      2      2      4      6   : 37752
aida09 :     39      9      8      4      1      3      3      2      2      4      6   : 37540
aida10 :     43     12      8      4      4      3      3      3      1      4      6   : 37260
aida11 :     34     14     10      5      3      4      3      3      1      4      6   : 37368
aida12 :     46     12      3      4      4      4      3      3      1      4      6   : 37320
aida13 :     41     11      5      4      5      3      3      3      1      4      6   : 37260
aida14 :     39     10      2      6      4      4      3      3      1      4      6   : 37324
aida15 :     19     12      4      3      2      2      3      2      2      4      6   : 37324
aida16 :     45     10      6      4      5      3      3      3      1      4      6   : 37284

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.05 analysis of file S496/R32_698 - attachment 9
      max dead time 0.05% aida04

15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365

      successful DAQ stop
      aida15 DAQ reset, setup, go 
      successful DAQ go 

12.26 DAQ continues file S496/R32_428
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     11      6      3      4      4      2      2      2      4      6   : 37540
aida02 :     30      8      7      3      4      4      3      2      2      4      6   : 37768
aida03 :     38      8     13      3      4      3      1      3      3      3      6   : 36744
aida04 :     37      9      7      2      3      4      2      3      3      3      6   : 36940
aida05 :     29      5      9      3      3      3      1      3      2      4      6   : 37580
aida06 :     38      6      2      3      3      3      2      3      2      4      6   : 37768
aida07 :     31      9      2      7      4      2      3      2      2      4      6   : 37572
aida08 :     34      6      3      5      4      4      3      2      2      4      6   : 37768
aida09 :     29      2     12      5      1      3      3      2      2      4      6   : 37540
aida10 :     24      8     15      3      5      3      3      3      1      4      6   : 37296
aida11 :     35      4     15      3      3      4      3      3      1      4      6   : 37308
aida12 :     36      5      3      5      6      4      3      3      1      4      6   : 37384
aida13 :     31     10      4      3      6      4      3      3      1      4      6   : 37356
aida14 :     37      7      1      4      5      4      3      3      1      4      6   : 37276
aida15 :     34     10      6      3      4      1      2      4      2      4      6   : 38168
aida16 :     31      9      3      3      5      4      3      3      1      4      6   : 37268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 202

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.07 analysis of file S496/R32_431 - attachment 9
      max dead time aida04 0.33%

12.30 DAQ continues OK file S496/R32_373
 
      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5     12      7      2      4      2      2      1      4      7   : 40572
aida02 :     11      6      8      6      1      3      3      2      1      4      7   : 40540
aida03 :     26      5     11      5      1      2      2      3      2      3      7   : 39712
aida04 :     18      7      8      3      2      3      2      3      2      3      7   : 39776
aida05 :     11      5      9      4      1      2      2      3      1      4      7   : 40612
aida06 :     17      3      6      5      5      2      1      3      1      4      7   : 40604
aida07 :      9      3      4      9      1      3      3      2      1      4      7   : 40540
aida08 :     11      8      5      5      3      3      3      2      1      4      7   : 40604
aida09 :     13      5      8      5      2      3      3      2      1      4      7   : 40572
aida10 :     13      8     13      5      1      2      2      2      1      4      7   : 40228
aida11 :     11      5     14      6      3      3      3      3      2      3      7   : 40244
aida12 :     10      9      5      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      7      7      4      3      4      3      3      2      3      7   : 40228
aida14 :      9      6      7      6      3      4      3      3      2      3      7   : 40260
aida15 :     14      3      8      5      1      2      2      4      1      4      7   : 41136
aida16 :     15      4      9      4      4      4      3      3      2      3      7   : 40300

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      1.8.L specttra - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      Lost activity monitor - attachment 4
      1.8.W spectra - 20us FSR - attachments 5 & 6
      ADC data items - attachment 7
      FEE64 temperatures OK - attachment 8
      DSSSD bias & leakage currents OK - attachment 9
      Merger/Tape Server/Merger statistics - attachment 10
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_372 - attachment 11
       max deadtime 0.07% aida04

11.52 DAQ continues file S496/R32_340
      High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning

11.53 ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      3     12      8      3      3      2      2      1      4      7   : 40508
aida02 :      5      5     10      6      1      3      3      2      1      4      7   : 40540
aida03 :     25      7     12      5      1      2      2      3      2      3      7   : 39740
aida04 :     19      6     11      5      1      3      2      3      2      3      7   : 39820
aida05 :      9      6      7      5      0      2      2      3      1      4      7   : 40548
aida06 :     10      8      7      4      3      3      1      3      1      4      7   : 40600
aida07 :      9      3      4      8      1      3      3      2      1      4      7   : 40508
aida08 :     12      5      8      5      3      3      3      2      1      4      7   : 40632
aida09 :     15      4      8      5      1      3      3      2      1      4      7   : 40508
aida10 :     23      5     12      5      1      2      2      2      1      4      7   : 40228
aida11 :     15      9     11      6      2      4      3      3      2      3      7   : 40308
aida12 :     14      9      4      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      9      6      3      4      4      3      3      2      3      7   : 40260
aida14 :     11      5      7      6      4      4      3      3      2      3      7   : 40324
aida15 :     16      8      7      4      1      2      2      4      1      4      7   : 41136
aida16 :     15      8      7      4      3      4      3      3      2      3      7   : 40236

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_339 - attachment 9
       max deadtime 0.24% aida04

11.30 DAQ continues OK - file S496/R32_144
      alpha background

      All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      5     11      6      3      4      2      2      1      4      7   : 40572
aida02 :      1      3      6      7      2      3      3      2      1      4      7   : 40540
aida03 :     21      7      7      8      2      4      1      3      2      3      7   : 39804
aida04 :     13      7      5      6      1      3      2      3      2      3      7   : 39740
aida05 :      2      8      6      5      1      2      2      3      1      4      7   : 40584
aida06 :     10      4      5      5      4      3      1      3      1      4      7   : 40632
aida07 :      9      3      4      7      3      3      3      2      1      4      7   : 40604
aida08 :     11      2      6      4      4      3      3      2      1      4      7   : 40604
aida09 :     10      2      5      4      2      2      2      3      1      4      7   : 40584
aida10 :     14      5      8      6      2      1      1      3      1      4      7   : 40352
aida11 :      8      6      9      5      4      4      3      3      2      3      7   : 40320
aida12 :     10      5      3      7      3      3      2      2      1      4      7   : 40352
aida13 :      7      5      4      6      5      4      3      3      2      3      7   : 40324
aida14 :      7      5      4      8      4      4      3      3      2      3      7   : 40324
aida15 :      8      4      7      4      2      2      2      4      1      4      7   : 41136
aida16 :     13      5      7      6      3      4      3      3      2      3      7   : 40268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

11.35 analysis of file S496/R32_143 - attachment 9
       max deadtime 0.29% aida03

14.15 DAQ continues file S496/R32_117
      Hot HEC channels multiple FEE64s from c. 06.00 this morning
      ASIC check fixes

      System wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15     12     12      3      3      4      2      2      1      4      7   : 40572
aida02 :      7      8     10      5      2      3      3      2      1      4      7   : 40604
aida03 :     27     10      4      8      1      4      1      3      2      3      7   : 39740
aida04 :     18      7      4      5      1      3      2      3      2      3      7   : 39712
aida05 :     15     13      8      4      1      3      1      3      1      4      7   : 40548
aida06 :     14      8      8      6      3      2      1      3      1      4      7   : 40568
aida07 :     11      4      5      6      2      3      3      2      1      4      7   : 40540
aida08 :     15      6      5      3      4      3      3      2      1      4      7   : 40604
aida09 :     18      4      4      3      3      2      2      3      1      4      7   : 40648
aida10 :     12      6      8      6      1      1      1      3      1      4      7   : 40288
aida11 :     16      8      6      6      4      4      3      3      2      3      7   : 40352
aida12 :     10      7      4      8      3      4      3      3      2      3      7   : 40288
aida13 :     15      7      5      4      5      4      3      3      2      3      7   : 40324
aida14 :      9      4      6      7      3      4      3      3      2      3      7   : 40260
aida15 :     12      4      8      3      2      2      2      4      1      4      7   : 41136
aida16 :     11      8      6      6      3      4      3      3      2      3      7   : 40268

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_118 - attachment 9
       max deadtime 0.28% aida04

12.25 DAQ continues OK - file S496/R32_48
      alpha background

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     10      6      1      2      4      4      2      2      1      4      7   : 40360
aida02 :      4      3      3      3      3      4      2      2      1      4      7   : 40312
aida03 :     13      5      4      4      2      4      1      3      2      3      7   : 39580
aida04 :     16      5      3      3      2      4      1      3      2      3      7   : 39544
aida05 :      8      6      1      2      3      3      1      3      1      4      7   : 40416
aida06 :      4      3      2      1      3      3      1      3      1      4      7   : 40360
aida07 :      6      6      2      4      4      2      3      2      1      4      7   : 40424
aida08 :      6      4      1      1      4      3      3      2      1      4      7   : 40424
aida09 :     16      1      3      1      0      2      2      3      1      4      7   : 40344
aida10 :      2      5      1      4      1      3      2      4      2      3      7   : 40064
aida11 :     14      3      2      1      4      4      3      3      2      3      7   : 40080
aida12 :     11      2      2      4      4      4      3      3      2      3      7   : 40156
aida13 :     11      3      2      2      3      4      3      3      2      3      7   : 40036
aida14 :      6      5      0      4      3      4      3      3      2      3      7   : 40064
aida15 :     11      6      3      1      3      1      2      4      1      4      7   : 40940
aida16 :     12      5      0      4      4      4      3      3      2      3      7   : 40152

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

      analysis of file S496/R32_47 - attachment 8
       max deadtime 0.35% aida04

13.56 Lost activity monitor - see attachment 9

03/23:58:55|kernel BUG at mm/slab.c:2974!
03/23:58:55|Oops: Exception in kernel mode, sig: 5 [#1]
03/23:58:55|PREEMPT Xilinx Virtex440
03/23:58:55|Modules linked in: aidamem xdriver xh_spidev_register
03/23:58:55|NIP: c009211c LR: c00920b0 CTR: c0038e70
03/23:58:55|REGS: c631fc60 TRAP: 0700   Not tainted  (2.6.31)
03/23:58:55|MSR: 00021000 <ME,CE>  CR: 22022048  XER: 00000000
03/23:58:56|TASK = c627a0c0[375] 'AidaExecV9' THREAD: c631e000
03/23:58:56|GPR00: 00000001 c631fd10 c627a0c0 c680daf0 00000020 00000001 c6940020 00000000 
03/23:58:56|GPR08: fffffff6 00010002 c680e400 c631e000 22008022 1005520c c03a0000 00000020 
03/23:58:56|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 
03/23:58:56|GPR24: 00100100 c631e000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400 
03/23:58:56|NIP [c009211c] cache_alloc_refill+0x130/0x608
03/23:58:56|LR [c00920b0] cache_alloc_refill+0xc4/0x608
03/23:58:56|Call Trace:
03/23:58:56|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
03/23:58:56|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
03/23:58:56|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8
03/23:58:56|[c631fdb0] [c0042938] __send_signal+0x78/0x260
03/23:58:56|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c
03/23:58:56|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c
03/23:58:56|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30
03/23:58:56|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240
03/23:58:56|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c
03/23:58:56|[c631feb0] [c0040bf4] update_process_times+0x30/0x70
03/23:58:57|[c631fed0] [c005a000] tick_periodic+0x34/0xe8
03/23:58:57|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120
03/23:58:57|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c
03/23:58:57|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18
03/23:58:57|Instruction dump:
03/23:58:57|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 
03/23:58:57|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 
03/23:58:57|Kernel panic - not syncing: Fatal exception in interrupt
03/23:58:57|Call Trace:
03/23:58:57|[c631fab0] [c0005de8] show_stack+0x44/0x16c (unreliable)
03/23:58:57|[c631faf0] [c00345bc] panic+0x94/0x168
03/23:58:57|[c631fb40] [c000bd44] die+0x178/0x18c
03/23:58:57|[c631fb60] [c000c000] _exception+0x164/0x1b4
03/23:58:57|[c631fc50] [c000e978] ret_from_except_full+0x0/0x4c
03/23:58:57|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608
03/23:58:57|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
03/23:58:57|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8
03/23:58:57|[c631fdb0] [c0042938] __send_signal+0x78/0x260
03/23:58:57|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c
03/23:58:58|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c
03/23:58:58|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30
03/23:58:58|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240
03/23:58:58|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c
03/23:58:58|[c631feb0] [c0040bf4] update_process_times+0x30/0x70
03/23:58:58|[c631fed0] [c005a000] tick_periodic+0x34/0xe8
03/23:58:58|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120
03/23:58:58|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c
03/23:58:58|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18
03/23:58:58|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
04/00:01:59|
Found 0 errors checking kernel image
04/00:02:00|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
04/00:02:00|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
04/00:02:00|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
04/00:02:00|SPIconfig done now to set up the LMK3200 registers
04/00:02:00|LMK #0 : regInit[0]=0x80000000
04/00:02:00|LMK #0 : regInit[1]=0x10070600
04/00:02:00|LMK #0 : regInit[2]=0x60601
04/00:02:00|LMK #0 : regInit[3]=0x60602
04/00:02:00|LMK #0 : regInit[4]=0x60603
04/00:02:00|LMK #0 : regInit[5]=0x70624
04/00:02:00|LMK #0 : regInit[6]=0x70605
04/00:02:00|LMK #0 : regInit[7]=0x70606
04/00:02:00|LMK #0 : regInit[8]=0x70627
04/00:02:00|LMK #0 : regInit[9]=0x10000908
04/00:02:00|LMK #0 : regInit[10]=0xA0022A09
04/00:02:00|LMK #0 : regInit[11]=0x82800B
04/00:02:01|LMK #0 : regInit[12]=0x28C800D
04/00:02:01|LMK #0 : regInit[13]=0x830020E
04/00:02:01|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
04/00:02:01|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
04/00:02:01|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
04/00:02:01|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
04/00:02:01|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
04/00:02:05|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 
04/00:02:11|Using Xilinx Virtex440 machine description
04/00:02:12|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
04/00:02:12|Zone PFN ranges:
04/00:02:12|  DMA      0x00000000 -> 0x00007000
04/00:02:12|  Normal   0x00007000 -> 0x00007000
04/00:02:12|Movable zone start PFN for each node
04/00:02:12|early_node_map[1] active PFN ranges
04/00:02:12|    0: 0x00000000 -> 0x00007000
04/00:02:12|MMU: Allocated 1088 bytes of context maps for 255 contexts
04/00:02:12|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
04/00:02:12|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
04/00:02:12|PID hash table entries: 512 (order: 9, 2048 bytes)
04/00:02:12|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
04/00:02:12|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
04/00:02:12|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
04/00:02:12|Kernel virtual memory layout:
04/00:02:12|  * 0xffffe000..0xfffff000  : fixmap
04/00:02:12|  * 0xfde00000..0xfe000000  : consistent mem
04/00:02:12|  * 0xfde00000..0xfde00000  : early ioremap
04/00:02:13|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
04/00:02:13|NR_IRQS:512
04/00:02:13|clocksource: timebase mult[a00000] shift[22] registered
04/00:02:13|Console: colour dummy device 80x25
04/00:02:13|Mount-cache hash table entries: 512
04/00:02:13|NET: Registered protocol family 16
04/00:02:13|PCI: Probing PCI hardware
04/00:02:13|bio: create slab <bio-0> at 0
04/00:02:13|NET: Registered protocol family 2
04/00:02:13|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
04/00:02:13|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
04/00:02:13|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
04/00:02:13|TCP: Hash tables configured (established 4096 bind 4096)
04/00:02:13|TCP reno registered
04/00:02:13|NET: Registered protocol family 1
04/00:02:13|ROMFS MTD (C) 2007 Red Hat, Inc.
04/00:02:13|msgmni has been set to 214
04/00:02:13|io scheduler noop registered
04/00:02:13|io scheduler anticipatory registered
04/00:02:13|io scheduler deadline registered
04/00:02:13|io scheduler cfq registered (default)
04/00:02:13|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
04/00:02:13|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
04/00:02:14|console [ttyS0] enabled
04/00:02:14|brd: module loaded
04/00:02:14|loop: module loaded
04/00:02:14|Device Tree Probing 'ethernet'
04/00:02:14|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
04/00:02:14|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
04/00:02:14|XLlTemac: DCR address: 0x80
04/00:02:14|XLlTemac: buffer descriptor size: 32768 (0x8000)
04/00:02:14|XLlTemac: Allocating DMA descriptors with kmalloc
04/00:02:14|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
04/00:02:14|XTemac: PHY detected at address 7.
04/00:02:14|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
04/00:02:14|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14|Using buffer write method
04/00:02:15|cfi_cmdset_0001: Erase suspend on write enabled
04/00:02:15|cmdlinepart partition parsing not available
04/00:02:15|RedBoot partition parsing not available
04/00:02:15|Creating 5 MTD partitions on "fc000000.flash":
04/00:02:15|0x000000000000-0x000000500000 : "golden_firmware"
04/00:02:15|0x000000500000-0x000000800000 : "golden_kernel"
04/00:02:15|0x000000800000-0x000000d00000 : "user_firmware"
04/00:02:15|0x000000d00000-0x000000fe0000 : "user_kernel"
04/00:02:15|0x000000fe0000-0x000001000000 : "env_variables"
04/00:02:15|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
04/00:02:15|SPI: XIlinx spi: bus number now 32766
04/00:02:15|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
04/00:02:15|SPI: XIlinx spi: bus number now 32765
04/00:02:15|mice: PS/2 mouse device common for all mice
04/00:02:15|Device Tree Probing 'i2c'
04/00:02:15| #0 at 0x81600000 mapped to 0xD1030000, irq=22
04/00:02:15|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
04/00:02:15|TCP cubic registered
04/00:02:15|NET: Registered protocol family 17
04/00:02:15|RPC: Registered udp transport module.
04/00:02:15|RPC: Registered tcp transport module.
04/00:02:16|eth0: XLlTemac: Options: 0x3fa
04/00:02:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
04/00:02:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
04/00:02:16|eth0: XLlTemac: speed set to 1000Mb/s
04/00:02:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
04/00:02:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
04/00:02:18|Sending DHCP requests ., OK
04/00:02:19|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.15
04/00:02:20|IP-Config: Complete:
04/00:02:20|     device=eth0, addr=192.168.11.15, mask=255.255.255.0, gw=255.255.255.255,
04/00:02:20|     host=aida15, domain=dl.ac.uk, nis-domain=nuclear.physics,
04/00:02:20|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15
04/00:02:20|Looking up port of RPC 100003/2 on 192.168.11.99
04/00:02:20|Looking up port of RPC 100005/1 on 192.168.11.99
04/00:02:20|VFS: Mounted root (nfs filesystem) on device 0:12.
04/00:02:20|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
04/00:02:20|Starting sysinit...
04/00:02:21|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
04/00:02:22|		Press 'I' to enter interactive startup.
04/00:02:22|Setting clock  (utc): Thu Jun  3 23:02:22 BST 2021 [  OK  ]
04/00:02:22|Building the cache [  OK  ]
04/00:02:22|Setting hostname aida15:  [  OK  ]
04/00:02:23|Mounting local filesystems:  [  OK  ]
04/00:02:25|Enabling /etc/fstab swaps:  [  OK  ]
04/00:02:27|Finishing sysinit...
INIT: Entering runlevel: 3
04/00:02:30|Entering non-interactive startup
04/00:02:30|FATAL: Module ipv6 not found.
04/00:02:31|Bringing up loopback interface:  [  OK  ]
04/00:02:33|FATAL: Module ipv6 not found.
04/00:02:34|Starting system logger: [  OK  ]
04/00:02:34|Starting kernel logger: [  OK  ]
04/00:02:35|Starting rpcbind: [  OK  ]
04/00:02:36|Mounting NFS filesystems:  [  OK  ]
04/00:02:37|Mounting other filesystems:  [  OK  ]
04/00:02:37|Starting xinetd: [  OK  ]
04/00:02:38|Starting midas:  Starting MIDAS Data Acquisition for aida15
04/00:02:38|xaida: device parameters: base=0x81000000 size=0x200000
04/00:02:44|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
04/00:02:44|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
04/00:02:44|xaida: driver assigned major number 254
04/00:02:44|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
04/00:02:50|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
04/00:02:50|AIDAMEM: aidamem: driver assigned major number 253
04/00:02:50|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
04/00:02:56|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
04/00:02:57|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
04/00:02:57|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
04/00:02:57|Computer Name = aida15; Temp Directory = /tmp/tcl361
04/00:03:01|package limit is not available: can't find package limit
04/00:03:04|Running with default file descriptor limit
04/00:03:04|package setuid is not available: can't find package setuid
04/00:03:05|Running as user 0 group 0
04/00:03:05|
04/00:03:06|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
04/00:03:06|
04/00:03:06|Built without pthreads
04/00:03:06|
04/00:03:06|Creating NetVars
04/00:03:06|Output buffer length = 65xaida: open:
04/00:03:06|504; format option = 4AIDAMEM: aidamem_open:
04/00:03:06|; transfer option = 3
04/00:03:06|EB transfer option = 3
04/00:03:06|NetVars created and initialised
04/00:03:06|Statistics thread starting
04/00:03:06|Statistics thread created
04/00:03:06|Stat/Rate creation thread starting
04/00:03:06|Data Acquisition task has PID 375
04/00:03:06|Stat/Rate creation thread created
04/00:03:06|Hit/Rate creation thread starting
04/00:03:06|Hit/Rate creation thread created
04/00:03:07|AIDA Heartbeat thread starting
04/00:03:07|Heartbeat thread created
04/00:03:07|Installing signal handlers
04/00:03:07|Done
04/00:03:07|ModuleNum = 0
04/00:03:07|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706 
04/00:03:07|Spectra table initialised
04/00:03:07|AIDA Data Acquisition now all ready to start
04/00:03:07|SIGBUS, SIGSEGV and SIGPIPE traps setup
04/00:03:07|/debug user "debug" password " bjkhcfyuqis"
04/00:03:14|httpd started on port 8015
04/00:03:14|
04/00:03:14|Cannot use /MIDAS/config/TclHttpd/aida15@8015/startup.tcl
04/00:03:14|Custom startup from /MIDAS/config/TclHttpd/aida15/startup.tcl
04/00:03:14|XAIDA Access package 1.0
04/00:03:16|/XAIDAAccessServer
04/00:03:16|XAD9252 Access package 1.0
04/00:03:16|/XAD9252AccessServer
04/00:03:16|/DataBaseAccessServer
04/00:03:16|[  OK  ]
04/00:03:16|/NetVarService
04/00:03:17|/SigTaskService
04/00:03:17|Loaded MemSasAccess
04/00:03:17|/SpectrumService
04/00:03:17|loading tcl/AIDARunControl.tcl for namespace ::
04/00:03:17|
04/00:03:17|DENX ELDK version 4.2 build 2008-04-01
04/00:03:17|Linux 2.6.31 on a ppc
04/00:03:17|
04/00:03:17|aida15 login: /DataAcquisitionControlServer
04/00:03:17|DefineMessage unknown
04/00:03:17|Run Control Server Implementation for AIDA
04/00:03:17|RunControlServer loaded
04/00:03:17|loading Html/RunControl/implementation.tcl
04/00:03:18|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
04/00:03:18|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
04/00:03:18|Created UI registers
04/00:03:18|RunControl loaded
04/00:03:19|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
04/00:03:19|AIDA RunControl loaded
04/00:03:19|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
04/00:03:19|do_GetState returned z=0 and 8
04/09:31:05|do_GetState returned z=0 and 8
04/09:31:17|do_GetState returned z=0 and 8
04/09:31:30|do_GetState returned z=0 and 8
04/09:31:43|do_GetState returned z=0 and 8
04/09:31:55|do_GetState returned z=0 and 8
04/09:32:07|do_GetState returned z=0 and 8
04/09:32:19|do_GetState returned z=0 and 8
04/09:32:32|do_GetState returned z=0 and 8
04/09:32:45|do_GetState returned z=0 and 8
04/09:32:57|do_GetState returned z=0 and 8
04/09:33:10|do_GetState returned z=0 and 8
04/09:33:22|do_GetState returned z=0 and 8
04/09:33:35|do_GetState returned z=0 and 8
04/09:33:47|do_GetState returned z=0 and 8
04/09:34:00|do_GetState returned z=0 and 8
04/09:34:13|do_GetState returned z=0 and 8
04/09:34:25|do_GetState returned z=0 and 8
04/09:34:38|do_GetState returned z=0 and 8
04/09:34:50|do_GetState returned z=0 and 8
04/09:35:03|do_GetState returned z=0 and 8
04/09:35:16|do_GetState returned z=0 and 8
04/09:35:28|do_GetState returned z=0 and 8
04/09:35:41|do_GetState returned z=0 and 8
04/09:35:53|do_GetState returned z=0 and 8
04/09:36:06|do_GetState returned z=0 and 8
04/09:36:18|do_GetState returned z=0 and 8
04/09:36:31|do_GetState returned z=0 and 8
04/09:36:43|do_GetState returned z=0 and 8
04/09:36:56|do_GetState returned z=0 and 8
04/09:37:09|do_GetState returned z=0 and 8
04/09:37:21|do_GetState returned z=0 and 8
04/09:37:34|do_GetState returned z=0 and 8
04/09:37:46|do_GetState returned z=0 and 8
04/09:37:59|do_GetState returned z=0 and 8
04/09:38:12|do_GetState returned z=0 and 8
04/09:38:24|do_GetState returned z=0 and 8
04/09:38:37|do_GetState returned z=0 and 8
04/09:38:49|do_GetState returned z=0 and 8
04/09:39:02|do_GetState returned z=0 and 8
04/09:39:15|do_GetState returned z=0 and 8
04/09:39:27|do_GetState returned z=0 and 8
04/09:39:40|do_GetState returned z=0 and 8
04/09:39:52|do_GetState returned z=0 and 8
04/09:40:05|do_GetState returned z=0 and 8
04/09:40:18|do_GetState returned z=0 and 8
04/09:40:30|do_GetState returned z=0 and 8
04/09:40:43|do_GetState returned z=0 and 8
04/09:40:55|do_GetState returned z=0 and 8
04/09:41:08|do_GetState returned z=0 and 8
04/09:41:21|do_GetState returned z=0 and 8
04/09:41:33|do_GetState returned z=0 and 8
04/09:41:46|do_GetState returned z=0 and 8
04/09:41:58|do_GetState returned z=0 and 8
04/09:42:11|do_GetState returned z=0 and 8
04/09:42:23|do_GetState returned z=0 and 8
04/09:42:36|do_GetState returned z=0 and 8
04/09:42:48|do_GetState returned z=0 and 8
04/09:43:01|do_GetState returned z=0 and 8
04/09:43:14|do_GetState returned z=0 and 8
04/09:43:26|do_GetState returned z=0 and 8
04/09:43:39|do_GetState returned z=0 and 8
04/09:43:51|do_GetState returned z=0 and 8
04/09:44:04|do_GetState returned z=0 and 8
04/09:44:17|do_GetState returned z=0 and 8
04/09:44:29|do_GetState returned z=0 and 8
04/09:44:42|do_GetState returned z=0 and 8
04/09:44:54|do_GetState returned z=0 and 8
04/09:45:07|do_GetState returned z=0 and 8
04/09:45:19|do_GetState returned z=0 and 8
04/09:45:32|do_GetState returned z=0 and 8
04/09:45:44|do_GetState returned z=0 and 8
04/09:45:57|do_GetState returned z=0 and 8
04/09:46:09|do_GetState returned z=0 and 8
04/09:46:22|do_GetState returned z=0 and 8
04/09:46:34|do_GetState returned z=0 and 8
04/09:46:47|do_GetState returned z=0 and 8
04/09:47:00|do_GetState returned z=0 and 8
04/09:47:12|do_GetState returned z=0 and 8
04/09:47:25|do_GetState returned z=0 and 8
04/09:47:37|do_GetState returned z=0 and 8
04/09:47:50|do_GetState returned z=0 and 8
04/09:48:03|do_GetState returned z=0 and 8
04/09:48:15|do_GetState returned z=0 and 8
04/09:48:28|do_GetState returned z=0 and 8
04/09:48:40|do_GetState returned z=0 and 8
04/09:48:53|do_GetState returned z=0 and 8
04/09:49:05|do_GetState returned z=0 and 8
04/09:49:18|do_GetState returned z=0 and 8
04/09:49:31|do_GetState returned z=0 and 8
04/09:49:43|do_GetState returned z=0 and 8
04/09:49:56|do_GetState returned z=0 and 8
04/09:50:08|do_GetState returned z=0 and 8
04/09:50:21|do_GetState returned z=0 and 8
04/09:50:33|do_GetState returned z=0 and 8
04/09:50:46|do_GetState returned z=0 and 8
04/09:50:58|do_GetState returned z=0 and 8
04/09:51:11|do_GetState returned z=0 and 8
04/09:51:23|do_GetState returned z=0 and 8
04/09:51:36|do_GetState returned z=0 and 8
04/09:51:49|do_GetState returned z=0 and 8
04/09:52:01|do_GetState returned z=0 and 8
04/09:52:14|do_GetState returned z=0 and 8
04/09:52:26|do_GetState returned z=0 and 8
04/09:52:39|do_GetState returned z=0 and 8
04/09:52:51|do_GetState returned z=0 and 8
04/09:53:04|do_GetState returned z=0 and 8
04/09:53:16|do_GetState returned z=0 and 8
04/09:53:29|do_GetState returned z=0 and 8
04/09:53:42|do_GetState returned z=0 and 8
04/09:53:54|do_GetState returned z=0 and 8
04/09:54:07|do_GetState returned z=0 and 8
04/09:54:19|do_GetState returned z=0 and 8
04/09:54:32|do_GetState returned z=0 and 8
04/09:54:44|do_GetState returned z=0 and 8
04/09:54:57|do_GetState returned z=0 and 8
04/09:55:09|do_GetState returned z=0 and 8
04/09:55:22|do_GetState returned z=0 and 8
04/09:55:34|do_GetState returned z=0 and 8
04/09:55:47|do_GetState returned z=0 and 8
04/09:56:00|do_GetState returned z=0 and 8
04/09:56:12|do_GetState returned z=0 and 8
04/09:56:25|do_GetState returned z=0 and 8
04/09:56:37|do_GetState returned z=0 and 8
04/09:56:50|do_GetState returned z=0 and 8
04/09:57:02|do_GetState returned z=0 and 8
04/09:57:15|do_GetState returned z=0 and 8
04/09:57:27|do_GetState returned z=0 and 8
04/09:57:40|do_GetState returned z=0 and 8
04/09:57:53|do_GetState returned z=0 and 8
04/09:58:05|do_GetState returned z=0 and 8
04/09:58:18|do_GetState returned z=0 and 8
04/09:58:30|do_GetState returned z=0 and 8
04/09:58:43|do_GetState returned z=0 and 8
04/09:58:56|do_GetState returned z=0 and 8
04/09:59:08|do_GetState returned z=0 and 8
04/09:59:21|do_GetState returned z=0 and 8
04/09:59:33|do_GetState returned z=0 and 8
04/09:59:46|do_GetState returned z=0 and 8
04/09:59:58|do_GetState returned z=0 and 8
04/10:00:11|do_GetState returned z=0 and 8
04/10:00:23|do_GetState returned z=0 and 8
04/10:00:36|do_GetState returned z=0 and 8
04/10:00:48|do_GetState returned z=0 and 8
04/10:01:01|do_GetState returned z=0 and 8
04/10:01:13|do_GetState returned z=0 and 8
04/10:01:26|do_GetState returned z=0 and 8
04/10:01:38|do_GetState returned z=0 and 8
04/10:01:51|do_GetState returned z=0 and 8
04/10:02:04|do_GetState returned z=0 and 8
04/10:02:16|do_GetState returned z=0 and 8
04/10:02:29|do_GetState returned z=0 and 8
04/10:02:41|do_GetState returned z=0 and 8
04/10:02:54|do_GetState returned z=0 and 8
04/10:03:06|do_GetState returned z=0 and 8
04/10:03:19|do_GetState returned z=0 and 8
04/10:03:31|do_GetState returned z=0 and 8
04/10:03:44|do_GetState returned z=0 and 8
04/10:03:57|do_GetState returned z=0 and 8
04/10:04:09|do_GetState returned z=0 and 8
04/10:04:22|do_GetState returned z=0 and 8
04/10:04:34|do_GetState returned z=0 and 8
04/10:04:47|do_GetState returned z=0 and 8
04/10:04:59|do_GetState returned z=0 and 8
04/10:05:12|do_GetState returned z=0 and 8
04/10:05:25|do_GetState returned z=0 and 8
04/10:05:37|do_GetState returned z=0 and 8
04/10:05:50|do_GetState returned z=0 and 8
04/10:06:02|do_GetState returned z=0 and 8
04/10:06:15|do_GetState returned z=0 and 8
04/10:06:27|do_GetState returned z=0 and 8
04/10:06:40|do_GetState returned z=0 and 8
04/10:06:52|do_GetState returned z=0 and 8
04/10:07:05|do_GetState returned z=0 and 8
04/10:07:18|do_GetState returned z=0 and 8
04/10:07:30|do_GetState returned z=0 and 8

09.20 DAQ found stopped
      files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight
      aida15 rebooted at 00.02

      regained control of DAQ and issued DAQ STOP by restarting NewMerger *twice*

11.00 DAQ reset and setup
      All system wide checks OK *except*

		 Base 		Current 	Difference
aida05 fault 	 0x4d3c : 	 0x4d3c : 	 0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      4      2      8      5      4      1      2      2      4      7   : 41352
aida02 :      0      3      3      6      6      4      1      2      2      4      7   : 41352
aida03 :      2      4      2     12      1      3      2      2      2      4      7   : 41352
aida04 :      2      2      1      7      4      3      2      2      2      4      7   : 41352
aida05 :      2      3      1      5      3      2      3      2      2      4      7   : 41360
aida06 :      2      4      2      6      4      3      2      2      2      4      7   : 41352
aida07 :      1      5      2      9      4      2      4      3      1      4      7   : 41324
aida08 :      0      3      1      7      4      3      4      3      1      4      7   : 41352
aida09 :      3      3      0      8      2      2      3      2      2      4      7   : 41380
aida10 :      1      2      2     11      3      2      3      3      3      3      7   : 41044
aida11 :      2      3      1      6      6      4      2      3      3      3      7   : 41072
aida12 :      4      2      1      8      3      3      3      3      1      4      7   : 41072
aida13 :      2      4      1      8      4      4      2      3      3      3      7   : 41016
aida14 :      1      4      1      9      4      4      2      3      3      3      7   : 41044
aida15 :      0      5      1      2      2      1      2      4      3      4      7   : 42872
aida16 :      1      3      2      9      4      4      4      4      2      3      7   : 41052

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

11.03 DAQ resumes file S496/R32
      alpha background

      slow comparator 0x64
      BNC PB-5 pulser OFF

11.09 1.8.W spectra - 20us FSR - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

13.07 analysis of file S496/R32_16 - attachment 8
       max deadtime 0.18% aida04

11.19 Hot HEC channels multiple FEE64s
      ASIC check fixes

      System wide checks OK * except*

		 Base 		Current 	Difference
aida01 fault 	 0x8940 : 	 0x8941 : 	 1  
aida02 fault 	 0x9b32 : 	 0x9b33 : 	 1  
aida03 fault 	 0xf266 : 	 0xf267 : 	 1  
aida04 fault 	 0x579b : 	 0x579c : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      4      2      2      2      2      2      3      2      4      7   : 41516
aida02 :     10      7      2      0      1      2      2      3      2      4      7   : 41408
aida03 :     10      8      2      2      2      4      1      4      2      3      7   : 40008
aida04 :      5      4      4      2      2      3      1      4      2      3      7   : 39860
aida05 :     12      3      2      2      1      2      2      3      2      4      7   : 41448
aida06 :     15      9      2      2      3      2      1      3      2      4      7   : 41380
aida07 :     12      8      1      2      2      3      3      2      2      4      7   : 41408
aida08 :      9      6      3      1      2      3      3      2      2      4      7   : 41380
aida09 :     15      6      1      1      2      2      2      3      2      4      7   : 41500
aida10 :     15      6      3      2      1      3      3      2      1      4      7   : 40348
aida11 :      8      2      1      1      1      3      1      3      2      4      7   : 41248
aida12 :     18      3      2      1      3      3      2      2      2      4      7   : 41184
aida13 :      7      4      1      4      3      4      3      3      1      4      7   : 41100
aida14 :     18      6      1      4      3      4      2      2      1      4      7   : 40392
aida15 :      7      7      1      3      1      2      1      3      2      4      7   : 41220
aida16 :      4      4      2      3      3      3      2      4      1      4      7   : 41200

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Jun 02 16:27:30 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36

13.09 analysis of file S496/R31_16 - attachment 8
       max deadtime 0.31% aida04

On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen.
We were also unable to connected to MIDAS via ssh port forwarding.

Since the restart we have been unable to write to file.

The tapeserver will allocate/mount/open a file but nothing will be written to it.
The merger will be receiving data but never shows as having any "Current links with data".

/MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23  2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119

The current disk structure is:
[npg@aidas-gsi npg]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
sdb      8:16   0   7.3T  0 disk 
├─sdb1   8:17   0   200M  0 part 
├─sdb2   8:18   0   500M  0 part 
└─sdb3   8:19   0   7.3T  0 part 
  ├─vg_aidas2-lv_root (dm-2)
       253:2    0    50G  0 lvm  
  ├─vg_aidas2-lv_home (dm-3)
       253:3    0   7.2T  0 lvm  /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
  └─vg_aidas2-lv_swap (dm-4)
       253:4    0   7.8G  0 lvm  
sdc      8:32   0   7.3T  0 disk 
└─sdc1   8:33   0   7.3T  0 part /media/ThirdDrive
sdd      8:48   0   7.3T  0 disk 
└─sdd1   8:49   0   7.3T  0 part /media/SecondDrive
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0   200M  0 part /boot/efi
├─sda2   8:2    0   500M  0 part /boot
└─sda3   8:3    0 465.1G  0 part 
  ├─vg_aidasgsi-lv_root (dm-0)
       253:0    0    50G  0 lvm  /
  ├─vg_aidasgsi-lv_swap (dm-1)
       253:1    0   7.8G  0 lvm  [SWAP]
  └─vg_aidasgsi-lv_home (dm-5)
       253:5    0 407.3G  0 lvm  /home


Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3

TapeServer config file - attachment 4
Merger options - attachment 5

Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png

09.55 DAQ continues - file S496/R18_588
      alpha background

      increased data rate due to hot HEC channels since c. 06.00 this am - ASIC check fixes problem

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b7 : 	 7  
aida02 fault 	 0xfe72 : 	 0xfe79 : 	 7  
aida03 fault 	 0x97f4 : 	 0x97fb : 	 7  
aida04 fault 	 0x950 : 	 0x957 : 	 7  
aida05 fault 	 0x8e93 : 	 0x8e98 : 	 5  
aida06 fault 	 0x545a : 	 0x545f : 	 5  
aida07 fault 	 0xc04d : 	 0xc052 : 	 5  
aida08 fault 	 0xa0f7 : 	 0xa0fc : 	 5  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      6      9      5      1      3      2      2      2      2      4      6   : 37008
aida02 :      3      5      5      2      2      4      3      3      2      4      6   : 37956
aida03 :     22     12      7      5      3      3      2      3      3      3      6   : 36872
aida04 :     40     13      5      5      3      4      2      3      3      3      6   : 37048
aida05 :     15      5      5      4      2      2      2      2      2      4      6   : 37044
aida06 :     11      6      2      1      2      3      3      4      2      4      6   : 38300
aida07 :     17      5      4      3      1      3      3      2      2      4      6   : 37324
aida08 :      1      8      1      2      2      4      4      3      2      4      6   : 38164
aida09 :     10      8      6      1      2      4      3      3      1      4      6   : 36968
aida10 :     14      7      5      2      1      3      3      3      1      4      6   : 36800
aida11 :      7      9      3      1      2      3      3      3      1      4      6   : 36788
aida12 :      3      5      2      3      1      4      2      3      2      4      6   : 37620
aida13 :     12      4      4      2      1      3      2      2      2      4      6   : 37008
aida14 :     11     10      5      1      1      3      2      2      2      4      6   : 37036
aida15 :     11      5      1      2      1      3      3      4      2      4      6   : 38244
aida16 :     10      4      8      1      2      3      2      2      2      4      6   : 37096

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.24 All histograms, statistics and merger statistics zero'd

16.16 analysis of file S496/R18_541 - attachment 9
       max deadtime 0.02% aida03

18.40 DAQ continues - file S496/R18_589
      alpha background

      increased data rate due to hot HEC channels since c. 18.00 this am - ASIC check fixes problem

14.10 DAQ continues - file S496/R18_539
      alpha background

      increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b7 : 	 7  
aida02 fault 	 0xfe72 : 	 0xfe79 : 	 7  
aida03 fault 	 0x97f4 : 	 0x97fb : 	 7  
aida04 fault 	 0x950 : 	 0x957 : 	 7  
aida05 fault 	 0x8e93 : 	 0x8e97 : 	 4  
aida06 fault 	 0x545a : 	 0x545e : 	 4  
aida07 fault 	 0xc04d : 	 0xc051 : 	 4  
aida08 fault 	 0xa0f7 : 	 0xa0fb : 	 4  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      7      2      2      2      2      2      2      4      6   : 37036
aida02 :     14      3      5      2      2      4      3      3      2      4      6   : 37984
aida03 :     15     11      8      6      3      3      2      3      3      3      6   : 36884
aida04 :     43     13      5      5      3      4      2      3      3      3      6   : 37060
aida05 :     19      3      5      4      2      2      2      2      2      4      6   : 37044
aida06 :     17      4      1      3      2      3      3      4      2      4      6   : 38356
aida07 :     16      6      2      1      0      2      2      3      2      4      6   : 37296
aida08 :     21      8      1      1      3      4      4      3      2      4      6   : 38276
aida09 :     13     10      2      1      2      3      2      2      2      4      6   : 37060
aida10 :     13      9      4      1      1      3      3      3      1      4      6   : 36764
aida11 :     13      9      2      1      2      3      3      3      1      4      6   : 36796
aida12 :     11      6      4      2      2      4      2      3      2      4      6   : 37724
aida13 :      7      6      6      2      1      3      2      2      2      4      6   : 37036
aida14 :      9      7      7      1      1      3      2      2      2      4      6   : 37036
aida15 :     14      5      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.24 All histograms, statistics and merger statistics zero'd

16.16 analysis of file S496/R18_541 - attachment 9
       max deadtime 0.12% aida04

18.40 DAQ continues - file S496/R18_551
      alpha background

      increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem

11.30 DAQ continues - file S496/R18_472
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      7      1      3      2      2      2      2      4      6   : 36980
aida02 :      3      1      5      1      2      4      3      3      2      4      6   : 37892
aida03 :      8      8      6      6      3      3      2      3      3      3      6   : 36800
aida04 :     37     12      5      6      3      4      2      3      3      3      6   : 37060
aida05 :     11      5      6      4      2      2      2      2      2      4      6   : 37044
aida06 :      4      3      3      3      2      2      2      3      3      4      6   : 38456
aida07 :      1     12      3      2      0      3      3      2      2      4      6   : 37204
aida08 :     13      8      1      2      3      4      3      3      2      4      6   : 38020
aida09 :      7      7      5      1      1      3      2      2      2      4      6   : 36996
aida10 :     12      7      9      2      1      3      3      3      1      4      6   : 36856
aida11 :      2      9      3      1      2      3      3      3      1      4      6   : 36768
aida12 :     11      4      3      3      2      4      2      3      2      4      6   : 37724
aida13 :     10      8      6      2      2      3      2      2      2      4      6   : 37128
aida14 :     16      7      7      1      0      4      2      2      2      4      6   : 37128
aida15 :     12      6      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_471 - attachment 9
       max deadtime 0.18% aida04

13.29 DAQ continues - file S496/R18_443
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      7      7      1      3      2      2      2      2      4      6   : 37008
aida02 :      3      1      5      1      3      4      3      3      2      4      6   : 37956
aida03 :      2      6      5      6      2      4      2      3      3      3      6   : 36808
aida04 :     24     12      5      5      4      3      2      3      3      3      6   : 36912
aida05 :      8      1      9      3      2      2      2      2      2      4      6   : 37016
aida06 :      3      6      2      2      2      2      2      3      3      4      6   : 38428
aida07 :      6     17      1      2      0      3      3      2      2      4      6   : 37232
aida08 :      4      2      1      2      3      4      3      3      2      4      6   : 37936
aida09 :     15      7      5      2      3      4      3      3      1      4      6   : 37060
aida10 :      7      6      9      2      0      3      3      3      1      4      6   : 36764
aida11 :     19     10      2      0      2      3      3      3      1      4      6   : 36796
aida12 :     11      6      4      2      2      4      2      3      2      4      6   : 37724
aida13 :     17      9      2      0      0      4      2      2      2      4      6   : 37036
aida14 :     15      4      7      1      1      3      2      2      2      4      6   : 37036
aida15 :      9      6      1      2      1      3      3      4      2      4      6   : 38244
aida16 :     11     11      8      2      0      3      2      2      2      4      6   : 37060

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_442 - attachment 9
       max deadtime 0.28% aida04

14.22 DAQ continues - file S496/R18_369
      alpha background

      All system wide checks OK *except*

	
		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     12      4      6      1      2      3      2      2      2      4      6   : 37072
aida02 :     12      2      4      1      3      4      3      3      2      4      6   : 37984
aida03 :     12      6      7      6      2      4      3      2      3      3      6   : 36624
aida04 :     36     11      4      6      4      3      2      3      3      3      6   : 36968
aida05 :      2      4      6      3      2      2      2      2      2      4      6   : 36968
aida06 :      8      5      3      2      2      2      2      3      3      4      6   : 38456
aida07 :     10      4      6      4      3      3      3      3      2      4      6   : 37992
aida08 :     21      8      3      2      3      3      3      3      2      4      6   : 37956
aida09 :     17     10      3      2      3      4      3      3      1      4      6   : 37060
aida10 :      3      6     10      2      1      3      3      3      1      4      6   : 36828
aida11 :     12      6      4      2      2      3      3      3      1      4      6   : 36832
aida12 :      2      9      5      1      3      3      2      3      2      4      6   : 37632
aida13 :      4      7      2      1      3      3      3      3      2      4      6   : 37832
aida14 :     15      6      6      1      0      4      2      2      2      4      6   : 37100
aida15 :      5      4      3      2      1      3      3      4      2      4      6   : 38244
aida16 :     17      6      6      2      1      3      2      2      2      4      6   : 37076

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_340 - attachment 9
       max deadtime 0.31% aida04

12.16 DAQ continues file S496/R18_324
      alpha background

      Tape server c. 2.5Mb/s
      ASIC check
      Tape server c. 300kb/s - presumably hot HEC channels

12.20 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      6      2      1      3      2      2      2      4      6   : 37016
aida02 :      3      6      5      3      3      4      3      2      1      3      7   : 38572
aida03 :     21     10      8      6      3      3      2      3      2      4      6   : 37924
aida04 :     29     15      4      5      3      4      2      3      3      3      6   : 37004
aida05 :     17      5      2      2      2      2      2      2      2      4      6   : 36940
aida06 :      1      6      0      1      2      2      2      3      3      4      6   : 38356
aida07 :     11      6      7      4      3      3      3      3      2      4      6   : 38028
aida08 :     13      8      3      2      2      3      3      2      1      3      7   : 38372
aida09 :      7      9      2      1      3      3      3      3      2      4      6   : 37860
aida10 :      9      8      4      2      0      3      3      3      1      4      6   : 36708
aida11 :     16      6      3      2      2      3      3      3      1      4      6   : 36832
aida12 :     21     10      7      4      2      4      2      2      1      3      7   : 38420
aida13 :      8      3      3      1      3      3      3      3      2      4      6   : 37832
aida14 :     14      3      6      1      0      4      2      2      2      4      6   : 37072
aida15 :      2      4      2      2      1      3      3      4      2      4      6   : 38216
aida16 :     16      5      5      3      0      3      2      2      2      4      6   : 37016


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
      Rate spectra - attachment 2
      ADC data item stats - attachment 3
      FEE64 temps OK - attachment 4
      DSSSD bias & leakage current OK - attachment 5
      Merger/Tape Server/Merger stats - attachment 6
       no merger errors reported since previous restart

17.30 analysis of file S496/R18_340
       max deadtime 0.36% aida04

07.30 Hot HEC channels multiple FEE64s
      ASIC check fixes

10.56 DAQ continues OK file S496/R18_75

      System wide checks OK * except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      2      7      2      1      4      3      2      1      4      7   : 40452
aida02 :     13      3      3      1      3      4      2      2      2      4      7   : 41308
aida03 :     17      6      2      2      1      4      2      3      2      3      7   : 39700
aida04 :      8      6      1      0      1      3      1      4      2      3      7   : 39712
aida05 :     10      3      5      3      4      3      3      2      1      4      7   : 40560
aida06 :      4      4      2      0      2      3      2      3      3      4      7   : 42576
aida07 :      5      3      2      3      1      3      1      3      2      4      7   : 41324
aida08 :     11      5      1      1      2      3      3      2      2      4      7   : 41348
aida09 :      7      4      3      0      2      3      1      3      2      4      7   : 41324
aida10 :     11      4      2      2      2      3      2      4      1      4      7   : 41132
aida11 :     18      5      0      2      2      4      2      2      1      4      7   : 40240
aida12 :      2      3      0      1      2      3      2      2      2      4      7   : 41024
aida13 :      4      5      1      2      2      3      2      4      1      4      7   : 41096
aida14 :      9      3      1      2      1      4      2      2      2      4      7   : 41164
aida15 :      2      3      1      3      2      2      3      3      2      4      7   : 41744
aida16 :      5      5      2      0      3      3      2      4      1      4      7   : 41116

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
      1.8.H spectra - attachments 6-8
      1.8.L spectra - attachments 9 & 10
      Stat spectra - attachments 11 & 12
      1.8.W spectra - 20us FSR - attachments 13 & 14
      ADC data items - attachment 15
      FEE64 temperatures OK - attachment 16
      DSSSD bias & leakage currents OK - attachment 17
      Merger/Tape Server/Merger statistics - attachment 18
       no merger errors reported since previous restart

      analysis of file R18_74
       max dead time 0.05%

10.17 ASIC check
      All histograms, statistics, merger statistics zero'd

00.40 Unable to connect to FEE64s 1-7
      DAQ power cycled OK

      ASIC settings 2021Apr29-13-16-00
      slow comparator 0x64 *all* FEE64s

      BNC PB-5 OFF

      file S496/R16

      alpha background

07.17 Some hot HEC channels
      ASIC check fixes issue
      System wide checks OK *except*

		 Base 		Current 	Difference
aida06 fault 	 0x5457 : 	 0x5457 : 	 0  
aida06 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1cd : 	 461  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      3      1      4      3      2      1      4      7   : 40552
aida02 :      3      2      2      1      2      3      3      2      2      4      7   : 41308
aida03 :     14      5      0      1      1      3      2      4      2      3      7   : 40000
aida04 :     14      4      5      1      3      3      1      4      2      3      7   : 39944
aida05 :     12      4      1      3      2      3      3      2      2      4      7   : 41408
aida06 :      8      3      2      1      2      4      2      3      3      4      7   : 42744
aida07 :     10      4      3      2      1      2      2      3      2      4      7   : 41464
aida08 :      7      5      2      0      3      3      3      2      2      4      7   : 41380
aida09 :     10      5      1      1      1      2      2      3      2      4      7   : 41408
aida10 :     11      6      3      2      1      3      2      2      2      4      7   : 41100
aida11 :     12      8      3      2      2      2      3      2      1      4      7   : 40288
aida12 :      9      9      3      1      2      3      3      2      1      4      7   : 40380
aida13 :     10      4      0      1      1      2      1      3      2      4      7   : 41128
aida14 :      9      4      2      1      1      2      1      3      2      4      7   : 41156
aida15 :      5      4      0      3      3      3      3      3      2      4      7   : 41940
aida16 :      9      6      3      2      0      2      1      3      2      4      7   : 41156

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 00:29:48 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      FEE64 temperatures OK - attachment 1
      ADC data item stats - attachment 2
      Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3
      DSSSD bias & leakage currents OK - attachment 4

11.18 Analysis of file S496/R16_33 - attachment 5

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  685M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.0T  867G  88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

867/2 x 3279s = 16.5 days to filesystem full - OK

12.11 aida15 zero data/poll stats - re-loaded DAQ program OK
      aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK

      DAQ continues file S496/R18

14.37 DAQ continues file S496/R15_14

      Grafana - DSSSD bias & leakage current - attachment 1
      1.8.L spectra - attachments 2 & 3
      Stat spectra - attachments 4 & 5
      FEE64 temperatures OK - attachment 6
      stats - ADC data items, pause items OK - attachments 7 & 8
      DSSSD bias & leakage current OK - attachment 9
      Merger/Tape Server/Merger stats - attachment 10
      analysis of file S496/R14_1920 - attachment 11
       all FEE64 deadtime 0% as expected

17.30 aida01, 3, 5, 9 & 11 have hot HEC channels
       ASIC check fixes
      All histogram, statistics & merger statistics zero'd

      DAQ continues
      Note gamma-ray source tests have started with FATIMA and DEGAS

20.20 Analysis file R15_19 - attachment 12

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  665M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.3T  530G  93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

        estimate 90/2 x 3168s = 40h to filesystem full - OK

20.22 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xf82c : 	 0xf82f : 	 3  
aida02 fault 	 0xdda6 : 	 0xdda9 : 	 3  
aida03 fault 	 0xb13d : 	 0xb140 : 	 3  
aida04 fault 	 0xf4cd : 	 0xf4d0 : 	 3  
aida05 fault 	 0x500 : 	 0x575 : 	 117  
aida06 fault 	 0xc6e7 : 	 0xc6e9 : 	 2  
aida07 fault 	 0x4920 : 	 0x4922 : 	 2  
aida08 fault 	 0xbd9a : 	 0xbd9c : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      8      4      6      6      1      4      3      3      4      3      6   : 38048
aida02 :     11      8     10      3      4      4      3      4      3      3      6   : 37740
aida03 :      2      8      3      3      2      2      1      4      2      4      6   : 37720
aida04 :     26     15     17      4      4      2      1      4      2      4      6   : 38256
aida05 :     15      6      4      1      1      2      1      3      3      4      6   : 38156
aida06 :      7      7     16      6      3      4      2      2      3      4      6   : 38612
aida07 :     11      5     12      3      3      3      2      3      1      3      7   : 38836
aida08 :     13      6      7      2      1      2      2      2      3      4      6   : 37972
aida09 :     23      3      8      5      0      5      3      3      4      3      6   : 38164
aida10 :     12      5      7      1      2      2      4      3      2      4      6   : 37992
aida11 :     19      2      7      1      2      2      1      2      3      4      6   : 37740
aida12 :     21      5      3      3      3      3      1      2      3      4      6   : 37964
aida13 :     18      7      7      1      2      4      2      3      2      4      6   : 37776
aida14 :      8      3     13      7      2      4      2      3      2      4      6   : 37992
aida15 :     11      5     14      1      3      4      2      3      2      4      6   : 37908
aida16 :     17      6      6      2      2      3      1      2      3      4      6   : 37908

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19.30 Lost activity monitors - attachments 13-15

20.54 1.8.W spectra 20us FSR attachments 16 & 23
      1.8.W spectra 200us FSR attachments 17 & 22
      1.8.W spectra 2ms FSR attachments 18 & 21
      1.8.W spectra 20ms FSR attachments 19 & 20

00.16 413Gb remaining /TapeData filesystem
      estimate 413/2 x 177s = 10.15h to filesystem full - OK

00.18 All histograms, statistics and merger statistics zero'd

00.57 All system wide checks OK *except*

	 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      9      3      4      1      5      2      2      3      3      6   : 36356
aida02 :      4      8      7      2      4      3      3      3      2      3      6   : 35968
aida03 :     12      9      6      2      5      3      2      3      3      3      6   : 36824
aida04 :     11      5      9      2      2      2      2      4      3      3      6   : 37028
aida05 :     24      9      2      1      2      3      2      3      4      3      6   : 37608
aida06 :     25      6      3      4      2      4      3      3      3      3      6   : 37060
aida07 :      3      3      3      1      3      3      2      2      2      2      7   : 37044
aida08 :     20      4      2      3      1      3      2      2      3      3      6   : 36016
aida09 :     15      8      2      5      0      3      3      2      3      3      6   : 36284
aida10 :     16      6      4      1      2      3      3      2      2      4      6   : 37328
aida11 :     20      2      2      1      1      3      2      2      3      3      6   : 35936
aida12 :      1      2      1      2      3      4      1      2      4      3      6   : 36900
aida13 :     25     10      4      1      0      3      2      3      3      3      6   : 36500
aida14 :     20      6      9      2      2      5      1      2      2      4      6   : 37200
aida15 :     31      9      5      2      3      3      1      3      3      3      6   : 36500
aida16 :     24      4      2      3      1      4      2      3      3      3      6   : 36672

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

01.12 Lost activity monitor - attachment 1
      1.8.H spectra - attachments 2 & 3
      1.8.L spectra - attachments 4-6
       aida01 pulser peak width 120 ch FWHM
      Rate specttra - attachments 7 & 8
01.06 Lost activity monitor - attachment 9
      ucesb - attachment 10
      Grafana - DSSSD bias & leakage currents past 7 days - attachment 11
      statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17
      FEE64 temperatures OK - attachment 18
      DSSSD bias & leakage currents OK - attachment 19
      Merger/Tape Server/Merger stats - attachment 20
       no merger warning/error messages since last restart
      
01.26 Analysis file S496/R14_1798 - attachment 21
       aida08 5.5% deadtime

01.17 Disk space remaining /TapeData filesystem

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  706M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.4T  395G  95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 395/2 x 177s = 9.7h to filesystem full - OK

01.30 despec shift reports no data reported by ucesb
      aida stats & tape server ok - no data being forwarded to MBS by data relay

01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811

04.35 System wide checks all OK *except*

Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x573 : 	 115  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  661M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.5T  337G  96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 337/2 x 177s = 8.3h to filesystem full - OK

FEE64 temperatures OK - attachment 22
ADC data item statistics OK - attachment 24
DSSSD bias & leakage current OK - attachment 26
analysis of file S496/R14_1920 - attachment 28
 aida04 deadtime 5.0%

07.10 Ssystem wide checks all OK *except*

	 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x573 : 	 115  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  662M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.5T  287G  96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 287/2 x 178s = 7.1h to filesystem full - OK

FEE64 temperatures OK - attachment 23
ADC data item statistics OK - attachment 25
DSSSD bias & leakage current OK - attachment 27
analysis of file S496/R14_1920 - attachment 29
 aida04 deadtime 4.4%

07.35 Found MBS data monitor kill'd
      Restarted MBS data relay and data monitor OK

08.00 beam off - end of S496

08.28 Tape Server -> no storage mode
      DAQ continues

09.58 All slow comparators -> 0x64
      BNC PB-5 OFF

      file S496/R15

      alpha background

      Tape server 200kb/s, merger 260k data items/s

16:00 CA takes over
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

16:14 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x570 : 	 112  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

16:17 temperatures ok - attachment 1
      statistics ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      analysis of R14_1625 - FEE7 deadtime at 4.5%

18:06 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x571 : 	 113  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:07 temperatures ok - attachment 4
      statistics ok - attachment 5
      detector bias / leakage currents ok - attachment 6
      analysis of R14_1662 - FEE7 deadtime at 4.3%

19:07 DAQ continues ok - writing to file R14_1682

20:07 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

20:10 temperatures ok - attachment 7
      statistics ok - attachment 8
      detector bias / leakage currents ok - attachment 9
      analysis of R14_1703 - max deadtime at 4.0% - attachment 10

20:35 rate spectra - attachment 11
      HEC spectra - attachment 12 & 13

20:36 all histograms zeroed

20:37 DAQ continues ok - writing to file R14_1712

20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.

      ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files

21:12 no beam - writing to file R14_1723

21:17 beam is back

22:07 system wide checks all ok - WR decoder fault same as entry at 20:07

22:08 temperatures ok - attachment 14
      statistics ok - attachment 15
      detector bias / leakage currents ok - attachment 16
      analysis of R14_1740 - max deadtime at 4.2% - attachment 17








   

08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8

System  wide checks at 7am CET

All seems fine, nothing significant to report

Slight increase of dead time noted in FEE 3, 5 and 7.

-----
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

-----

White Rabbit:

         Base         Current     Difference
aida05 fault      0x500 :      0x56e :      110  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     23      5      6      5      0      4      1      3      3      3      6   : 36484
aida02 :     10      3      3      2      3      2      2      2      3      3      6   : 35952
aida03 :      2      5      1      4      5      2      2      3      3      3      6   : 36608
aida04 :      1      3      4      0      4      3      2      3      3      3      6   : 36572
aida05 :     19      9      3      2      2      3      1      3      4      3      6   : 37380
aida06 :     13     11      1      5      3      4      4      2      3      3      6   : 36860
aida07 :     12      2      4      3      3      2      1      3      2      2      7   : 37280
aida08 :     25      5      7      1      1      3      3      3      3      3      6   : 36828
aida09 :     22      5      6      4      1      3      2      3      3      3      6   : 36640
aida10 :     23      8      6      3      2      3      2      2      2      4      6   : 37212
aida11 :     24     14      7      3      3      2      3      3      3      3      6   : 36960
aida12 :     16     10      5      3      1      2      1      2      4      3      6   : 36736
aida13 :     23      6      4      2      2      3      2      3      3      3      6   : 36620
aida14 :     22      7      8      4      3      4      1      2      2      4      6   : 37200
aida15 :     10     12      9      2      1      3      1      3      3      3      6   : 36376
aida16 :     18      5      6      2      0      3      2      4      3      3      6   : 37008

Wide check at 3am -

All fine.

Same output as at the beginning of the shift (see my previous entry) and attachments.

System Wide check at begining of the night shift:

-----

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

-----

White rabbit status:

         Base         Current     Difference
aida05 fault      0x500 :      0x56d :      109  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Memory info from FEE cards

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      7      7      6      1      5      2      2      3      3      6   : 36472
aida02 :     19      5      4      4      4      3      3      3      2      3      6   : 36020
aida03 :     22     10      2      2      5      2      2      3      3      3      6   : 36680
aida04 :     12      7      1      0      4      3      1      4      3      3      6   : 36856
aida05 :     16      7      5      3      1      3      2      3      3      3      6   : 36584
aida06 :     20      7      5      6      2      4      4      2      3      3      6   : 36888
aida07 :     13      7      5      2      3      2      1      3      2      2      7   : 37308
aida08 :     18      7      5      3      0      2      3      3      3      3      6   : 36656
aida09 :     21      6      4      4      1      4      2      3      3      3      6   : 36740
aida10 :     21     10      2      3      2      3      2      2      2      4      6   : 37156
aida11 :     23      2      4      4      3      2      2      3      2      4      6   : 37612
aida12 :     24      1      4      3      2      2      1      2      4      3      6   : 36744
aida13 :     26      9      4      2      1      4      3      2      3      3      6   : 36464
aida14 :     19      9      9      2      3      4      1      2      2      4      6   : 37156
aida15 :     10      7     10      0      1      3      1      3      3      3      6   : 36288
aida16 :     28      7      3      2      1      3      3      3      3      3      6   : 36824

16:00 p+n threshold 0xc (120keV)
      n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4
      n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4

      Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure.

17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector

18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now.
      The deadtimes in FEE6 and 8 have risen to around 6.25%.
      Looking at this mornings and yesterday evenings fill times this behavior isn't observed

18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago

18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2
      This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file)

18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8

18:15 System wide checks ok except WR as usual
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56d : 	 109  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics increased at 18:02 still working on - attachment 2
      FEE temp - attachment 3
      Bias and leakage current - attachment 4


20:51 System wide checks ok - WR same as before
      Statistics still high - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7 

22:52 Statistics still high - attachment 8
      Temp ok - attachment 9
      Bias and leakage current ok - attachment 10
      System wide checks ok - WR difference still 109

08:00 CA takes over

08:04 DAQ continues ok - writing to file R14_989

09:30 all system wide checks ok *except* WR decoder status;

      
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56a : 	 106  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

09:34 temperatures ok - attachment 1
      stats ok - attachment 2
      bias/leakage currents ok - attachment 3
      R14_1016 analysis - max deadtime ~4.3% - attachment 4

11:30 all system wide checks ok *except* WR decoder status;

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56b : 	 107  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

11:34 temperatures ok - attachment 5
      stats ok - attachment 6
      bias/leakage currents ok - attachment 7
      R14_1052 analysis - max deadtime ~3.6% - attachment 8

13:05 beam optimisation currently taking place
      DAQ continues ok and writing to file R14_1085

13:17 no beam

13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30.

13:31 temperatures ok - attachment 9
      stats ok - attachment 10
      bias/leakage currents ok - attachment 11
      R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12

13:37 beam is back - rate spectra - attachment 13
      DAQ ok - writing to file R14_1088

13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14

14:21 beam off - issue with UNILAC
      DESPEC also report issues with their DAQ
      AIDA continues to forward data to MBS ok - lower rates while beam off

14.30 1.8.H spectra - attachments 16-18
      1.8.L spectra - attachments 19-22
       aida01 pulser peak width 99 ch FWHM
      Rate spectra - attachment 23
      Grafana - DSSSD bias & leakage currents past 7 days - attachment 24
      DSSSD bias & leakage currents - attachment 25
      Merger/Tape Server/Merger stats - attachment 26



14.41 All histograms & stats zero'd

14.55 Lost Activity Monitors - attachments 27, 28 & 35
      Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler

15:00 beam is back

15:05 AIDA writing to file R14_1108
      ADC data items - attachment 36

15:27 system wide checks ok *except* WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56c : 	 108  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

15:30 temperatures ok - attachment 37
      stats ok - attachment 38
      bias/leakage currents ok - attachment 39
      R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40

02:00 System check

The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!

The rates on ucesb got back to work by around 01:12

Deat Time check ok R14_870, attachment 1

Statistics are ok attachment 2

Temperatures are ok attacment 3

Voltages are ok attachment 4

system wide check

All ok except

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbite

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      8      8      5      0      6      1      2      2      2      7   : 37296
aida02 :     19      8     11      2      3      4      2      2      3      3      6   : 36412
aida03 :     14      6      3      4      1      3      1      3      3      3      6   : 36312
aida04 :     25     12     12      4      3      4      1      4      3      3      6   : 37316
aida05 :      4      8      3      4      1      2      2      4      3      3      6   : 36928
aida06 :     42     12      8      7      2      3      3      4      3      3      6   : 37736
aida07 :     24      6      7      3      3      3      1      3      2      2      7   : 37536
aida08 :     10     15      7      2      1      4      2      3      3      3      6   : 36752
aida09 :     29     13      6      6      4      3      3      3      3      3      6   : 37244
aida10 :     18     12      7      2      2      3      3      2      2      4      6   : 37464
aida11 :     20      8      0      5      5      4      2      3      3      3      6   : 36976
aida12 :      7      2      4      3      2      3      2      2      4      3      6   : 37068
aida13 :      2      5      5      4      2      3      2      3      3      3      6   : 36608
aida14 :      9     13      8      2      2      2      3      3      2      4      6   : 37836
aida15 :     24     12     12      1      0      3      2      3      3      3      6   : 36640
aida16 :     12      9      9      2      1      3      2      4      3      3      6   : 37128

6:45 System check

Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9

System wide check are all ok except:

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit 1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20     11      3      5      0      6      1      2      3      2      7   : 38264
aida02 :     14      9      9      2      3      3      3      2      3      3      6   : 36496
aida03 :     13      7      3      5      2      3      2      3      3      3      6   : 36668
aida04 :     30     15     13      4      3      4      1      3      4      3      6   : 37888
aida05 :     17      9      3      3      1      2      2      4      3      3      6   : 36956
aida06 :     40     16      5      6      2      3      2      3      2      4      6   : 37936
aida07 :      3      2      3      2      3      4      2      2      2      2      7   : 37196
aida08 :     17     10      6      2      1      3      3      3      3      3      6   : 36852
aida09 :     29     11      7      6      3      3      2      4      3      3      6   : 37436
aida10 :      1     10      5      1      1      3      3      2      2      4      6   : 37252
aida11 :     30      9      0      5      4      4      2      3      3      3      6   : 36960
aida12 :     19     10      5      3      2      3      2      2      4      3      6   : 37196
aida13 :     22      9      6      3      2      3      3      3      3      3      6   : 36960
aida14 :     19      6      9      2      2      3      3      2      2      4      6   : 37452
aida15 :      6      8     11      2      1      3      2      3      3      3      6   : 36616
aida16 :     19     14      5      3      1      3      1      4      3      3      6   : 36908

07:33 system check

Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13

System wide check

all ok except

ADC all failed

White rabbit

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      9      4      6      0      5      1      2      2      2      7   : 37152
aida02 :     19      6     13      1      3      3      4      2      3      3      6   : 36780
aida03 :     23      5      6      4      2      3      1      3      3      3      6   : 36452
aida04 :     24     11     14      3      3      4      2      3      3      3      6   : 37048
aida05 :     17      3      6      3      1      2      2      3      3      3      6   : 36444
aida06 :     36     15      4      9      2      4      2      4      3      3      6   : 37608
aida07 :     20      8      5      3      3      4      2      2      2      2      7   : 37376
aida08 :     21     11     10      2      1      4      2      2      2      4      6   : 37324
aida09 :     27     14      7      6      3      3      2      4      3      3      6   : 37452
aida10 :     25      9      5      2      2      3      3      2      2      4      6   : 37436
aida11 :     30     14      2      4      5      3      2      3      3      3      6   : 36936
aida12 :     22      3      5      3      2      3      2      3      3      3      6   : 36640
aida13 :     10      7      1      4      2      3      3      2      3      3      6   : 36336
aida14 :     13      9     12      2      3      3      2      3      2      4      6   : 37820
aida15 :     20      8     11      3      0      3      2      3      3      3      6   : 36640
aida16 :     23      6      6      2      0      2      1      3      4      3      6   : 37164

Some plots of AIDA dead time (From PAUSE/RESUME data items) against time.

The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead.

Beam spills are indicated by the red bars (1 = on spill)

Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown
Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates .

The much quieter p+n strips show a similar pattern but much less frequently.

Deadtime does not always sync up between FEE64s it seems.

IN all case the pause/resume interval seems to only last for less than 100 ms.

20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)

20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)

21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.8M items/s
Tape server ~10MB/s

Analysed R14_781 deadtimes are okay (Fig 9)

21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)

22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)

22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)

23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~3M items/s
Tape server ~10MB/s

Analysed R14_819 deadtimes are okay (Fig 19)

23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21)

15:55 Took over from OH

16:00
Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System wide checks
All OK, except ADC all fail (as before) and WR

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_691. Attach 4. Dead time <5%

16:34 MBS changes file.


17:00
Stats OK - attach 5
Temps OK - as before
System wide checks as before
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


17:55
Stats OK - attach 6
Temps OK - as before

Leakage current approaching 10 uA - attach 7

System-wide checks - as before except
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x568 : 	 104  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



18:01 MBS switches to new file



18:52
Stats OK - attach 8
Temps OK - as before

System-wide checks as before

Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9

19:36 Stats OK - attach 10
Bias - attach 11

19:48 Shift handed over to LS

System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed.

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2


		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd056 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x16 : 	 22  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      2      0      5      5      1      2      3      3      6   : 36120
aida02 :     25     10      0      4      3      5      2      3      3      3      6   : 36980
aida03 :     22     12      5      2      1      3      2      3      3      3      6   : 36616
aida04 :     16      3      3      3      4      4      4      2      3      3      6   : 36840
aida05 :     14      5      6      2      3      4      4      2      3      3      6   : 36800
aida06 :     30     10      6      3      3      4      2      3      3      3      6   : 36936
aida07 :     27     11      1      0      0      3      2      3      3      3      6   : 36436
aida08 :     38      7      7      3      3      2      2      3      3      3      6   : 36704
aida09 :      4      6      4      2      4      4      1      4      3      3      6   : 37056
aida10 :     21     10      3      3      1      5      1      3      3      3      6   : 36596
aida11 :     24      8      2      1      4      3      2      4      2      3      6   : 36192
aida12 :     31     12      4      3      5      5      2      2      3      3      6   : 36668

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Mon Apr 19 10:32:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

There should be a separate elog note about changing one of the bias voltages.

All histograms zeroed at 16:13.

16.20 DAQ continues file NULL/R33_156
      aida09 HEC fast comparator 0x2->0x1
      DSSSD#3 bias -100->-120V

Auther: Muneerah and Betool
02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok
      Statistics ok - 210517_0202_Stats
      Temperatures ok - 210517_0202_Temp
      Bias and leakage ok - 210517_0200_Bias 
02:05 System wide checks:
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155d : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      8      3      3      3      5      2      2      2      3      7   : 39460
aida02 :     22      4      5      0      3      2      2      3      2      3      7   : 39560
aida03 :     21     10      2      1      1      3      2      4      2      3      7   : 40100
aida04 :     18      7      2      0      3      3      3      2      2      3      7   : 39392
aida05 :     16      6      5      1      2      4      3      2      2      3      7   : 39520
aida06 :     18      6      6      2      2      2      2      3      2      3      7   : 39576
aida07 :     12      8      6      1      2      3      1      3      2      3      7   : 39408
aida08 :     13     11      4      0      1      3      3      2      2      3      7   : 39308
aida09 :     10      8      4      2      3      4      2      3      2      3      7   : 39848
aida10 :     16      8      2      1      2      4      2      3      1      4      7   : 40768
aida11 :     26     10      2      2      1      2      2      3      2      3      7   : 39512
aida12 :     16      9      2      1      2      4      2      2      2      3      7   : 39240
aida13 :     15      9      3      2      2      4      2      2      2      3      7   : 39284
aida14 :     25     10      2      3      3      3      2      3      2      3      7   : 39796
aida15 :      4      2      2      1      1      3      2      2      2      3      7   : 38944
aida16 :     16      7      3      3      1      3      3      2      2      3      7   : 39368

04:00   Statistics ok - 210517_0403_Stats
        Temperatures ok - 210517_0403_Temp
        Bias and leakage ok - 210517_0400_Bias 
04:05 System wide checks:
      Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1560 : 	 14  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      7      5      2      3      5      2      2      2      3      7   : 39492
aida02 :     23      6      7      2      1      3      2      3      2      3      7   : 39676
aida03 :     17      9      3      2      0      3      2      3      2      3      7   : 39548
aida04 :     16      4      1      2      1      3      3      3      2      3      7   : 39792
aida05 :     22      7      3      3      1      4      3      3      2      3      7   : 40032
aida06 :     18      6      7      3      1      3      2      3      2      3      7   : 39688
aida07 :     18      7      4      1      2      4      1      4      2      3      7   : 40032
aida08 :     19      5      4      1      1      3      3      3      2      3      7   : 39828
aida09 :     10      6      7      1      3      4      2      2      2      3      7   : 39336
aida10 :     12     10      2      3      1      4      2      3      1      4      7   : 40768
aida11 :     10      9      4      3      2      1      2      3      2      3      7   : 39440
aida12 :      8      7      6      1      2      4      2      3      2      3      7   : 39768
aida13 :     11      2      6      1      2      3      2      3      2      3      7   : 39612
aida14 :     27      7      4      2      3      3      2      3      2      3      7   : 39780
aida15 :      4      2      3      1      1      3      2      2      2      3      7   : 38960
aida16 :     18      8      4      2      1      4      2      2      2      3      7   : 39240


06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA
06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again.
07:01 it is working fine.
07:40 Screen shots after the restart
 	Statistics ok - 210517_0742_Stats
        Temperatures ok - 210517_0743_Temp
        Bias and leakage ok - 210517_0740_Bias 

System wide checks

	
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0x61da : 	 0x61dd : 	 3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      6      3      1      1      3      1      3      3      3      7   : 40336
aida02 :     11      6      0      1      1      5      2      4      2      3      7   : 40252
aida03 :      0      5      2      3      3      3      3      4      2      3      7   : 40424
aida04 :      2      4      2      1      2      4      3      3      2      3      7   : 39912
aida05 :     11      7      5      1      2      2      4      2      3      3      7   : 40532
aida06 :     20      6      0      2      3      2      1      3      3      3      7   : 40320
aida07 :     15      5      1      2      2      3      2      2      3      3      7   : 40116
aida08 :      2      3      4      1      3      3      2      2      3      3      7   : 40128
aida09 :     18      5      1      3      2      4      2      2      3      3      7   : 40288
aida10 :     11      6      3      4      2      3      3      2      2      4      7   : 41484
aida11 :      4      4      5      2      2      2      3      3      2      3      7   : 39744
aida12 :     18      7      2      3      2      2      4      3      2      3      7   : 40064
aida13 :      2      2      1      3      0      3      2      4      2      3      7   : 39944
aida14 :     16      4      5      3      3      3      3      2      2      4      7   : 41552
aida15 :      9      3      1      1      2      3      2      3      3      3      7   : 40556
aida16 :     16      5      1      2      1      2      3      4      2      3      7   : 40184

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs


12:52 Check FPGA Timestamp errors gives following error message:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"
Tcl Call Trace
6: DocSubstSystemFile sock8 error 500 can't\ read\ &quot\;TS_Base_Collected&quot\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n&quot\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}&quot\;\n\ \ \ \ (procedure\ &quot\;do_TS_errors&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;do_TS_errors\ &quot\;\n\ \ \ \ (&quot\;TSERRORS&quot\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...&quot\;\n\ \ \ \ (procedure\ &quot\;do_click&quot\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n&quot\;do_click&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ &quot\;&quot\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ &quot\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...&quot\;\n\ \ \ \ (file\ &quot\;Check.tcl&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;source\ Check.tcl&quot\;\n\ \ \ \ (in\ namespace\ eval\ &quot\;::Check&quot\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n&quot\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n&lt\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n&gt\;...&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ #0\ [list\ subst\ \$script\]&quot\;\n\ \ \ \ (procedure\ &quot\;SubstFile&quot\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n&quot\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]&quot\;\n\ \ \ \ (procedure\ &quot\;Subst_File&quot\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n&quot\;Subst_File\ \$template\ \$interp&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_application/x-tcl-template&quot\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n&quot\;\$cmd\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_Handle&quot\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n&quot\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;DocDomain&quot\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n&quot\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml&quot\;\n\ \ \ \ (&quot\;eval&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]&quot\;
5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\"
4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE
3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {}
2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml}
1: HttpdReadPostGlobal sock8 Httpds

Dead time check:

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_24
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252995858 (  451517.9 Hz)
        Other data format:    8924143 (   15926.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.2 Hz)
                               RESUME:         85 (       0.2 Hz)
                              SYNC100:      34087 (      60.8 Hz)
                              WR48-63:      34087 (      60.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8855799 (   15804.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      560.323 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000              349.960
  1                0.002                0.000
  2                0.000                5.974
  3                1.105                0.000
  4                0.000              447.454
  5                0.000                2.809
  6                0.000               16.727
  7                0.137                0.000
  8                0.090                0.000
  9                0.000              429.897
 10                0.461                0.000
 11                0.018                0.000
 12                0.000                8.985
 13                0.000              466.311
 14                0.000               15.095
 15                0.346                0.000

13:24 Beam is off at the moment...

13:43 Changing spill structure from 1.5s/3.5s to 1s. 

13:50

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_29
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251763810 (  930342.1 Hz)
        Other data format:   10156190 (   37530.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        205 (       0.8 Hz)
                               RESUME:        205 (       0.8 Hz)
                              SYNC100:      33009 (     122.0 Hz)
                              WR48-63:      33009 (     122.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10089762 (   37284.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     527077 (    1947.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      270.614 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.034                6.803
  1                8.461                0.000
  2                0.569                0.000
  3               11.664                0.000
  4                0.024               26.416
  5                0.854                0.000
  6                0.069                0.000
  7                0.741                0.000
  8                1.336                0.000
  9                0.000               75.926
 10                0.353                0.000
 11                0.105                0.000
 12                0.070                0.000
 13                0.000               93.585
 14                0.017                0.000
 15                0.121                0.000

15:06 Spill structure back to 1.5 on 2s off.

16th May 08:00 - 12:00 shift
Author: MS


08:00


	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7163 : 	 23  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      7      2      3      2      3      2      4      2      3      7   : 40228
aida02 :      3      8      3      2      2      2      2      4      2      3      7   : 39996
aida03 :     23      9      6      1      0      3      2      4      2      3      7   : 40100
aida04 :     34     27     17      7      2      4      3      3      2      2      7   : 38608
aida05 :     19      9      5      2      2      2      3      3      2      3      7   : 39844
aida06 :      5      5      2      1      0      4      2      4      2      3      7   : 40060
aida07 :     28      2     10      3      2      4      1      4      2      3      7   : 40192
aida08 :     19      5      5      1      2      5      1      3      2      3      7   : 39652
aida09 :     20      8      3      3      1      3      2      3      2      3      7   : 39648
aida10 :     27     10      0      2      2      2      2      3      1      4      7   : 40572
aida11 :      3      3      2      1      2      4      2      3      2      3      7   : 39652
aida12 :     16      7     11      2      3      4      2      2      2      3      7   : 39464
aida13 :     14     10      4      3      1      5      2      2      2      3      7   : 39400
aida14 :     21      9     10      1      1      2      2      3      1      4      7   : 40604
aida15 :     19      8      2      3      0      4      3      2      2      3      7   : 39436
aida16 :     24      5      8      3      3      4      2      2      2      3      7   : 39464







 *** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000






10:00


	
FEE64 module aida06 global clocks failed, 6
 FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7166 : 	 26  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      7      6      2      2      3      2      3      2      3      7   : 39716
aida02 :      8      4      3      2      2      3      2      3      2      3      7   : 39600
aida03 :     23     11      5      1      0      4      2      3      1      4      7   : 40740
aida04 :     42     25     16      7      2      4      4      3      2      2      7   : 38864
aida05 :     24      5      5      1      2      2      3      3      1      4      7   : 40824
aida06 :     11      4      4      1      1      4      2      4      2      3      7   : 40172
aida07 :     21      8      5      1      2      3      2      4      2      3      7   : 40196
aida08 :     15      5      6      1      1      4      2      4      2      3      7   : 40228
aida09 :     15     10      4      1      2      3      2      3      2      3      7   : 39660
aida10 :     23      8      2      2      2      2      2      3      1      4      7   : 40572
aida11 :      6      4      4      2      2      4      2      3      2      3      7   : 39736
aida12 :     18      8      8      1      4      4      3      2      2      3      7   : 39720
aida13 :     23      6      2      3      2      5      2      2      2      3      7   : 39436
aida14 :     29      3     11      1      1      2      2      3      1      4      7   : 40604
aida15 :     29      7      2      2      0      4      3      3      2      3      7   : 39948
aida16 :      2      3      6      2      3      4      2      2      2      3      7   : 39296






*** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

12:00-16:00
16th May 12:00 - 16:00 shift
Author: JS

11:57 Taking over from Magda. Running full checks.
usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75%
Current ok  06.410 uA      006.835 uA
Stats good 1Statistics aidas-gsi(6).png
Temps ok 1Temperature and status scan aidas-gsi(6).png
Analysis ok R7_385. Dead time FEE1 a little hight 6%

PAUSE: 166  RESUME: 166

 *** Timestamp elapsed time:      196.305 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.044                0.000
  1               12.405                0.000
  2                0.807                0.000
  3                7.260                0.000
  4                0.014                0.000
  5                0.458                0.000
  6                0.027                0.000
  7                0.497                0.000
  8                0.723                0.000
  9                0.000               88.857
 10                6.189                0.000
 11                1.443                0.000
 12                0.474                0.000
 13                0.000               35.565
 14                0.000                0.000
 15                0.147                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x716e : 	 34  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      7      4      2      2      3      2      3      1      4      7   : 40712
aida02 :     25     10      7      1      1      4      1      3      2      3      7   : 39556
aida03 :     22     10      5      1      0      4      2      4      2      3      7   : 40216
aida04 :     40     24     17      7      2      5      3      3      2      2      7   : 38736
aida05 :      4      8      3      0      1      3      3      3      1      4      7   : 40768
aida06 :     21      6      6      2      2      3      2      4      2      3      7   : 40228
aida07 :     25      4      6      3      2      3      2      4      2      3      7   : 40260
aida08 :     19      9      4      1      1      4      2      4      2      3      7   : 40244
aida09 :     16      9      4      2      2      3      2      3      2      3      7   : 39688
aida10 :     19     10      5      1      2      2      2      4      1      4      7   : 41100
aida11 :     21     10      2      1      2      3      3      3      2      3      7   : 39908
aida12 :     25      7      7      2      2      3      2      3      2      3      7   : 39756
aida13 :     13      7      3      4      2      5      2      3      2      3      7   : 39964
aida14 :     21      7      9      2      1      2      2      4      1      4      7   : 41116
aida15 :     23      6      2      3      0      4      3      3      2      3      7   : 39948
aida16 :      9     11     10      2      3      4      2      2      2      3      7   : 39452

Tom says Aida09 clock fail is ok as its status bit is "6".
The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken.

12:35 -
usbec ok.
Current ok 
Stats good 
Temps ok 
Analysis ok R7_395. Dead time FEE1 still high 6.6%

13:33 -
usbec ok. Max implants ~ 1.8kHz
Current ok    006.850 uA      007.250 uA 
Stats - Aida11 runing low < 5k was ~20k overnight
Temps ok 
Analysis R7_415. Dead time FEE1 & FEE10 high 10%

14:00
usbec ok - ucesb1.png
Current ok - bias1.png
Stast -  Aida11 low - 

14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n.

15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool.

18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00.
      During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise.
      The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs.
      We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy.
      During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips.

18:08 System wide checks all ok - bar some ADC but waveforms disabled
      Statistics ok - 210516_1809_Stats
      Temperatures ok - 210516_1809_Temp
      Bias and leakage ok - 210516_1810_Bias

18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time
18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side.

19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves.
      Now have around 4.2TB left which will provide around 80 hours of writing

20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not.

20:57 System wide checks. Clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1556 : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics ok - 210516_2056_Stats
      Temp ok -  210516_2058_Temp
      Bias and leakage current ok -  210516_2058_Bias

23:16 System wide checks:
      Clock still ok 
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155a : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

       Statistics - 210516_2315_Stats
       Temperature - 210516_2316_Temp
       Bias and leakage current ok - 210516_231

16th May 2021 - 00:00 - 08:00 Shift
Author: CA

00:15 System wide checks;

FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x7154 : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19     10      3      1      2      3      2      3      2      3      7   : 39660
aida02 :     29      5      3      1      1      3      2      3      2      3      7   : 39596
aida03 :     15      7      1      1      2      3      2      3      2      3      7   : 39588
aida04 :     44     26     19      5      2      4      3      3      2      2      7   : 38608
aida05 :     20     10      3      0      1      4      2      2      3      3      7   : 40208
aida06 :     27      6      2      1      2      4      2      4      2      3      7   : 40284
aida07 :     25      5      3      1      2      3      2      3      2      3      7   : 39644
aida08 :     23      8      2      3      1      4      2      3      2      3      7   : 39772
aida09 :     23     10      3      2      3      4      1      3      2      3      7   : 39644
aida10 :     18     10      3      2      3      2      2      2      2      4      7   : 41160
aida11 :     16      7      3      2      1      2      2      3      2      3      7   : 39464
aida12 :     21      4      7      1      3      3      3      3      2      3      7   : 40004
aida13 :     25      6      3      2      1      3      1      4      2      3      7   : 39876
aida14 :     20      8     10      1      1      2      2      2      2      4      7   : 41104
aida15 :     26      6      3      3      0      4      3      2      2      3      7   : 39464
aida16 :      9      8      2      4      1      3      1      4      2      3      7   : 39876

00:30 Stats ok - 210516_0030_stats.png
      Temps ok - 210516_0030_temps.png
      Bias ok - 210516_0030_bias.png

00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt

02:21 system wide checks as above, except;

      		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x7159 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x7 : 	 7  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:28 Stats ok - 210516_0230_stats.png
      Temps ok - 210516_0230_temps.png
      Bias ok - 210516_0230_bias.png

02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt

04:36 system wide checks same as above, except;

		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x715c : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x9 : 	 9  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:37 Stats ok - 210516_0430_stats.png
      Temps ok - 210516_0430_temps.png
      Bias ok - 210516_0430_bias.png

04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt

06:38 system wide checks same as before, except;

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7161 : 	 21  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

06:40 Stats ok - 210516_0640_stats.png
      Temps ok - 210516_0640_temps.png
      Bias ok - 210516_0640_bias.png

06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt

08:00 OH Takes over for CA
      Thresholds 0xc for p+n and 0x19 for n+n ASICs 1-3 and 0x64 for n+nASIC4

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

       Max average dead time in R14_586 3.2% which is 9.6% on spill
       Stable from yesterday

10:02 System wide checks. All ok expect WR
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x554 : 	 84  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics - Attachment 1
      Temp - attachment 2
      Bias and leakage currents ok - Attachment 3


11:17 A new run was started as the microspill has been changed. I believe to optimise FRS deadtime.
11:45 They are still working on it

12:05 Beam off for a bit while they optimise spill

12:15 System wide checks ok expect WR
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - Attachment 4
     Temp - Attachment 5
     Bias - Attachment 6

13:11 Magda has said that the changes to the spill structure only gained them a few percent in dead time
      Further changes could be done, but would take at least two hours

13:19 Dead time following the current spill changes
	R14_648	R14_649	R14_650	R14_651	Sum DT (S)   Avergage DT (%)   On Spill DT (%)
FEE	244.823	252.168	247.155	240.421	984.567		
0	0.063	0.025	0.105	0.096	0.289	     0.029353005	0.088059015
1	3.981	4.907	4.17	2.484	15.542	     1.578561947	4.73568584
2	0.01	0	0.121	0.779	0.91	     0.092426417	0.277279251
3	3.625	2.972	2.766	3.335	12.698	     1.289704002	3.869112006
4	0.088	0.1	0.12	0.136	0.444	     0.045095966	0.135287898
5	9.392	5.798	9.835	8.01	33.035	     3.355282068	10.0658462
6	0.153	0.075	0.01	0.517	0.755	     0.076683456	0.230050367
7	7.4	7.565	7.703	6.415	29.083	     2.953887343	8.86166203
8	0.221	0.606	1.321	0.539	2.687	     0.272911849	0.818735546
9	0	0	0	0	0	     0          	0
10	1.381	0.368	1.196	1.091	4.036	     0.409926394	1.229779182
11	0.066	0.025	0.039	0.017	0.147	     0.014930421	0.044791264
12	0.166	0.041	0.062	0.024	0.293	     0.029759275	0.089277825
13	0	0	0	0	0	     0          	0
14	0.71	0	0.005	0.074	0.789	     0.08013675	        0.240410251
15	0.239	0.17	0.208	0.286	0.903	     0.091715444	0.275146333
16	0	0	0	0	0	     0          	0

14:03 System wide checks ok - WR same difference as before
      Statistics - attachment 7
      Temperatures - attachment 8
      Bias and leakage currents - attachment 9

00:00 CA takes over
      LS has performed usual checks, see previous Elog entry - all ok

00:30 stats ok
      R14_470 deadtime at 4.4%

02:10 system wide checks ok except:

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x52f : 	 47  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

02:12 FEE64 Temps ok - attachment 1
      Stats ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      dead-times ok - R14_497 max 4% - attachment 4

03:27 DAQ continues ok - writing to file R14_517

04:07 system wide checks ok except:

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x537 : 	 55  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 Temps ok - attachment 6
      Stats ok - attachment 5
      detector bias / leakage currents ok - attachment 7
      dead-times ok - R14_526 max 4% - attachment 8

05:27 bad timestamp error in merger terminal - attachment 9

06:05 all system wide checks ok except;

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x53d : 	 61  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

06:07 FEE64 Temps ok - attachment 10
      Stats ok - attachment 11
      detector bias / leakage currents ok - attachment 12
      dead-times ok - R14_558 max 3.6% - attachment 13

07:01 another bad timestamp error in merger terminal - attachment 14

19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x520 : 	 32  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~1.8M items/s
Tape server ~7MB/s

Analysed R14_399, deadtimes are okay (Fig 4)

20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)

21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)

21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)

22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x524 : 	 36  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.3M items/s
Tape server ~8MB/s

Analysed R14_430, deadtimes are okay (Fig 14)

22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%

23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)

23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)

24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x529 : 	 41  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

The difference has increased over time (not sure if an issue)

Merger rate ~3M items/s
Tape server ~8MB/s

Analysed R14_462, deadtimes are okay (Fig 24)

15:58 Took shift over from OH and PP

Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x510 : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     22      8      7      4      1      4      2      2      2      2      7   : 37320
aida02 :     12      9      4      2      4      2      3      3      3      3      6   : 36856
aida03 :     19      6      7      2      2      2      2      4      3      3      6   : 37036
aida04 :     18      9     10      6      1      4      1      4      3      3      6   : 37168
aida05 :     17     12      2      1      2      3      3      3      3      3      6   : 36836
aida06 :     19      8      7      7      2      4      2      4      3      3      6   : 37468
aida07 :     28      5      4      1      2      4      2      3      2      2      7   : 37752
aida08 :     17      5      4      2      1      2      2      3      3      3      6   : 36396
aida09 :     21     10      4      5      4      4      3      3      3      3      6   : 37252
aida10 :     15     13      4      2      2      2      3      2      2      4      6   : 37284
aida11 :     22     13      3      3      4      3      1      4      3      3      6   : 37072
aida12 :      2      8      1      3      1      3      2      3      3      3      6   : 36472
aida13 :     23      5      4      4      1      4      3      2      3      3      6   : 36484
aida14 :     19      7      4      3      1      3      2      4      2      4      6   : 38116
aida15 :     20     10      2      2      3      2      3      2      3      3      6   : 36288
aida16 :     29      6      1      2      0      3      1      4      3      3      6   : 36724

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

Analysed R14_342. Attach 4. Dead time <5%



17:00

Stats OK - attach 5
Temps OK - as before
Bias  increasing quite quickly - attach 6

System-wide checks

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x512 : 	 18  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     28      7      8      4      1      4      2      2      2      2      7   : 37352
aida02 :     24     11      6      4      3      2      3      2      3      3      6   : 36440
aida03 :     17     11      5      2      1      2      2      3      3      3      6   : 36460
aida04 :     25     13     10      6      2      4      2      3      3      3      6   : 37036
aida05 :     23      7      3      2      2      3      3      3      3      3      6   : 36868
aida06 :      1     12      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     22     12      1      1      2      4      2      2      2      2      7   : 37224
aida08 :     21      8      7      2      1      3      3      2      3      3      6   : 36356
aida09 :     23      7      4      6      4      3      2      4      3      3      6   : 37396
aida10 :     27      7      2      2      2      2      3      3      2      4      6   : 37764
aida11 :     12      5      3      5      4      3      1      4      3      3      6   : 37032
aida12 :     23      9      4      1      2      3      2      3      3      3      6   : 36612
aida13 :     10     12      6      3      1      4      3      2      3      3      6   : 36488
aida14 :     23      7      3      3      1      3      2      3      2      4      6   : 37604
aida15 :     22      6      6      4      2      2      3      2      3      3      6   : 36328
aida16 :     27      7      3      1      0      3      1      4      3      3      6   : 36724



18:00

Stats OK - attach 7
Temps OK - as before
Bias - attach 8

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x517 : 	 23  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_371. Attach 9. Dead time below 5%.



18:35 MBS DAQ crashed



18:40
Stats OK - attach 9
Temps OK - as before
Bias - attach 10

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51a : 	 26  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:42 MBS DAQ restarted - new file being opened.


19:36
Stats OK - attach 11
Temps OK - attach 12
Bias - attach 13

System-wide checks
Clock OK
ADC all fail

	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51e : 	 30  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA OK

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     24     11      5      4      1      4      2      3      2      2      7   : 37832
aida02 :      0      5      6      3      4      2      3      2      3      3      6   : 36328
aida03 :     15      8      5      3      2      2      1      3      4      3      6   : 37292
aida04 :     14      8      7      5      3      4      2      3      3      3      6   : 36936
aida05 :     31      6      1      1      2      3      3      3      2      4      6   : 37852
aida06 :      9      8      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     21      7      2      1      2      4      1      3      2      2      7   : 37452
aida08 :      8      4      5      3      1      3      2      3      3      3      6   : 36528
aida09 :     24      7      3      5      4      3      2      4      3      3      6   : 37352
aida10 :     19      9      7      1      2      2      3      2      2      4      6   : 37284
aida11 :     27      5      0      5      4      3      1      4      3      3      6   : 37044
aida12 :     31      9      2      1      2      3      2      3      3      3      6   : 36612
aida13 :     21     10      4      3      1      4      3      2      3      3      6   : 36484
aida14 :     22      6      4      2      1      3      2      4      2      4      6   : 38088
aida15 :     21      9      3      4      2      2      3      2      3      3      6   : 36300
aida16 :     23     11      4      2      1      4      2      3      3      3      6   : 36724


19:43 Shift handed over to LS

08:00 OH Takes over

      Statistics still look good from yesterday. Rate is still down.
      Dead time in FEEs 2 and 4 is 2.04%

      System wide checks clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x500 : 	 38  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      *Baseline reset*
      FPGA check ok
      Stats - attachment 1
      Temp - attachment 2
      Bias - attachment 3
      

08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips.
      I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4
      The results of the test appear to indicate that it is the network link that we are saturating causing dead time
***	Before		                  After	
FEE	DT (s)  Dead time %	           DT (s)	Dead time %
0	0.013	0.004184451	           0.026	0.010025681
1	6.352	2.04458693	           4.097	1.57981599
2	0.295	0.09495484	           0.292	0.112596112
3	6.349	2.043621288	           8.255	3.183153771
4	0.021	0.006759497	           0.03	        0.011568094
5	0.285	0.091736032	           3.388	1.306423377
6	0.02	0.006437616	           0.149	0.057454865
7	1.219	0.392372712	           2.133	0.822491459
8	1.137	0.365978485	           0.457	0.176220627
9	0	0	                       0	0
10	1.633	0.525631369	           1.545	0.595756823
11	0.061	0.01963473	           0.166	0.064010118
12	0.059	0.018990968	           0.085	0.032776265
13	0	0	                       0	0
14	0.016	0.005150093	           0.023	0.008868872
15	0.282	0.09077039	           0.168	0.064781324


      Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased

09:00 They are stopping the run to decrease the bias on the bPlast detector
09:25 No noticable change in AIDA
      They will now begin to change the spill structure.

09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change

10:21 Beam is back on AIDA

10:39 System wide checks. 
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x506 : 	 6  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

11:00 
      Stats - attachment 4
      Temp - attachment 5
      Bias - attachment 6

12:46 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50b : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

14:36 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50f : 	 15  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

0.00 - Taking over from Lewis 

00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2)

01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4)

01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6)

02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8)

Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4ee :      20  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     43     12      6      4      1      4      2      2      1      3      7   : 38444
aida02 :     21     12      4      4      2      3      3      3      3      3      6   : 36980
aida03 :     23      8      4      2      2      4      2      3      3      3      6   : 36764
aida04 :     34     12     10      5      1      4      1      4      3      3      6   : 37224
aida05 :     18      5      5      1      2      3      2      2      1      3      7   : 38112
aida06 :     40     12      5      7      2      4      2      4      3      3      6   : 37552
aida07 :     40     10      6      2      2      4      2      2      2      2      7   : 37392
aida08 :     19     11      3      3      1      4      3      3      3      3      6   : 36980
aida09 :     43      6      2      4      3      4      3      3      3      3      6   : 37180
aida10 :     32     14     10      3      4      1      2      4      1      3      7   : 39280
aida11 :     40      7      2      5      4      3      1      4      3      3      6   : 37144
aida12 :     37     14     13      2      2      3      2      4      3      3      6   : 37396
aida13 :     18      9      3      1      1      5      3      3      3      3      6   : 37024
aida14 :     36     13      6      4      2      3      2      3      1      3      7   : 38872
aida15 :     32     11      7      4      3      4      2      3      2      4      6   : 38024
aida16 :     26      9      4      1      0      3      1      4      3      3      6   : 36752

02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13)

03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15)

03.05 - No beam, problem with SIS, don't know how long will be off

03.14 - Beam is back

03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17)

04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19)

Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4f2 :      24  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     26     11      5      6      1      4      2      2      1      3      7   : 38416
aida02 :     16      9      5      4      2      4      2      2      3      3      6   : 36312
aida03 :      9      7      4      4      2      4      2      3      3      3      6   : 36764
aida04 :     36     11      8      4      3      4      1      3      4      3      6   : 37800
aida05 :     20      4      3      2      2      3      2      2      1      3      7   : 38112
aida06 :     22     13     10      7      2      4      3      3      4      3      6   : 38336
aida07 :     28      7      7      2      2      4      2      2      2      2      7   : 37336
aida08 :      2      3      4      4      1      3      2      3      3      3      6   : 36512
aida09 :     40      7      1      4      4      3      3      3      3      3      6   : 37096
aida10 :     26      9     12      2      3      2      2      4      1      3      7   : 39280
aida11 :     30      7      1      5      4      3      1      3      2      4      6   : 37600
aida12 :     15     10      6      4      2      3      2      4      3      3      6   : 37228
aida13 :      1      3      2      1      2      4      3      2      3      3      6   : 36316
aida14 :     34     12      7      4      2      3      2      3      1      3      7   : 38872
aida15 :     18     11      5      3      4      4      2      3      2      4      6   : 37968
aida16 :     22      7      4      2      0      3      1      4      3      3      6   : 36752

04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24)

05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26)

05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28)

06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30)

Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4f9 :      31  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     16      9      3      7      1      5      1      3      1      3      7   : 38744
aida02 :     18      3      5      2      3      3      3      2      3      3      6   : 36400
aida03 :     21      9      4      2      2      4      2      4      3      3      6   : 37276
aida04 :     32     12      7      3      3      4      1      3      4      3      6   : 37744
aida05 :     12      8      3      2      2      4      2      2      1      3      7   : 38240
aida06 :     28     14      4      8      2      4      3      3      3      3      6   : 37280
aida07 :     40     12      5      2      2      4      1      3      2      2      7   : 37648
aida08 :     13      8      4      4      1      3      2      3      3      3      6   : 36596
aida09 :     35      5      2      4      3      4      2      4      3      3      6   : 37396
aida10 :      6     10      4      1      1      3      2      2      1      3      7   : 38024
aida11 :     41      9      1      4      4      3      2      4      3      3      6   : 37372
aida12 :     29     11      9      4      2      3      2      4      3      3      6   : 37340
aida13 :     24     10      5      1      2      4      3      2      3      3      6   : 36512
aida14 :     34     16      5      4      2      3      2      3      1      3      7   : 38872
aida15 :     42      8      6      2      4      4      2      3      2      4      6   : 38024
aida16 :      8      9      6      1      0      3      1      4      3      3      6   : 36712

06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35)

07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37)

07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39)

20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s

21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)

21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)

22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4e7 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.4M items/s
Tape server 6.3MB/s

Ran R14_119 through analyser, dead times all good (Fig 11)

22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)

23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)

23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)

24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4eb : 	 17  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.5M items/s
Tape server 6.5MB/s

Ran R14_142 through analyser, dead times all good (Fig 21)

16:25

Informed at shift handover that there has been no beam.

Statistics  - See attachment 1

Temperatures - See attachment 2

Bias & Leakage currents - See attachment 3

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit decoder status:

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4dd :      3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:00

Usecb scalars now showing DSSD implant rates.

Statistics  - See attachment 4

Temperatures - See attachment 5

Bias & Leakage currents - See attachment 6

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx

19:45

Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.

Statistics  - See attachment 7

Temperatures - See attachment 8

Bias & Leakage currents - See attachment 9

System Wide checks - All passed, EXCEPT:

ADC Calibration:

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit decoder status:

     Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes

19:55 - Beam now back again - DSSD implants showing

20:00 Taking over from CB

20:33 Doing full checks.
Temps ok elog:319/8  
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

FPGA Timestamp errors -
		 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     34     30      8      1      4      1      2      2      3      6   : 35772
aida02 :     37     30     28      5      3      1      2      4      2      3      6   : 36644
aida03 :     41     29     30     16      4      3      1      5      1      3      6   : 36588
aida04 :     13      6     23     12      4      4      4      4      2      2      6   : 35412
aida05 :     20     32     23     10      2      2      2      2      1      4      6   : 36736
aida06 :     14      5     10      5      1      3      2      3      2      3      6   : 35680
aida07 :     44     23     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     39     30     28      9      0      3      2      3      3      2      6   : 35308
aida09 :     41     31     28     10      2      5      1      3      1      3      6   : 35484
aida10 :     43     23     27      7      3      1      1      3      3      3      6   : 36916
aida11 :     39     35     17     10      3      3      2      2      2      3      6   : 35908
aida12 :     37     36     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     47     32     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     23     33     14     10      3      4      3      4      3      2      6   : 36164
aida15 :     44     28     24     12      3      2      4      3      3      2      6   : 35920
aida16 :     22     17     24     17      3      4      2      3      1      3      6   : 35648

Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27 
Resume info #3 - elog:319/3 same as elog:317/26 
Pause info #2 - elog:319/4 same as elog:317/32 
AIDA disk info #6 - elog:319/5 same as elog:317/31 
AIDA ADC data items - elog:319/6 same as elog:317/30 
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see 
clearly once uploaded the screenshots] 

Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%

21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay

21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     39     34     29      8      1      4      1      2      2      3      6   : 35772
aida02 :     46     29     28      5      3      2      2      3      2      3      6   : 36288
aida03 :     39     25     29     16      4      2      2      5      1      3      6   : 36660
aida04 :     14      6     19     12      4      4      3      4      2      2      6   : 35096
aida05 :     20     30     24     10      2      2      2      2      1      4      6   : 36736
aida06 :     26      4     11      5      2      4      3      2      2      3      6   : 35672
aida07 :     46     22     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     28     32     28      9      0      2      2      4      3      2      6   : 35664
aida09 :     32     30     29     10      3      5      2      3      1      3      6   : 35776
aida10 :     39     23     25      7      3      1      1      3      3      3      6   : 36868
aida11 :     40     31     21     10      3      3      1      4      1      3      6   : 35688
aida12 :     17     31     12     10      2      2      3      4      3      2      6   : 35772
aida13 :     35     28     22      7      4      2      3      4      3      2      6   : 36012
aida14 :     30     33     14     10      3      4      3      4      3      2      6   : 36192
aida15 :     34     26     24     12      3      2      4      3      3      2      6   : 35864
aida16 :     38     24     27     17      3      4      2      3      1      3      6   : 35816

Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12 
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14 
Aida Correlation Info #8 - elog:319/15  - rates showed zero, checked again and normal elog:319/17

Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%

22:00
Temps ok
Stats ok 
ucesb ok

22:35 
Temps ok
Stats ok 
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up 
again elog:319/17

22:54 They are closing the file, we are on R5_155

23:00 

S496/R5_18 - attachment 1

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 0xf
                             n+n FEE64s 0x20
Sampling ADCs OFF


S496/R5_32 - attachment 2

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 & aida15 0xf
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


S496/R5_45 - attachment 3

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xc
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


* N.B. Ignore 'elapsed idle time(s)' metric which does not work with WR timestamps
 

16:00 Shift taken over from CA

16:00 
Temperatures OK. Attach 1. 
Upon reloading aida11 first returns "No respose" for all values. 
Upon refreshing, aida11 responds but aida 9 returns "No response". 
Upon refreshing again, attach 1.

Bias OK. Attach 2.

Statistics OK. Attach 3-7

System-wide checks
Clock OK
ADC - all fail
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     32     28     25      7      1      4      1      3      2      3      6   : 36112
aida02 :     43     28     29      5      4      2      3      4      1      3      6   : 36092
aida03 :     39     26     28      5      4      3      2      5      1      3      6   : 36428
aida04 :      9      5     21      8      5      4      4      4      2      2      6   : 35292
aida05 :     46     32     26      7      2      2      2      2      2      3      6   : 35768
aida06 :      4      5     16      1      3      4      2      4      1      3      6   : 35352
aida07 :     47     21     25      5      4      3      1      4      1      3      6   : 35604
aida08 :     62     33     32      6      0      2      3      3      3      2      6   : 35520
aida09 :     33     25     28      7      3      5      2      4      1      3      6   : 36140
aida10 :     47     29     22      6      3      2      2      2      2      3      6   : 35716
aida11 :     39     30     29      6      2      3      1      3      2      3      6   : 36124
aida12 :     38     36     20      6      3      2      3      3      3      2      6   : 35448
aida13 :     42     29     24      3      4      2      3      3      3      2      6   : 35440
aida14 :     31     28     18      7      3      3      4      4      2      2      6   : 35228
aida15 :     48     25     30      5      3      2      4      4      2      2      6   : 35272
aida16 :     37     16     28      6      3      3      2      2      2      3      6   : 35796

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021



16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time. 
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
      Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20



17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17

System-wide checks
Clock - all pass
ADC - all pass
WR
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     56     35     27      8      1      4      1      2      2      3      6   : 35816
aida02 :     43     31     27      5      3      1      3      3      2      3      6   : 36404
aida03 :     45     28     28     16      4      2      2      4      1      3      6   : 36180
aida04 :      2      2     18     12      4      3      3      4      2      2      6   : 34872
aida05 :     28     28     24     10      2      2      2      2      1      4      6   : 36752
aida06 :     28     11      9      5      2      3      3      3      1      3      6   : 35064
aida07 :     52     24     23      9      4      3      1      2      2      3      6   : 35744
aida08 :     21     26     27      8      0      2      3      3      3      2      6   : 35284
aida09 :     40     32     26     10      3      5      1      3      1      3      6   : 35520
aida10 :     46     23     27      7      3      1      1      3      3      3      6   : 36928
aida11 :     49     31     20     10      2      2      1      2      2      3      6   : 35516
aida12 :     39     35     17      9      2      2      3      4      3      2      6   : 35940
aida13 :     40     32     20      7      4      2      3      4      3      2      6   : 36032
aida14 :     33     33     16     10      3      4      3      4      3      2      6   : 36236
aida15 :     53     27     25     12      3      2      4      4      2      2      6   : 35452
aida16 :     35     22     32     15      3      4      3      3      1      3      6   : 36060

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24

System-wide checks
Clock - all pass
ADC - all pass
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     31     31      8      1      4      1      2      2      3      6   : 35788
aida02 :     42     31     29      5      3      2      4      4      1      3      6   : 36304
aida03 :     29     30     29     16      4      3      2      4      1      3      6   : 36276
aida04 :     28      4     20     12      4      3      4      4      2      2      6   : 35280
aida05 :     25     28     23     10      2      2      2      2      1      4      6   : 36724
aida06 :      5      3     10      5      2      4      2      2      2      3      6   : 35308
aida07 :     38     26     22      9      4      3      1      2      2      3      6   : 35688
aida08 :     52     30     28      8      0      2      2      3      3      2      6   : 35200
aida09 :     22     26     21     10      3      4      1      4      1      3      6   : 35704
aida10 :     37     28     25      7      3      1      1      3      3      3      6   : 36900
aida11 :     53     30     20     10      2      3      2      2      2      3      6   : 35908
aida12 :     37     34     17     10      2      2      3      4      3      2      6   : 35956
aida13 :     47     34     16      7      4      2      3      4      3      2      6   : 36012
aida14 :     37     29     17     10      3      4      3      4      3      2      6   : 36236
aida15 :     44     28     25     12      3      2      4      4      2      2      6   : 35424
aida16 :     40     21     33     15      3      4      1      3      1      3      6   : 35576

18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%


19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32

System-wide checks

Clock, ADC - all pass
WR

	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc4 : 	 12  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     33     30      8      1      4      1      2      2      3      6   : 35788
aida02 :     51     30     30      5      3      2      3      4      1      3      6   : 36092
aida03 :     52     31     28     16      4      2      2      5      1      3      6   : 36744
aida04 :      6      8     20     12      4      4      3      4      2      2      6   : 35096
aida05 :     30     31     22     10      2      2      2      2      1      4      6   : 36752
aida06 :     12      6     10      5      2      3      3      2      2      3      6   : 35488
aida07 :     53     24     21      9      4      3      1      3      2      3      6   : 36228
aida08 :     36     28     28      9      0      2      3      3      3      2      6   : 35408
aida09 :     27     28     25     10      3      4      2      3      1      3      6   : 35548
aida10 :     50     23     26      7      3      1      1      3      3      3      6   : 36928
aida11 :     42     28     22     10      3      3      2      2      2      3      6   : 35944
aida12 :     39     35     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     43     34     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     42     32     16     10      3      4      3      4      3      2      6   : 36264
aida15 :     36     30     25     12      3      2      4      3      3      2      6   : 35920
aida16 :     40     21     25     15      3      4      2      3      1      3      6   : 35704

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19:40 Analysed R5_86. Attached. Dead time as before.

19:57 Shift handed over to James S

08:00 CA takes over

08:28 Problem at UNILAC - no beam

09:30 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveforms)

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA timestamp errors:

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

09:36 FEE64 Temperatures ok - attachment 1

      Detector bias / leakage currents ok - attachment 2

      Statistics - attachments 3-8

09:50 beam is back - writing to file R4_265

09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA

10:10 FRS increase degrader thickness

10.28 all histograms, stats and merger stats zero'd

10.50 all system wide checks ok *except*
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      2      2      2      2      2      2      4      3      3      6   : 36860
aida02 :      7      3      5      1      2      5      2      4      3      3      6   : 37284
aida03 :      7      5      3      1      4      4      2      3      3      3      6   : 36756
aida04 :      3      1     20      4      2      3      3      5      1      3      6   : 36052
aida05 :     19     25     26      6      2      2      3      2      2      3      6   : 35828
aida06 :      1      3     16      1      3      4      1      5      1      3      6   : 35580
aida07 :      6      4      1      2      3      3      2      3      3      3      6   : 36552
aida08 :      8      5      2      3      2      3      2      2      2      4      6   : 37064
aida09 :      9      7      1      2      2      4      2      3      3      3      6   : 36652
aida10 :      2      5      3      1      1      4      2      3      3      3      6   : 36544
aida11 :     16      4      4      2      1      2      2      3      3      3      6   : 36384
aida12 :      2      1      1      3      1      2      2      3      3      3      6   : 36288
aida13 :      0      1      1      2      0      4      3      2      3      3      6   : 36184
aida14 :      6      3      2      0      2      3      2      3      3      3      6   : 36432
aida15 :     25      8      2      1      2      3      1      2      2      4      6   : 36836
aida16 :      3      5      1      2      2      2      3      2      3      3      6   : 36100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.50 DAQ continues S496/R4_304

11:18 FEE64 Temperatures ok - attachment 9

      Detector bias & leakage currents ok - attachment 10

      statistics - attachments 11-16

15:00 FEE64           avg. CPU usage (%)

      1               52
      2               55
      3               55
      4               60
      5               93
      6               50
      7               53
      8               55
      9               55
      10              51
      11              50
      12              67
      13              56
      14              55
      15              55
      16              58

15:08 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveform)

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA errors:

 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:10 Temperatures ok - attachment 17

      Detector bias / leakage current ok - attachment 18

      Statistics - attachments 18 -24

00:00 OH Takes over.
 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
0x210 

 p+n Thresholds 0xa (100keV)
 n+n Threshold 0x20 (320keV)

01:00 System wide checks
 Clock status ok
 ADC calibration fail as expected (Waveforms off)
 WR Errors
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dbf : 	 7  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 FPGA Errors
	
                Base Current Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Statistics ok - attachment 1
 Temp ok - attachment 2
 Bias and leakage current ok - attachment 3

01:42 Comparing the CS of AIDA3 and 4
      Stats have been running for a while (Believe since 5am)
      Looking at the counter ratio - attachment 4
      We still have a deadtime of around 14% in FEE4 (Assuming FEE3 has no dead time)

01:56 Noticed /media/ThirdDrive was at 97% full
      Let DESPEC locals know I would like to use the next run stop to change the directory.
      They said they could do it now so they stopped their run.
      Changed the symbolic link to /media/1e2....
      Now writing to R4_

03:26 MBS DAQ Stopped as beam stopped
03:33 Beam back now and DAQ running again

04:44 System wide checks as before
      Statistics ok - attachment 5
      Temperatures ok - attachment - 7
      Bias and leakage currents ok - attachment 6

05:53 MBS stops run as beam drops
06:00 MBS started a new run as beam is back

07:37 System wide checks clock ok and FPGA as before
      WR additional difference on FEE5
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc0 : 	 8  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics ok - attachment 8
     Temp ok - attachment 9
     Bias and leakage ok - attachment 10
 

Wide checks at Time 22:22 CET

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dbc :      4  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

             Base         Current         Difference
aida05 fault      0x0 :      0x2 :      2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     35     11      3      5      5      4      1      3      2      4      6   : 37876
aida02 :     20      5      4      3      2      5      2      4      3      3      6   : 37400
aida03 :     20      3      4      3      4      4      1      3      2      4      6   : 37640
aida04 :     25      6      2      3      4      4      3      3      3      3      6   : 37140
aida05 :     30     10      5      7      2      6      2      4      2      4      6   : 38776
aida06 :     21     11      4      4      3      5      2      4      3      3      6   : 37548
aida07 :     23      8      4      4      2      6      1      3      2      4      6   : 37852
aida08 :     27      7      2      3      4      3      2      4      3      3      6   : 37284
aida09 :     31      6      5      6      3      3      2      4      3      3      6   : 37372
aida10 :     22      5      3      4      3      6      2      2      2      4      6   : 37616
aida11 :     23      5      4      2      5      2      1      3      2      4      6   : 37444
aida12 :     20      7      4      7      3      5      2      3      3      3      6   : 37096
aida13 :     28      5      3      3      1      5      2      3      3      3      6   : 36840
aida14 :     38     12      2      4      2      4      1      4      3      3      6   : 37144
aida15 :     27      9      5      4      2      5      3      2      3      3      6   : 36740
aida16 :     21      6      4      4      1      4      2      3      3      3      6   : 36740

At 20:15, We lost connection with AnyDesk.

Liliana restarted Anydesk on the AIDA DAQ PC. All seemed fine on AIDA DAQ but there was a more generic issue on the main DAQ, possibly the timesorter.

Helena restarted all after ~30mn and it's all working again now.

19:08 System Check

Rates from ucesb below 1 kHz

Atachment 1 Stats

Atachment 2 Temperatures

Atachment 3 Voltages

System wide checks are all ok except

ADC calibration all failed  
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit  
         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dba :      2  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA all passed

Memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6      2      2      1      2      4      3      4      2      3      7   : 40424
aida02 :      3      4      2      2      1      2      1      3      3      3      7   : 40140
aida03 :     10     10      0      3      2      4      2      4      2      3      7   : 40280
aida04 :      3      2      0      2      2      3      2      3      1      4      7   : 40540
aida05 :      4      9      2      3      2      4      3      3      1      4      7   : 41048
aida06 :      8      6      1      1      2      3      2      2      3      3      7   : 40064
aida07 :      1      4      0      1      1      3      1      3      3      3      7   : 40196
aida08 :      1      4      2      2      2      4      3      4      2      3      7   : 40452
aida09 :      8      5      2      2      1      2      1      3      3      3      7   : 40168
aida10 :     22      5      4      2      1      3      1      4      2      3      7   : 39872
aida11 :      2      1      2      1      2      2      3      3      2      3      7   : 39632
aida12 :      8      5      2      2      1      3      1      4      2      3      7   : 39784
aida13 :     18      8      2      1      0      2      3      3      2      3      7   : 39624
aida14 :     23     12      2      2      1      2      2      4      2      3      7   : 40028
aida15 :      3      5      3      2      2      3      1      4      2      3      7   : 39844
aida16 :      0      2      1      2      1      4      2      4      2      3      7   : 40096

08:10 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9db9 : 	 1  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

08:16 attempt to recalibrate ADCs using FADC align and control - unsuccessful on all

08:18 FEE64 temps ok - attachment 1

      bias/leakage currents ok - attachment 2

      statistics - attachments 3-8

10:13 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9db9 : 	 1  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

10:16 FEE64 temps ok - attachment 9

      bias/leakage currents ok - attachment 10

      statistics - attachments 11-16

12:16 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dba : 	 2  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

12:16 FEE64 temps ok - attachment 17

      bias/leakage currents ok - attachment 18

      statistics - attachments 19-25

14:08 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dba : 	 2  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FEE64 temps ok - attachment 16

      bias/leakage currents ok - attachment 27

      statistics - attachments 28-35

01:50 Beam on
The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing.
Tempetratures OK

Rates OK

Attachment 1 Voltages

Atachment 2 Rates

Atachment 3 Temperatures

Attachment 4 ucesb rates less than 1 kHz


System wide check are ok except:

Base Current Difference
aida05 fault  0x36ca :  0x36ce :  4 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base Current Difference
aida13 fault 0xa :  0xe9 :  223 
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760
aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644
aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188
aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404
aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104
aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128
aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500
aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672
aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588
aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460
aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348
aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000
aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840
aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996
aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280
aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716



<strong>05:00 OH Took over at 04:00</strong>

System wide checks ok except

   Base   Current  Difference
aida05 fault   0x36ca :   0x36d0 :   6 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


    Base   Current   Difference
aida05 fault   0x0 :   0x1 :   1 
aida13 fault   0xa :   0xe9 :   223 
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



05:41 AIDA 05 has dropped out. Will restart the system

05:58 DAQ recovered from powercycle
 p+n strips at 0xa
 n+n strips at 0x20
 ADC control register at 0xFF
 All waveforms turned off in LED

 Statistics - attachment 5
 Correlation scaler in FEE 3 and 4 are mostly tracking 1:1

06:34 Local users have been unable to recover MBS
      And have been unable to contact local experts

07:22 Bias and leakage currents ok - attachment 6
 Stats attachment -7

19:51 Shift change

Stats ok
ucesb ok, Max implants ~1500 Hz 

20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)

20:51 
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.

20:59 

Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36ce : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0xb6 : 	 172  
FPGA Timestamp error counter test result: Passed 15, Failed 1

--- both errors above same module as earlier shift
	 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      3      2      0      1      3      3      2      3      3      6   : 36092
aida02 :      6      4      4      2      2      3      1      3      3      3      6   : 36280
aida03 :      6      3      2      0      0      3      3      2      3      3      6   : 36048
aida04 :     10      3      1      1      3      3      1      3      3      3      6   : 36272
aida05 :     19      8      2      0      2      3      1      3      3      3      6   : 36268
aida06 :      2      2      0      2      3      4      2      2      3      3      6   : 36120
aida07 :      6      7      3      0      3      4      3      2      3      3      6   : 36416
aida08 :      7      5      4      0      3      2      2      3      3      3      6   : 36420
aida09 :     21      6      3      2      2      4      2      3      2      3      6   : 35700
aida10 :     13      3      1      1      3      3      1      4      2      3      6   : 35772
aida11 :     25      6      4      2      1      4      2      4      2      3      6   : 36180
aida12 :      0      4      0      3      2      3      3      3      2      3      6   : 35712
aida13 :     18      5      2      3      2      3      2      4      2      3      6   : 36080
aida14 :      2      1      1      1      2      3      2      4      2      3      6   : 35904
aida15 :      0      1      1      3      0      4      2      2      3      3      6   : 35960
aida16 :      4      5      3      2      3      4      1      3      2      3      6   : 35432

ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4

21:36
ucesb ok
Stats ok 

22:01
ucesb ok 1500 Hz peak implant
Stats ok 
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.

22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler

23:40 FRS magnet issue, possibly power, no beam

00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah

16:00 Shift change

Statistics OK (attach 1)
Temps OK (attach 2)
Bias OK (attach 3)

System wide checks
Clock, ADC - OK

WR
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36cb : 	 1  
White Rabbit error counter test result: Passed 15, Failed 1


FPGA

			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0x4b : 	 65  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      5      6      1      1      2      2      3      3      3      6   : 36388
aida02 :     21     10      2      2      2      3      2      2      3      3      6   : 36100
aida03 :     25      5      3      1      0      3      3      2      3      3      6   : 36188
aida04 :      5      2      0      3      2      3      2      3      3      3      6   : 36484
aida05 :     16      6      3      1      2      4      1      3      2      4      6   : 37440
aida06 :      2      2      2      2      2      3      1      3      3      3      6   : 36216
aida07 :     16      9      3      0      3      4      3      2      3      3      6   : 36472
aida08 :      0      7      1      1      2      2      3      3      2      3      6   : 35560
aida09 :     15      7      4      2      2      4      2      3      2      3      6   : 35700
aida10 :      3      7      3      4      1      3      1      4      1      4      6   : 36788
aida11 :      5      1      3      1      1      4      3      3      2      3      6   : 35756
aida12 :      5      5      0      3      2      3      2      4      2      3      6   : 35996
aida13 :      2      5      1      2      2      4      3      3      2      3      6   : 35840
aida14 :     19      5      3      1      1      2      1      3      3      3      6   : 36100
aida15 :     20      6      6      2      1      3      2      2      3      3      6   : 36064
aida16 :      7      9      4      1      3      4      2      3      2      3      6   : 35716

File size change
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 10 16:47:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


16:20 Beam is back. Currently on File R1_200
16:43 Beam is being optimised.

17:00 UCESB OK
Stats OK (attach 4)
Temps OK
Bias OK (attach 5)
System-wide checks as before

17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK

17:52 Started compressing R1_1*

18:00 UCESB ok (attach 5)
Stats OK (attach 6)
Temps OK
Bias OK (attach 7)
System wide checks as before

19:00 UCESB ok (as before)
Stats OK (attach 8)
Temps OK
Bias OK (attach 9)
Merger OK (attach 10)

19:32 UCESB ok
Stats OK (attach 11)
Temps OK (attach 12)
Bias OK (attach 13)
System wide checks as before

19:51 Shift handed over to James S

08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.

      They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
      They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID

08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2

09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
      Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
      Data rate 16.3MB/s
      ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.

09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage

09:47 Beam back turned off no storage. On file R1_10

11:04 Now that we have implants on the detector we get a better look at the strips
      From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
      This adaptor board corresponds to FEE11

12:17 Leakage currents screenshot 512.png

13:30 Beam off. Expected back at 15:00...

14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
      They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
      They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
      Rates are as before. No better, no worse.

15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...

03:12 System Check

attachment 1 : Current

attachment 2 : Temperature

attachment 3 : Rates

   
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

   
Calibration test result: Passed 16, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

   
             Base         Current         Difference
aida13 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      5      7      2      1      2      2      3      3      3      6   : 36464
aida02 :      9      8      3      3      1      4      1      2      4      3      6   : 36916
aida03 :      5      2      6      2      0      3      2      3      3      3      6   : 36420
aida04 :      6      5      2      3      4      4      2      3      3      3      6   : 36800
aida05 :     17      6      6      2      2      4      1      3      2      4      6   : 37524
aida06 :      7     12      3      4      3      3      1      3      3      3      6   : 36460
aida07 :     17     11      5      1      3      3      3      2      3      3      6   : 36428
aida08 :      3      5      0      1      4      2      2      4      2      3      6   : 35924
aida09 :     27      6      4      2      0      2      2      2      3      3      6   : 35868
aida10 :     16     11      8      0      2      2      1      3      2      4      6   : 37272
aida11 :     15      2      2      3      1      4      3      4      2      3      6   : 36364
aida12 :      1      6      4      3      1      3      2      4      2      3      6   : 35988
aida13 :     22     14     10      2      4      2      2      4      2      3      6   : 36264
aida14 :     26     10      4      3      2      1      1      3      3      3      6   : 36184
aida15 :     14      2      3      2      2      4      1      2      3      3      6   : 35896
aida16 :      7      5      4      0      2      4      2      3      2      3      6   : 35588

06:10 DSSD1 rate high!

Attached 4

Called OH and he woke up to fix it :)

It was a problem with one of the ASIC according to what he says:

One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35

07:03 System check

attachment 5 Spectrum rate

attachment 6 Voltages

attachment 7 Rates

attachment 8 Temperature

System wide clock are all ok except

White rabbit

 Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA

 Base         Current         Difference
aida13 fault      0xa :      0x14 :      10  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

08:20 System wide checks all ok except
		 Base 		Current 	Difference
aida01 fault 	 0xc10c : 	 0xc10e : 	 2  
aida02 fault 	 0xbe2a : 	 0xbe2c : 	 2  
aida03 fault 	 0xf665 : 	 0xf667 : 	 2  
aida04 fault 	 0xeabd : 	 0xeabf : 	 2  
aida05 fault 	 0x36c8 : 	 0x36ca : 	 2  
aida06 fault 	 0x2ea2 : 	 0x2ea4 : 	 2  
aida07 fault 	 0x9186 : 	 0x9188 : 	 2  
aida08 fault 	 0x5443 : 	 0x5445 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

and

	
		 Base 		Current 	Difference
aida01 fault 	 0xc10c : 	 0xc10e : 	 2  
aida02 fault 	 0xbe2a : 	 0xbe2c : 	 2  
aida03 fault 	 0xf665 : 	 0xf667 : 	 2  
aida04 fault 	 0xeabd : 	 0xeabf : 	 2  
aida05 fault 	 0x36c8 : 	 0x36ca : 	 2  
aida06 fault 	 0x2ea2 : 	 0x2ea4 : 	 2  
aida07 fault 	 0x9186 : 	 0x9188 : 	 2  
aida08 fault 	 0x5443 : 	 0x5445 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    Statistics still high - attachment 1
    Temperatures ok - attachment 2
    Bias and leakage currents ok - attachment 3

Testing the behaviour of powering down the fast ADCs in aida01 (the hot FEE)

Set local control 2 to 0xff in aida01
Wait... see temperature drpos by ~3 degrees (PJCS reports much larger drop in T9)
So far so normal

*but*

aida05 rates go very high when this waveform is disabled!!
Set local control to 0x00 again
They drop back...

Recalibrated FADCs and back to normal

09.51 All system wide checks OK *except*


		 Base 		Current 	Difference
aida05 fault 	 0x8441 : 	 0x8442 : 	 1  
aida06 fault 	 0xc5c5 : 	 0xc5c6 : 	 1  
aida07 fault 	 0x6f07 : 	 0x6f08 : 	 1  
aida08 fault 	 0x4814 : 	 0x4815 : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      6      3      5      5      2      3      2      3      6   : 36004
aida02 :      4      1      3      6      3      4      3      3      2      3      6   : 36040
aida03 :     12      8      5      2      3      3      3      4      2      3      6   : 36416
aida04 :      1      2      5      4      1      4      3      4      2      3      6   : 36388
aida05 :     22      8      4      2      4      5      1      4      1      4      6   : 37272
aida06 :      5      4      0      6      3      6      3      3      2      3      6   : 36276
aida07 :     38     16      7      7      1      6      1      3      3      3      6   : 37032
aida08 :     16      4      2      3      1      4      1      3      3      3      6   : 36384
aida09 :      2      3      7      6      5      5      2      3      2      3      6   : 36112
aida10 :     42     15      4      6      2      6      2      4      2      3      6   : 36768
aida11 :      1      5     13      7      3      3      3      3      2      3      6   : 36124
aida12 :     23      9      3      4      7      3      2      3      2      3      6   : 35988
aida13 :      2      6      7      3      4      4      3      2      2      3      6   : 35592
aida14 :     17     11      3      4      3      5      2      3      2      3      6   : 35980
aida15 :      4      7      5      2      1      6      2      3      2      3      6   : 35864
aida16 :      4      1      3      3      4      2      3      4      2      3      6   : 36264


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 09 15:10:52 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.52 All histograms, stats and merger stats zero'd

      FEE64 temps OK - attachment 1
      ADC data, disc/WR 28-27/pause/correlation stats OK - attachments 2-6
      Rate spectra - attachment 7
       ASIC check load
      Rate spectra - attachment 8
      1.8.W spectra 20us FSR - attachments 9 & 10
      Grafana - most recent 7 days DSSSD bias and leakage current - attachment 11
      Merger stats/tape server/merger - attachment 12
      DSSSD bias and leakage current OK - attachment 13

10.08 1.8.W spectra 20us, 200us, 2ms & 20ms FSR - attachments 14-21
       we are not missing extrinsic noise features at longer time ranges cf. 20us FSR

12.18 Cannot connect to port 8015, MIDAS terminal says "Killed"
      Restart MIDAS 8015 
      Collect FPGA baseline to avoid TCL error
      FEEs unaffected and running as before

13.25 FEEs power cycled to see if any change in weird noise/rates situation
      No change 
      ASICs synchronised and back to normal

14.34 Start a pulser walkthrough in May21/R10
      Attenuation 5x, Rep Rate 25 Hz
      Amplitudes 10..9...1

14.45 Back to 2 Hz @ 2V Pulser, walkthrough complete. No Storage again.

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0 (NB. actually it is 12 MeV)
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK

09.54 FEE64 temperatures OK - attachment 1
      ADC data item stats OK - attachment 2
       p+n FEE64s 11x <100k, 1x <200k
      Rate spectra - attachments 3 & 4
       before and after ASIC check load
      1.8.L spectra - attachments 5 & 6
      1.8.W spectra 20us FSR - attachments 7 & 8
      Disc, WR-28-47, pause, resume info data items - attachments 9-12
       note large number pause/resume for aida13 - legacy numbers since last restart/zero?
       attachment 16 for pause stats since stats zero at 10.02 - looks OK
      Grafana DSSSD bias & leakage currents for most recent 48h OK - atatchment 13
      DSSSD bias & leakage currents - attachment 14
      Merger stats/merger/tape server messages - attachment 15

10.02 All histograms and stats zero'd
      
10.06 All system wide checks OK *except*

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 06 15:29:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.25 Pulser peak width aida01 97 ch FWHM, aida02 350 ch FWHM
       peak widths are increasing cf. earlier this week - see https://elog.ph.ed.ac.uk/DESPEC/291 , https://elog.ph.ed.ac.uk/DESPEC/298

After tests in the Daresbury T9 system.

Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees.

To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF

The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected.

Alternatively STOP acquisition, set the ADC Control  register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.

14:11 - Alpha has been running most of morning

Just saw rates in tape spike to 6 MB/s... 
Stop output to tape and look:
ASIC check & load... all rates except aida04 back to 0
Start output to tape
Back to ~300 KB/s

R6_49 will be affected by this.
Others seem OK

System wide check failures:

		 Base 		Current 	Difference
aida01 fault 	 0xb405 : 	 0xb406 : 	 1  
aida02 fault 	 0xefc7 : 	 0xefc8 : 	 1  
aida03 fault 	 0xdaab : 	 0xdaac : 	 1  
aida04 fault 	 0x8f7c : 	 0x8f7d : 	 1  
aida05 fault 	 0xb5bd : 	 0xb5be : 	 1  
aida06 fault 	 0xeff1 : 	 0xeff2 : 	 1  
aida07 fault 	 0x8f57 : 	 0x8f58 : 	 1  
aida08 fault 	 0xbef7 : 	 0xbef8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x200 : 	 512  
aida13 fault 	 0x0 : 	 0x61 : 	 97  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page)

Temps & Rates OK

Will now stop Alpha run and check noise situation

Run stopped at R6_49

Switching to nostorage (NOTAPE/R2)

Set ASICs to threshold 0xa, Rates OK
Rate to Tape/MBS ca 9 MB/s

No change to situation since yesterday... good sign

During MBS test aida13 rebooted... crash log from ttyUSB5 attached

It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again...
cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :)

A power cycle of all FEEs was performed after aida13's reboot to make sure things are good.
All systems check OK

18.09 NH has installed jumpers LK2-4 for *all* adaptor PCBs

      per FEE64 rate spectra - attachments 1-4
       attachment 4 with BNC PB-5 pulser 10Hz
      ADC data items stats - attachment 5
      1.8.W spectra - attachments 6 & 7

      Observe significant reduction in DSSSD/ribbon cable boundary channel rates
       - status quo ante Tuesday 4 May https://elog.ph.ed.ac.uk/DESPEC/291 ?


      1.8.L spectra - attachments 8 & 9
       pulser peak width aida01 85 ch FWHM, aida2 295 ch FWHM

19:21 Alpha run started May21/R6
      Statistics dominated by aida04 with a rate of 6k
      System wide checks all ok except:
			 Base 		Current 		Difference
aida13 fault 	 0x0 : 	 0x17 : 	 23  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Writing to file at ~300kB/s
     Rate spectra for aida04 shows dominated by first asic - attachment 10
     1*L spectra for aida04 asic 1 - attachment 11

20.25 per FEE64 rate spectra - attachment 12
      handful of hot channels aida06, 09 & 10
      ADC data item stats - attachment 13

21:10 After all of the works in S4 today the noise in the system was considerably worse
      It was decided that before starting a new alpha run a powercycle would be performed.
      The powercycle has had no effect on the rates. They are still terrible.

      The difference between yesterday and today is:
      - Adaptor cards for FEE9 and 13 were re-aligned as they were previously misaligned
      - bPlast has been plugged in and biased

      All system wide checks ok
      Statistics threshold at 0xa - attachment 1
      Rate spectra (FEE5 does have events in all channels just noise is dominated by a few) - attachment 2
      Waveforms - attachments 3 and 4

      Pulser turned off and threshold raised to 0x64
      Statistics - attachment 5

21:19 Alpha run started to May21/R4*


08:51 DAQ continues OK - file May21/R4_23

      FEE64 temps OK - attachment 8
      ADC data items OK - attachment 9 
      1.8.W spectra - attachments 6 & 7
       no change cf. yesterday pm
      Grafana - most recent 12h DSSSD bias & leakage current - attachment 10
      DSSSD biases & leakage currents - attachment 11
      Tape Server/Merger - attachment 12

 09:44 R4 stopped

17.55 Grafana DSSSD bias & leakage current - attachment 1
       Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37 
      DSSSD bias & leakage currents - attachment 2

22:26 Tape server set up to write to directory May21
      DAQ stopped
      Slow comparator set to 0x64 for all FEE64 and an ASIC check performed
      Pulser turned off
      DAQ started
      ~300k in merger - 38kB per second to file
      Statistics good (all FEE show 0 bar FEE4 at 3k) - attachment 1
      All system wide checks ok apart from FPGA check
      	
	        Base	Current  	Difference
aida13 fault 	 0x0 : 	 0x62 : 	 98  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Histograms zeroed.
     DSSD1 at 130V
     DSSD2 at 100V

08:47 Found DAQ running at c. 3M data items/s / 13MB/s
      Switched Tape Server to no storage mode
      
[npg@aidas-gsi ~]$ ls -lrth /TapeData/May21
total 113G
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_0
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_1
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_2
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_3
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_4
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_5
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_6
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_7
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_8
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_9
-rw-rw-r--. 1 npg npg 1.3G May  5 06:15 R1_10
-rw-rw-r--. 1 npg npg 2.0G May  5 06:17 R1_11
-rw-rw-r--. 1 npg npg 2.0G May  5 06:20 R1_12
-rw-rw-r--. 1 npg npg 2.0G May  5 06:23 R1_13
-rw-rw-r--. 1 npg npg 2.0G May  5 06:25 R1_14
-rw-rw-r--. 1 npg npg 2.0G May  5 06:28 R1_15
-rw-rw-r--. 1 npg npg 2.0G May  5 06:30 R1_16
-rw-rw-r--. 1 npg npg 2.0G May  5 06:33 R1_17
-rw-rw-r--. 1 npg npg 2.0G May  5 06:36 R1_18
-rw-rw-r--. 1 npg npg 2.0G May  5 06:38 R1_19
-rw-rw-r--. 1 npg npg 2.0G May  5 06:41 R1_20
-rw-rw-r--. 1 npg npg 2.0G May  5 06:43 R1_21
-rw-rw-r--. 1 npg npg 2.0G May  5 06:46 R1_22
-rw-rw-r--. 1 npg npg 2.0G May  5 06:49 R1_23
-rw-rw-r--. 1 npg npg 2.0G May  5 06:51 R1_24
-rw-rw-r--. 1 npg npg 2.0G May  5 06:54 R1_25
-rw-rw-r--. 1 npg npg 2.0G May  5 06:57 R1_26
-rw-rw-r--. 1 npg npg 2.0G May  5 06:59 R1_27
-rw-rw-r--. 1 npg npg 2.0G May  5 07:02 R1_28
-rw-rw-r--. 1 npg npg 2.0G May  5 07:04 R1_29
-rw-rw-r--. 1 npg npg 2.0G May  5 07:07 R1_30
-rw-rw-r--. 1 npg npg 2.0G May  5 07:10 R1_31
-rw-rw-r--. 1 npg npg 2.0G May  5 07:12 R1_32
-rw-rw-r--. 1 npg npg 2.0G May  5 07:15 R1_33
-rw-rw-r--. 1 npg npg 2.0G May  5 07:18 R1_34
-rw-rw-r--. 1 npg npg 2.0G May  5 07:20 R1_35
-rw-rw-r--. 1 npg npg 2.0G May  5 07:23 R1_36
-rw-rw-r--. 1 npg npg 2.0G May  5 07:25 R1_37
-rw-rw-r--. 1 npg npg 2.0G May  5 07:28 R1_38
-rw-rw-r--. 1 npg npg 2.0G May  5 07:31 R1_39
-rw-rw-r--. 1 npg npg 2.0G May  5 07:33 R1_40
-rw-rw-r--. 1 npg npg 2.0G May  5 07:36 R1_41
-rw-rw-r--. 1 npg npg 2.0G May  5 07:39 R1_42
-rw-rw-r--. 1 npg npg 2.0G May  5 07:41 R1_43
-rw-rw-r--. 1 npg npg 2.0G May  5 07:44 R1_44
-rw-rw-r--. 1 npg npg 2.0G May  5 07:46 R1_45
-rw-rw-r--. 1 npg npg 2.0G May  5 07:49 R1_46
-rw-rw-r--. 1 npg npg 2.0G May  5 07:52 R1_47
-rw-rw-r--. 1 npg npg 2.0G May  5 07:54 R1_48
-rw-rw-r--. 1 npg npg 2.0G May  5 07:57 R1_49
-rw-rw-r--. 1 npg npg 2.0G May  5 08:00 R1_50
-rw-rw-r--. 1 npg npg 2.0G May  5 08:02 R1_51
-rw-rw-r--. 1 npg npg 2.0G May  5 08:05 R1_52
-rw-rw-r--. 1 npg npg 2.0G May  5 08:07 R1_53
-rw-rw-r--. 1 npg npg 2.0G May  5 08:10 R1_54
-rw-rw-r--. 1 npg npg 2.0G May  5 08:13 R1_55
-rw-rw-r--. 1 npg npg 2.0G May  5 08:15 R1_56
-rw-rw-r--. 1 npg npg 2.0G May  5 08:18 R1_57
-rw-rw-r--. 1 npg npg 2.0G May  5 08:20 R1_58
-rw-rw-r--. 1 npg npg 2.0G May  5 08:23 R1_59
-rw-rw-r--. 1 npg npg 2.0G May  5 08:26 R1_60
-rw-rw-r--. 1 npg npg 2.0G May  5 08:28 R1_61
-rw-rw-r--. 1 npg npg 2.0G May  5 08:31 R1_62
-rw-rw-r--. 1 npg npg 2.0G May  5 08:34 R1_63
-rw-rw-r--. 1 npg npg 2.0G May  5 08:36 R1_64
-rw-rw-r--. 1 npg npg 2.0G May  5 08:39 R1_65
-rw-rw-r--. 1 npg npg 2.0G May  5 08:41 R1_66
-rw-rw-r--. 1 npg npg 1.6G May  5 08:44 R1_67
[npg@aidas-gsi ~]$ 

      Rate increased c. 06.15 
       multiple ASICs in multiple FEE64s hot HEC channels
      ASIC check load restores rates to those at beginning of May21/R1
      Switched Tape Server to storage mode - file May21/R1_67

08.53 All system wide checks OK *except* 

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida08 fault 	 0x0 : 	 0x1 : 	 1  
aida13 fault 	 0x62 : 	 0x9380 : 	 37662  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Most recent merger errors

MERGE Data Link (2497): bad timestamp  7 3 0xd1fc813c 0x073033d6 0x00001083773033d6 0x167c1083773033d6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ce7ff9 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1dd806e 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ed7fc1 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1fd803c 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1cf8066 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1de8110 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ee8086 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1fe7ff7 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1c38041 0x0e60a6d6 0x000010832e60a6d6 0x167c10832e60a6d6 0x167c108377304b46

09.04 FEE64 temps OK - attachment 2
      ADC data item stats - attachment 3
      1.8.L spectra - attachments 4 & 5
      1.8.W spectra - attachments 6 & 7
      Grafana DSSSD bias & Leakage current most recent 24h - attachment 8
      DSSSD bias & leakage current - attacment 9

11:32 R1 stopped to realign boards and check situation in S4

A figure showing the FEE64 numbers (a previous diagram had 2/6 and 4/8 on the wrong side of the DSSD) and the wiring of pulser/HV
Also the jumpers connected to the boards

As always diagram is looking with direction of beam

Edit: Figure as signals (HV/Pulser) come from the beam right not beam left
Otherwise OK

During the installation of the 4 new HDMI cables for aidas13-16 the HDMI cables between the MACB Root and the MACB leaves were looped through the cable guide at the top to support them more
This may improve their connection and reduce the chance of WR errors seen in S460/S452
The connectors were all reseated a few times to clean the contacts too

NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290

14.40 Rate spectra - attachment 1
      adc data items - attachment 2
       slow comparator 0xa
       stats for p+n much improved, n+n remain poor
      1.8.W spectra - attachments 3 & 4
      1.8.L spectra - attachments 5 & 6
       pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM

17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7

13:11 System wide checks all ok except *aida02 fails ADC calibration*
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue May 04 10:57:38 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

ASIC clocks have been synchronised and ADC re-calibrated

Rate spectra - attachment 1
Layout7 - attachment 2
Layout8 - attachment 3

Detectors at 130V

Statistics - attachment 4

V-I Curve - attachment 5
Voltage	Ch0	Ch1
10	1.345	1.49
20	2.01	2.285
30	2.405	2.8
40	2.635	3.13
50	2.82	3.37
60	2.95	3.57
70	3.08	3.725
80	3.185	3.865
90	3.26	3.98
100	3.33	4.105
110	3.42	4.22
120	3.525	4.35
130	3.64	4.46
140	3.79	4.37
150	3.98	4.37
150+10min	3.9	4.485

Photos provided by Nic Hubbard and Helena Albers from the assembly of 2x triPlast
plus 2x MSL type BB18(DS)-1000 24cm x 8cm detector assembly - attachments 1-3.

Spacers installed between MSL type BB18(DS)-1000 24cm x 8cm PCBs - attachment 4.

AIDA support stand in 24cm x 8cm DSSSD mode ready for installation of snout assembly - attachments 5 & 6

AIDA 24cm x 8cm DSSSD plus 2x TriPlast installed in S4 - view downstream towards upstream window 
of AIDA detctor assembly - attachment 7

AIDA 24cm x 8cm DSSSD plus 2x TriPlast snout assembly installed at S4 - attachment 8

Side view of AIDA + triPLast + FATIMA + DEGAS - attachment 9

09.40 All system wide checks OK *except*

FEE64 module aida07 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Thu Apr 29 17:50:06 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.44 DSSSD bias & leakage currents Ok - attachment 1
      Merger time seq errors - attachment 2
       no merger errors since restart yesterday
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      Merger/Tape Server server consoles - attachment 5
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 6

      DAQ continues OK - no data storage

New set of ASIC settings produced 2021Apr29-13-16-00
- Note no longer odd/even for polarity of signal

New set of Options file for FEEs - Points to the new ASIC settings

New layouts for histograms grouped by side of the detector. Heavily weighted to the front side in terms of number of his

/MIDAS/Linux/startup/NewMerger Updated line from "./master64 -i 12 -l 16 -p 11001  &" to "./master64 -i 16 -l 16 -p 11001  &"

/MIDAS/DB/EXPERIMENTS/MERGE/Options/aidas-gsi/CONTENTS updated the number of links to 16.

System wide checks passed *except aida03 ADC calibration*

System left running to no storage.

Updated /etc/dhcpd.conf

cd /etc
cp dhcpd.conf dhcpd.conf.BAK-280421

Add MAC addresses https://elog.ph.ed.ac.uk/DESPEC/285

aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed

New version of dhcpd.conf (and backup) - attachments 1 & 2

Cannot find dhcpd (and therefore re-read config and restart) in System - Administration -> Services
- has the dhcpd service been re-named or amalgamated with another service?



Updated /MIDAS/config/TclHttpd/aidas-gsi@8015/startup.tcl 

New version of startup.tcl (and backup) - attachments 3 & 4



Created rfs filesytems for aida13-aida24

[root@aidas-gsi]# cd /home/Embedded/XilinxLinux/ppc_4xx/rfs
[root@aidas-gsi rfs]# cd aida01
[root@aidas-gsi rfs]# tar cvf ../rfs.tar .

Create directories aida13-aida24, uid=10101, gid=npg, mode 777

[root@aidas-gsi rfs]# cd aida13; tar xvf ../rfs.tar .
 :
 :
etc

[root@aidas-gsi rfs]# ls -l 
total 1491880
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida01
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida02
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida03
drwxrwxrwx. 19 10101 npg        4096 Apr 24 03:51 aida04
drwxrwxrwx. 19 10101 npg        4096 Apr 24 03:51 aida05
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida06
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida07
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida08
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:35 aida09
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida10
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:34 aida11
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida12
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida13
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida14
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida15
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida16
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida17
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida18
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida19
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida20
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida21
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida22
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida23
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida24
-rw-r--r--.  1 root  root 1527582720 Apr 28 18:26 rfs.tar



Once dhcpd has been restarted it should be possible to power cycle all FEE64s, start system
and update firmware of newly installed FEE64s.

11.05 All system wide checks OK *except*
WR
	
		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc971 : 	 17  
aida02 fault 	 0x8bbf : 	 0x8bd0 : 	 17  
aida03 fault 	 0xae9e : 	 0xaeaf : 	 17  
aida04 fault 	 0x6e3 : 	 0x6f4 : 	 17  
aida05 fault 	 0x31a4 : 	 0x31bd : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x2303 : 	 25  
aida07 fault 	 0x438f : 	 0x43a8 : 	 25  
aida08 fault 	 0x61ed : 	 0x6206 : 	 25  
aida09 fault 	 0x1c32 : 	 0x1c40 : 	 14  
aida10 fault 	 0x9332 : 	 0x933f : 	 13  
aida11 fault 	 0x227b : 	 0x2288 : 	 13  
aida12 fault 	 0xb2fc : 	 0xb309 : 	 13  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA
	
			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x1 : 	 1  
aida02 fault 	 0x0 : 	 0x3 : 	 3  
aida03 fault 	 0x0 : 	 0x1 : 	 1  
aida04 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida08 fault 	 0x0 : 	 0x1 : 	 1  
aida11 fault 	 0x0 : 	 0x3 : 	 3  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 4, Failed 8
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Looks like a big WR glitch happened at some point (master in issue?)

Memory
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26     20     26     15      3      1      3      2      2      3      4   : 27848
aida02 :     25      7      1      4      1      4      2      2      3      3      6   : 36204
aida03 :     30      5      3      3      3      3      1      3      3      3      6   : 36464
aida04 :      5     10      0      5      2      3      1      3      3      3      6   : 36356
aida05 :     34     40     16     13      4      3      3      2      3      3      4   : 29160
aida06 :     16      8      4      2      1      4      1      3      3      3      6   : 36416
aida07 :     28     23     23     10      4      3      2      4      3      4      3   : 27736
aida08 :     26      9      3      3      1      3      1      3      3      3      6   : 36352
aida09 :     54     42     10     17      3      4      3      4      3      3      4   : 30376
aida10 :     17      5      7      4      3      3      3      3      2      3      6   : 35996
aida11 :     19     10     11      2      1      1      2      4      2      3      6   : 35916
aida12 :     82     42     12      7      4      2      3      2      2      3      4   : 27960


Options unchanged from ELOG 284

DAQ continues OK but New Merger window has disappeared... clearly still running in background!

DSSSD leakage currents decreasing as expected

-

Plan later today:

Prepare AIDA Kapton Cables for triple

install 4 new FEE64s into S4 (13,14,15,16)

Proposed layout of FEE64s (perspective with beam) in fig 5

Cards 9-12 will be moved on Friday when triple is installed

When ready MAC addressed will be given to OH/TD who will do the PC configuration including:
- Network IP allocation
- MIDAS registration
- Firmware updating
- Checks 

17.20:

The four new FEEs have been installed and connected to Ethernet, Power, HDMI and Water
No front-end adapter is connected yet
The MAC addresss are as follows
aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed

09.25 All system wide checks OK *except*


		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc963 : 	 3  
aida02 fault 	 0x8bbf : 	 0x8bc2 : 	 3  
aida03 fault 	 0xae9e : 	 0xaea1 : 	 3  
aida04 fault 	 0x6e3 : 	 0x6e6 : 	 3  
aida05 fault 	 0x31a4 : 	 0x31ac : 	 8  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f2 : 	 8  
aida07 fault 	 0x438f : 	 0x4397 : 	 8  
aida08 fault 	 0x61ed : 	 0x61f5 : 	 8  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     18     24     16      3      1      3      2      2      3      4   : 27792
aida02 :     25      6      3      5      1      4      2      2      3      3      6   : 36260
aida03 :     19     10      3      6      3      3      1      3      3      3      6   : 36556
aida04 :     21      7      3      4      2      4      2      2      3      3      6   : 36284
aida05 :     40     37     16     13      4      3      2      2      3      3      4   : 28904
aida06 :     32      4      4      3      0      4      1      3      3      3      6   : 36416
aida07 :     26     22     18     13      3      3      2      4      3      4      3   : 27672
aida08 :     15      9      4      1      2      3      1      3      3      3      6   : 36324
aida09 :     29     41     11     19      3      4      3      4      3      3      4   : 30348
aida10 :     17      5      8      4      3      3      3      3      2      3      6   : 36012
aida11 :     25      9      3      5      1      1      2      4      2      3      6   : 35900
aida12 :     80     43     12      7      4      2      3      2      2      3      4   : 27960



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

09.25 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
       aida09 zero rate
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK *except* aida09 - no data storage

10.00 from aida09 system console

24/03:04:39|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
24/03:04:39|Also in RDOGo_Operate: Enabled waveform readout
24/03:04:39|
24/03:04:39|get_WAVEBlk (C) : which = 0 , Status = 00001031 , Count = 0
24/03:04:39|get_WAVEBlk (A) which = 0 ---- status = 0540, State machines = 050A
24/03:04:39|WV:0:Buffer info. Length = 131072 : Offset = 0 
24/03:04:40|WV:1:Buffer info. Length = 131070 : Offset = 2 
24/03:04:40|WV:2:Buffer info. Length = 131068 : Offset = 4 
24/03:04:40|WV:3:Buffer info. Length = 131066 : Offset = 6 
24/03:04:40|WV:4:Buffer info. Length = 131064 : Offset = 8 
24/03:04:40|WV:5:Buffer info. Length = 131062 : Offset = 10 
24/03:04:40|WV:6:Buffer info. Length = 131060 : Offset = 12 
24/03:04:40|WV:7:Buffer info. Length = 131058 : Offset = 14 
24/03:04:40|WV:8:Buffer info. Length = 131056 : Offset = 16 
24/03:04:40|WV:9:Buffer info. Length = 131054 : Offset = 18 
24/03:04:40|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:40|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 804441A5 : 1 => 09CD0000 
24/03:04:40|Last Data : Databuffer  274139128 : 0 => C01B7F88 : 1 => 0C05B135 
24/03:04:40| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:40| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:40| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:41| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:41| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:41| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:41| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:41| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:41|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C03A7F13 : 1 => 0C05B455 
24/03:04:41|Last Data : Databuffer  274139128 : 0 => C0127D9D : 1 => 0DB9DEED 
24/03:04:41| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:41| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:41| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:42| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:42| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:42| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:42| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:42| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:42|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:42|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C02D7E26 : 1 => 0DB9DEED 
24/03:04:42|Last Data : Databuffer  274139128 : 0 => C0217F73 : 1 => 0F816B3D 
24/03:04:42| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:42| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:42| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:43| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:43| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:43| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:43| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:43| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:43|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C02D7EB4 : 1 => 0F816D95 
24/03:04:43|Last Data : Databuffer  274139128 : 0 => C0167F53 : 1 => 0148F2DD 
24/03:04:43| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:43| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:43| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:44| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:44| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:44| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:44| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:44|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0207E17 : 1 => 0148F2DD 
24/03:04:44|Last Data : Databuffer  274139128 : 0 => C0177F46 : 1 => 047FF0CD 
24/03:04:44| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:44| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:44| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:45| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:45| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:45| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:45| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:47|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:56|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:05|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:05:15|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:24|197 spectra zeroed
25/10:21:10|do_GetState returned z=0 and 1
25/10:21:11|Data Acquisition Statistics counters now cleared
25/10:24:01|Clear Statistics (1)
25/10:24:01|bad event data (0x45000034 0x84e64000)
26/16:52:50|bad event data (0x40061e21 0xc0a80b63)
26/16:52:50|received undefined information code: 1
26/16:52:50|bad event data (0x0101080a 0xe9b68def)
26/16:52:50|bad event data (0x03527827 0x0101050a)
26/16:52:50|received undefined information code: 9

From system console

Apr 24 02:52:42 aida09 kernel: IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
Apr 24 02:52:42 aida09 kernel: IP-Config: Complete:
Apr 24 02:52:42 aida09 kernel:      device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
Apr 24 02:52:42 aida09 kernel:      host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
Apr 24 02:52:42 aida09 kernel:      bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100003/2 on 192.168.11.99
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100005/1 on 192.168.11.99
Apr 24 02:52:43 aida09 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Apr 24 02:52:43 aida09 kernel: Freeing unused kernel memory: 168k init
Apr 24 02:52:43 aida09 kernel: xaida: device parameters: base=0x81000000 size=0x200000
Apr 24 02:52:44 aida09 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Apr 24 02:52:44 aida09 kernel: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Apr 24 02:52:44 aida09 kernel: xaida: driver assigned major number 254
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service telnet, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service ftp, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Apr 24 02:52:44 aida09 xinetd[273]: Started working: 2 available services
Apr 24 02:52:48 aida09 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: driver assigned major number 253
Apr 24 02:53:04 aida09 kernel: xaida: open:
Apr 24 02:53:04 aida09 kernel: AIDAMEM: aidamem_open:
Apr 24 02:57:33 aida09 kernel: xaida: open:
Apr 24 13:39:36 aida09 xinetd[273]: START: telnet pid=396 from=192.168.11.99
Apr 24 13:42:05 aida09 xinetd[273]: EXIT: telnet status=0 pid=396 duration=149(sec)
Apr 27 09:01:50 aida09 xinetd[273]: START: telnet pid=422 from=192.168.11.99

10.42 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc962 : 	 2  
aida02 fault 	 0x8bbf : 	 0x8bc1 : 	 2  
aida03 fault 	 0xae9e : 	 0xaea0 : 	 2  
aida04 fault 	 0x6e3 : 	 0x6e5 : 	 2  
aida05 fault 	 0x31a4 : 	 0x31ab : 	 7  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f1 : 	 7  
aida07 fault 	 0x438f : 	 0x4396 : 	 7  
aida08 fault 	 0x61ed : 	 0x61f4 : 	 7  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25      8     28      8      3      2      2      3      2      3      4   : 27940
aida02 :     24      9      3      5      1      3      1      3      3      3      6   : 36408
aida03 :     24      6      5      4      3      3      1      3      3      3      6   : 36512
aida04 :     23      6      6      1      3      3      1      3      3      3      6   : 36428
aida05 :     15     21     27      7      4      2      2      3      3      3      4   : 29044
aida06 :     24      8      2      4      1      3      1      3      3      3      6   : 36352
aida07 :     29     24     28      9      3      3      2      2      2      3      4   : 27732
aida08 :     17     10      1      1      1      4      1      3      3      3      6   : 36356
aida09 :     41     49     18     14      3      4      3      4      2      3      4   : 29388
aida10 :      3      1      6      4      2      3      3      3      2      3      6   : 35828
aida11 :     22      7      5      4      2      1      2      4      2      3      6   : 35936
aida12 :     65     30     23      9      3      1      2      3      2      3      4   : 28100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021


10.47 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK - no data storage

11.25 All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0xc95f : 	 0xc960 : 	 1  
aida02 fault 	 0x8bbe : 	 0x8bbf : 	 1  
aida03 fault 	 0xae9d : 	 0xae9e : 	 1  
aida04 fault 	 0x6e2 : 	 0x6e3 : 	 1  
aida05 fault 	 0x31a2 : 	 0x31a4 : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22ea : 	 3  
aida07 fault 	 0x438d : 	 0x438f : 	 2  
aida08 fault 	 0x61eb : 	 0x61ed : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     55     21     31      6      3      2      2      3      2      3      4   : 28148
aida02 :     28      3      5      4      0      3      1      3      3      3      6   : 36312
aida03 :     25     10      2      2      2      4      1      3      3      3      6   : 36500
aida04 :     14      7     10      1      2      2      2      3      3      3      6   : 36528
aida05 :     58     35     28      6      4      2      2      3      3      3      4   : 29312
aida06 :     16     11      6      1      0      4      1      3      3      3      6   : 36376
aida07 :     75     31     25      9      4      3      2      2      2      3      4   : 27988
aida08 :     28      7      1      1      1      4      1      3      3      3      6   : 36376
aida09 :     50     48     28     10      3      3      3      4      2      3      4   : 29320
aida10 :     26      5      4      3      1      3      2      4      2      3      6   : 36080
aida11 :     21      7      5      3      5      2      2      4      2      3      6   : 36220
aida12 :     64     29     26      7      3      1      2      3      2      3      4   : 28072

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors reported - server terminal now reset & clear'd

MERGE Data Link (4408): bad timestamp  10 3 0xc2b67d8f 0x0b50c3e2 0x0000f437cb50c3e2 0x1678f437cb50c3e2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2877f27 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a07fcd 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77c9c 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc28b7d9c 0x0b50d382 0x0000f437cb50d382 0x1678f437cb50d382 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2937e6c 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a17d78 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2817fff 0x0b50eaf2 0x0000f437cb50eaf2 0x1678f437cb50eaf2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77ecf 0x06fb3d52 0x0000f437cb50eaf2 0x1678f437c6fb3d52 0x1678f43a16f1e6b2

11.25 Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 1
      DSSSD bias |& leakage currents Ok - attachment 2
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      merger time seq errors - attachment 5

      All histograms, stats and merge stats zero'd
      WR decoder errors *new*  baseline

11.45 1.8.L spectra - attachments 6 & 7
      peak width FWHM aida05 65 ch, aida06 95 ch
      1.8.W spectra = attachments 8 & 9

from ttyUSB5 system console log

19/07:25:59|page:c047b060 flags:(null) count:-1 mapcount:0 mapping:(null) index:0
19/07:25:59|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:00|Faulting instruction address: 0xc006e814
19/07:26:00|Oops: Kernel access of bad area, sig: 11 [#1]
19/07:26:00|PREEMPT Xilinx Virtex440
19/07:26:00|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:00|NIP: c006e814 LR: c006e78c CTR: c026327c
19/07:26:00|REGS: c60f3860 TRAP: 0300   Not tainted  (2.6.31)
19/07:26:00|MSR: 00021000 <ME,CE>  CR: 28222024  XER: 20000000
19/07:26:00|DEAR: 00100104, ESR: 00800000
19/07:26:00|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:00|GPR00: 00000000 c60f3910 c6b0a0a0 c038c3a8 00000000 000001a5 c047b078 00000041
19/07:26:00|GPR08: 00000001 00100100 00000004 c038c3a8 48282028 2015881c 00000001 00000000
19/07:26:00|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c60f2000
19/07:26:00|GPR24: 00000000 00000001 00029000 c038c384 00000001 c047b060 c038c39c c038c394
19/07:26:00|NIP [c006e814] get_page_from_freelist+0x188/0x4cc
19/07:26:00|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:00|Call Trace:
19/07:26:00|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:00|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:01|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:01|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:01|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:01|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:01|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:01|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:01|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:01|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:01|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:01|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:01|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:01|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:01|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:01|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:01|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:01|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:01|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:01|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:01|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:01|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:02|Instruction dump:
19/07:26:02|2f890000 419e0008 7c004a2c 38dd0018 7f861800 409effd8 7f861800 419e02e0
19/07:26:02|813d0018 81660004 7f3ccb78 912b0000 <91690004> 92dd0018 92a60004 813e0000
19/07:26:02|---[ end trace 87f7fdd1c04ff304 ]---
19/07:26:02|note: syslogd[224] exited with preempt_count 1
19/07:26:02|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:02|Faulting instruction address: 0xc006e388
19/07:26:02|Oops: Kernel access of bad area, sig: 11 [#2]
19/07:26:02|PREEMPT Xilinx Virtex440
19/07:26:02|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:02|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:02|REGS: c60f35d0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:02|MSR: 00021000 <ME,CE>  CR: 28222082  XER: 00000000
19/07:26:02|DEAR: 00100104, ESR: 00800000
19/07:26:02|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:02|GPR00: 00000000 c60f3680 c6b0a0a0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:02|GPR08: 00000008 c03c7738 00000006 00100100 00000000 2015881c c69e63f8 1fd84000
19/07:26:02|GPR16: c0390000 c60f2000 c69e63f8 c6ba2128 c6178740 c036f924 ffffffff 00000000
19/07:26:03|GPR24: 1fd84000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7720
19/07:26:03|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:03|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:03|Call Trace:
19/07:26:03|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:03|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:03|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:03|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:03|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:03|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:03|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:03|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:03|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:03|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:03|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:03|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:03|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:03|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:03|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:04|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:04|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:04|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:04|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:04|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:04|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:04|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:04|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:04|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:04|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:04|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:04|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:04|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:04|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:04|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:04|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:04|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:04|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:04|Instruction dump:
19/07:26:04|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:05|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:05|---[ end trace 87f7fdd1c04ff305 ]---
19/07:26:05|Fixing recursive fault but reboot is needed!
19/07:26:05|BUG: scheduling while atomic: syslogd/224/0x00000006
19/07:26:05|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:05|Call Trace:
19/07:26:05|[c60f34c0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:05|[c60f3500] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:05|[c60f3510] [c02a8020] schedule+0x2f4/0x34c
19/07:26:05|[c60f3550] [c00382a8] do_exit+0x544/0x61c
19/07:26:05|[c60f3590] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:05|[c60f35b0] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:05|[c60f35c0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:05|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:05|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:05|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:05|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:05|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:05|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:05|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:06|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:06|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:06|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:06|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:06|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:06|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:06|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:06|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:06|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:06|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:06|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:06|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:06|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:06|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:06|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:06|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:06|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:06|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:06|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:06|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:07|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:07|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:07|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:07|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:07|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:07|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:07|Unable to handle kernel paging request for data at address 0x00100100
19/07:26:07|Faulting instruction address: 0xc006e9e8
19/07:26:07|Oops: Kernel access of bad area, sig: 11 [#3]
19/07:26:07|PREEMPT Xilinx Virtex440
19/07:26:07|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:07|NIP: c006e9e8 LR: c006e78c CTR: c0285b38
19/07:26:07|REGS: c61adab0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:07|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:07|DEAR: 00100100, ESR: 00000000
19/07:26:07|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:07|GPR00: 00000010 c61adb60 c6b0a4c0 c038c3a8 00000000 000001a5 00000000 00000041
19/07:26:07|GPR08: 00000001 00100100 00000004 c61ac000 22024048 2079d364 00000001 00000000
19/07:26:08|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c61ac000
19/07:26:08|GPR24: 00000000 00000001 00029000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:26:08|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:08|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:08|Call Trace:
19/07:26:08|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:08|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:08|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:08|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:08|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:08|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:08|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:08|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:08|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:08|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:08|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:08|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:08|Instruction dump:
19/07:26:08|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:09|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:09|---[ end trace 87f7fdd1c04ff306 ]---
19/07:26:09|note: klogd[227] exited with preempt_count 1
19/07:26:09|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:09|Faulting instruction address: 0xc006e388
19/07:26:09|Oops: Kernel access of bad area, sig: 11 [#4]
19/07:26:09|PREEMPT Xilinx Virtex440
19/07:26:09|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:09|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:09|REGS: c61ad820 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:09|MSR: 00021000 <ME,CE>  CR: 22224082  XER: 00000000
19/07:26:09|DEAR: 00100104, ESR: 00800000
19/07:26:09|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:09|GPR00: 00000000 c61ad8d0 c6b0a4c0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:09|GPR08: 00000008 c03c7e98 00000006 00100100 00000000 2079d364 c6a5640c 20739000
19/07:26:09|GPR16: c0390000 c61ac000 c6a5640c c60a6320 c6178c80 c036f924 ffffffff 00000000
19/07:26:09|GPR24: 20739000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7e80
19/07:26:10|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:10|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:10|Call Trace:
19/07:26:10|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:10|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:10|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:10|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:10|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:10|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:10|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:10|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:10|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:10|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:10|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:10|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:10|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:10|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:10|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:10|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:10|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:11|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:11|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:11|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:11|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:11|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:11|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:11|Instruction dump:
19/07:26:11|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:11|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:11|---[ end trace 87f7fdd1c04ff307 ]---
19/07:26:11|Fixing recursive fault but reboot is needed!
19/07:26:11|BUG: scheduling while atomic: klogd/227/0x00000006
19/07:26:11|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:11|Call Trace:
19/07:26:11|[c61ad710] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:11|[c61ad750] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:11|[c61ad760] [c02a8020] schedule+0x2f4/0x34c
19/07:26:11|[c61ad7a0] [c00382a8] do_exit+0x544/0x61c
19/07:26:11|[c61ad7e0] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:11|[c61ad800] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61ad810] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:12|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:12|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:12|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:12|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:12|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:12|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:12|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:12|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:12|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:12|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:12|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:12|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:12|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:12|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:12|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:13|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:13|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:13|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:13|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:13|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:13|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:13|Faulting instruction address: 0xc006e9e8
19/07:26:13|Oops: Kernel access of bad area, sig: 11 [#5]
19/07:26:13|PREEMPT Xilinx Virtex440
19/07:26:13|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:13|NIP: c006e9e8 LR: c006e78c CTR: c006b2c4
19/07:26:13|REGS: c6349900 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:13|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:13|DEAR: 00000000, ESR: 00000000
19/07:26:13|TASK = c62fa500[379] 'AidaExecV9' THREAD: c6348000
19/07:26:13|GPR00: 00000009 c63499b0 c62fa500 c038c3a8 00000000 000001a5 00100100 00000041
19/07:26:13|GPR08: 00000001 00000000 00000004 c6348000 22022048 1005520c 00000001 00000000
19/07:26:13|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6348000
19/07:26:14|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 ffffffe8 c038c39c c038c394
19/07:26:14|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:14|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:14|Call Trace:
19/07:26:14|[c63499b0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:14|[c6349a40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:14|[c6349ac0] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:14|[c6349b20] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:26:14|[c6349b40] [c006b2e0] mempool_alloc_slab+0x1c/0x2c
19/07:26:14|[c6349b50] [c006b5d8] mempool_alloc+0x5c/0x13c
19/07:26:14|[c6349ba0] [c0294600] rpc_malloc+0x34/0x9c
19/07:26:14|[c6349bc0] [c028da44] call_allocate+0xc4/0x194
19/07:26:14|[c6349be0] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:14|[c6349c20] [c028d120] rpc_run_task+0x44/0x90
19/07:26:14|[c6349c30] [c028d234] rpc_call_sync+0x54/0x84
19/07:26:14|[c6349c60] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
19/07:26:14|[c6349c90] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
19/07:26:14|[c6349d50] [c0129b3c] nfs_permission+0x178/0x1c4
19/07:26:14|[c6349d70] [c00a0bf4] inode_permission+0x94/0xbc
19/07:26:15|[c6349d80] [c00a2bc0] __link_path_walk+0xb0/0xf7c
19/07:26:15|[c6349df0] [c00a3ad4] path_walk+0x48/0xc8
19/07:26:15|[c6349e20] [c00a3c44] do_path_lookup+0x64/0x6c
19/07:26:15|[c6349e40] [c00a4b14] do_filp_open+0xc8/0x914
19/07:26:15|[c6349f10] [c0095314] do_sys_open+0x64/0x130
19/07:26:15|[c6349f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:15|Instruction dump:
19/07:26:15|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:15|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:15|---[ end trace 87f7fdd1c04ff308 ]---
19/07:26:15|note: AidaExecV9[379] exited with preempt_count 1
19/07:26:15|Call Trace:
19/07:26:15|[c6311b50] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:15|[c6311b90] [c006da9c] bad_page+0x94/0x12c
19/07:26:15|[c6311bb0] [c006eadc] get_page_from_freelist+0x450/0x4cc
19/07:26:15|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:15|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:15|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:15|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:15|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:16|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:16|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:16|Faulting instruction address: 0xc006e9e8
19/07:26:16|Oops: Kernel access of bad area, sig: 11 [#6]
19/07:26:16|PREEMPT Xilinx Virtex440
19/07:26:16|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:16|NIP: c006e9e8 LR: c006e78c CTR: c01c13cc
19/07:26:16|REGS: c6311b00 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:16|MSR: 00021000 <ME,CE>  CR: 24222024  XER: 00000000
19/07:26:16|DEAR: 00000000, ESR: 00000000
19/07:26:16|TASK = c62fb9a0[375] 'AidaExecV9' THREAD: c6310000
19/07:26:16|GPR00: 00000009 c6311bb0 c62fb9a0 c038c3a8 c62fb9a0 00000010 00100100 0000594b
19/07:26:16|GPR08: 000000a0 00000000 00000000 c6310000 24222042 1005520c 00000001 00000000
19/07:26:16|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6310000
19/07:26:16|GPR24: 00000000 00000001 00029000 c038c384 00000001 ffffffe8 c038c39c c038c394
19/07:26:16|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:16|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:17|Call Trace:
19/07:26:17|[c6311bb0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:17|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:17|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:17|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:17|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:17|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:17|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:17|Instruction dump:
19/07:26:17|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:17|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:17|---[ end trace 87f7fdd1c04ff309 ]---
19/07:26:17|note: AidaExecV9[375] exited with preempt_count 1
19/07:26:17|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:18|Faulting instruction address: 0xc006e388
19/07:26:18|Oops: Kernel access of bad area, sig: 11 [#7]
19/07:26:18|PREEMPT Xilinx Virtex440
19/07:26:18|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:18|NIP: c006e388 LR: c006e348 CTR: 00000000
19/07:26:18|REGS: c6835e10 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:18|MSR: 00021000 <ME,CE>  CR: 24000082  XER: 00000000
19/07:26:18|DEAR: 00100104, ESR: 00800000
19/07:26:18|TASK = c6824880[5] 'events/0' THREAD: c6834000
19/07:26:18|GPR00: 00000000 c6835ec0 c6824880 00000000 00000003 c038c3a8 ffffffff c0498000
19/07:26:18|GPR08: 00000008 c0419a78 00000027 00100100 002515ba 00005aa8 bfffffff 5fffffff
19/07:26:18|GPR16: ffff7fd7 fffeffaf 00120110 7ffefd7f c031aef4 c02fb040 c02fb008 00000002
19/07:26:18|GPR24: c6834000 00000000 c038c384 c038c384 00000000 00029000 c038c39c c0419a60
19/07:26:18|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:18|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:18|Call Trace:
19/07:26:18|[c6835ec0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:18|[c6835ef0] [c00915c4] kmem_freepages+0xd8/0x134
19/07:26:19|[c6835f00] [c0091ab8] slab_destroy+0x34/0x8c
19/07:26:19|[c6835f10] [c0091e20] drain_freelist+0x9c/0x110
19/07:26:19|[c6835f40] [c0093158] cache_reap+0x104/0x138
19/07:26:19|[c6835f60] [c0048908] worker_thread+0x140/0x204
19/07:26:19|[c6835fc0] [c004cfc8] kthread+0x78/0x7c
19/07:26:19|[c6835ff0] [c000e140] kernel_thread+0x4c/0x68
19/07:26:19|Instruction dump:
19/07:26:19|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:19|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:19|---[ end trace 87f7fdd1c04ff30a ]---
19/07:26:19|note: events/0[5] exited with preempt_count 1
19/07:26:19|Unable to handle kernel paging request for data at address 0x00100100
19/07:29:19|Faulting instruction address: 0xc006e9e8
19/07:29:19|Oops: Kernel access of bad area, sig: 11 [#8]
19/07:29:19|PREEMPT Xilinx Virtex440
19/07:29:19|Modules linked in: aidamem xdriver xh_spidev_register
19/07:29:19|NIP: c006e9e8 LR: c006e78c CTR: c025c424
19/07:29:19|REGS: c0391a60 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:29:19|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:29:19|DEAR: 00100100, ESR: 00000000
19/07:29:19|TASK = c036e318[0] 'swapper' THREAD: c0390000
19/07:29:19|GPR00: 00000010 c0391b10 c036e318 c038c3a8 00000000 000001a5 00000000 00000041
19/07:29:19|GPR08: 00000001 00100100 00000004 c0390000 22022048 00005aa8 00000001 00000000
19/07:29:19|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c0390000
19/07:29:19|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:29:19|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:29:19|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:19|Call Trace:
19/07:29:19|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:29:20|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:20|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:20|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:20|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:20|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:20|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:20|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:20|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:20|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:20|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:20|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:20|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:20|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:20|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:20|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:20|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:20|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:20|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:20|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:21|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:21|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:21|Instruction dump:
19/07:29:21|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:29:21|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:29:21|Kernel panic - not syncing: Fatal exception in interrupt
19/07:29:21|Call Trace:
19/07:29:21|[c0391990] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:29:21|[c03919d0] [c00345bc] panic+0x94/0x168
19/07:29:21|[c0391a20] [c000bd44] die+0x178/0x18c
19/07:29:21|[c0391a40] [c001191c] bad_page_fault+0x90/0xd8
19/07:29:21|[c0391a50] [c000e834] handle_page_fault+0x7c/0x80
19/07:29:21|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:21|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:21|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:21|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:21|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:21|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:21|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:22|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:22|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:22|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:22|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:22|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:22|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:22|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:22|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:22|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:22|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:22|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:22|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:22|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:22|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:22|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:22|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
19/07:32:23|
Found 0 errors checking kernel image
19/07:32:24|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
19/07:32:24|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/07:32:24|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/07:32:24|SPIconfig done now to set up the LMK3200 registers
19/07:32:24|LMK #0 : regInit[0]=0x80000000
19/07:32:24|LMK #0 : regInit[1]=0x10070600
19/07:32:24|LMK #0 : regInit[2]=0x60601
19/07:32:24|LMK #0 : regInit[3]=0x60602
19/07:32:25|LMK #0 : regInit[4]=0x60603
19/07:32:25|LMK #0 : regInit[5]=0x70624
19/07:32:25|LMK #0 : regInit[6]=0x70605
19/07:32:25|LMK #0 : regInit[7]=0x70606
19/07:32:25|LMK #0 : regInit[8]=0x70627
19/07:32:25|LMK #0 : regInit[9]=0x10000908
19/07:32:25|LMK #0 : regInit[10]=0xA0022A09
19/07:32:25|LMK #0 : regInit[11]=0x82800B
19/07:32:25|LMK #0 : regInit[12]=0x28C800D
19/07:32:25|LMK #0 : regInit[13]=0x830020E
19/07:32:25|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/07:32:25|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
19/07:32:25|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/07:32:25|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
19/07:32:26|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/07:32:29|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xcf 0xac
19/07:32:35|Using Xilinx Virtex440 machine description
19/07:32:36|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/07:32:36|Zone PFN ranges:
19/07:32:36|  DMA      0x00000000 -> 0x00007000
19/07:32:36|  Normal   0x00007000 -> 0x00007000
19/07:32:36|Movable zone start PFN for each node
19/07:32:36|early_node_map[1] active PFN ranges
19/07:32:36|    0: 0x00000000 -> 0x00007000
19/07:32:36|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/07:32:36|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/07:32:36|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/07:32:36|PID hash table entries: 512 (order: 9, 2048 bytes)
19/07:32:36|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/07:32:37|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/07:32:37|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/07:32:37|Kernel virtual memory layout:
19/07:32:37|  * 0xffffe000..0xfffff000  : fixmap
19/07:32:37|  * 0xfde00000..0xfe000000  : consistent mem
19/07:32:37|  * 0xfde00000..0xfde00000  : early ioremap
19/07:32:37|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/07:32:37|NR_IRQS:512
19/07:32:37|clocksource: timebase mult[a00000] shift[22] registered
19/07:32:37|Console: colour dummy device 80x25
19/07:32:37|Mount-cache hash table entries: 512
19/07:32:37|NET: Registered protocol family 16
19/07:32:37|PCI: Probing PCI hardware
19/07:32:37|bio: create slab <bio-0> at 0
19/07:32:37|NET: Registered protocol family 2
19/07:32:37|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/07:32:37|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/07:32:37|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/07:32:37|TCP: Hash tables configured (established 4096 bind 4096)
19/07:32:37|TCP reno registered
19/07:32:37|NET: Registered protocol family 1
19/07:32:37|ROMFS MTD (C) 2007 Red Hat, Inc.
19/07:32:38|msgmni has been set to 214
19/07:32:38|io scheduler noop registered
19/07:32:38|io scheduler anticipatory registered
19/07:32:38|io scheduler deadline registered
19/07:32:38|io scheduler cfq registered (default)
19/07:32:38|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/07:32:38|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/07:32:38|console [ttyS0] enabled
19/07:32:38|brd: module loaded
19/07:32:38|loop: module loaded
19/07:32:38|Device Tree Probing 'ethernet'
19/07:32:38|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:cf:ac
19/07:32:38|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/07:32:38|XLlTemac: DCR address: 0x80
19/07:32:38|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/07:32:38|XLlTemac: Allocating DMA descriptors with kmalloc
19/07:32:38|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/07:32:38|XTemac: PHY detected at address 7.
19/07:32:38|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/07:32:38|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39|Using buffer write method
19/07:32:39|cfi_cmdset_0001: Erase suspend on write enabled
19/07:32:39|cmdlinepart partition parsing not available
19/07:32:39|RedBoot partition parsing not available
19/07:32:39|Creating 5 MTD partitions on "fc000000.flash":
19/07:32:39|0x000000000000-0x000000500000 : "golden_firmware"
19/07:32:39|0x000000500000-0x000000800000 : "golden_kernel"
19/07:32:39|0x000000800000-0x000000d00000 : "user_firmware"
19/07:32:39|0x000000d00000-0x000000fe0000 : "user_kernel"
19/07:32:39|0x000000fe0000-0x000001000000 : "env_variables"
19/07:32:39|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/07:32:39|SPI: XIlinx spi: bus number now 32766
19/07:32:39|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/07:32:39|SPI: XIlinx spi: bus number now 32765
19/07:32:40|mice: PS/2 mouse device common for all mice
19/07:32:40|Device Tree Probing 'i2c'
19/07:32:40| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/07:32:40|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/07:32:40|TCP cubic registered
19/07:32:40|NET: Registered protocol family 17
19/07:32:40|RPC: Registered udp transport module.
19/07:32:40|RPC: Registered tcp transport module.
19/07:32:40|eth0: XLlTemac: Options: 0x3fa
19/07:32:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/07:32:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/07:32:40|eth0: XLlTemac: speed set to 1000Mb/s
19/07:32:43|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/07:32:43|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/07:32:43|Sending DHCP requests ., OK
19/07:32:44|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.12
19/07:32:44|IP-Config: Complete:
19/07:32:44|     device=eth0, addr=192.168.11.12, mask=255.255.255.0, gw=255.255.255.255,
19/07:32:44|     host=aida12, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/07:32:44|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12
19/07:32:44|Looking up port of RPC 100003/2 on 192.168.11.99
19/07:32:44|Looking up port of RPC 100005/1 on 192.168.11.99
19/07:32:44|VFS: Mounted root (nfs filesystem) on device 0:12.
19/07:32:44|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/07:32:45|Starting sysinit...
19/07:32:45|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/07:32:46|            Press 'I' to enter interactive startup.
19/07:32:46|Setting clock  (utc): Mon Apr 19 07:32:47 BST 2021 [  OK  ]
19/07:32:47|Building the cache [  OK  ]
19/07:32:47|Setting hostname aida12:  [  OK  ]
19/07:32:48|Mounting local filesystems:  [  OK  ]
19/07:32:50|Enabling /etc/fstab swaps:  [  OK  ]
19/07:32:53|Finishing sysinit...
INIT: Entering runlevel: 3
19/07:32:56|Entering non-interactive startup
19/07:32:56|FATAL: Module ipv6 not found.
19/07:32:57|Bringing up loopback interface:  [  OK  ]
19/07:32:59|FATAL: Module ipv6 not found.
19/07:33:00|Starting system logger: [  OK  ]
19/07:33:01|Starting kernel logger: [  OK  ]
19/07:33:01|Starting rpcbind: [  OK  ]
19/07:33:03|Mounting NFS filesystems:  [  OK  ]
19/07:33:03|Mounting other filesystems:  [  OK  ]
19/07:33:04|Starting xinetd: [  OK  ]
19/07:33:05|Starting midas:  Starting MIDAS Data Acquisition for aida12
19/07:33:05|xaida: device parameters: base=0x81000000 size=0x200000
19/07:33:12|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/07:33:12|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/07:33:12|xaida: driver assigned major number 254
19/07:33:12|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/07:33:17|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/07:33:17|AIDAMEM: aidamem: driver assigned major number 253
19/07:33:17|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/07:33:24|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/07:33:24|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/07:33:24|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/07:33:24|Computer Name = aida12; Temp Directory = /tmp/tcl361
19/07:33:31|
19/07:33:34|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting

From ttyUSB7 system console log

20/09:59:30|get_ASICBlk : Blk=1 : bytes = 793600 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0017F1D : 1 => 0FAC4BAF
20/09:59:31|Last Data : Databuffer  273884152 : 0 => 80426EF4 : 1 => 0BD70000
20/09:59:31| C03E7F21 : 03FCCB77 | C03D7EF5 : 03FCEE9F | C0047DA3 : 03FCF5A7 | C0207E21 : 03FCF5A7
20/09:59:31| C0317E77 : 03FCF5A7 | C0087E6D : 03FCF66F | C0107E98 : 03FCF66F | C0217ED8 : 03FCF66F
20/09:59:31| C0327E19 : 03FCF66F | C00B7DF0 : 03FCF737 | C0117DD5 : 03FCF737 | C0227F27 : 03FCF737
20/09:59:31| C03C7F23 : 03FCF737 | C0137EF9 : 03FCF7FF | C0297EB7 : 03FCF7FF | C03E7EC7 : 03FCF7FF
20/09:59:31| C0167DFE : 03FCF8C7 | C02A7FC3 : 03FCF8C7 | C03F7E82 : 03FCF8C7 | C01B7E91 : 03FCF98F
20/09:59:31| C02F7E8F : 03FCF98F | C01D7DC4 : 03FCFA57 | C01E7D40 : 03FCFB1F | 80426EF3 : 03FD0000
20/09:59:32| 8050023F : 03FD0000 | C03D7F1A : 03FD303F | 80426EF3 : 03FD4000 | 8050023F : 03FD4000
20/09:59:32| C0317F2F : 03FD5047 | C0047DF2 : 03FD510F | C01B7EAD : 03FD54F7 | C03D7F09 : 03FD5D8F
20/09:59:32|SIGPIPE ERROR - Error in xfer ASIC procedures: location=0
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_bxaida: release:
20/10:00:41|loAIDAMEM: aidamem_release:
20/10:00:41|ck busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|SIGSEGV ERROR - Error in xfer ASIC procedures: location=0
20/10:00:41|xfer_block busy
20/10:00:41|do_GetState returned z=0 and 1
20/10:01:27|executing generic doStop
20/10:02:41|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:02:42|StopC:
20/10:02:42|do_GetState returned z=0 and 6
20/10:02:47|do_GetState returned z=0 and 6
20/10:02:58|do_GetState returned z=0 and 6
20/10:03:10|do_GetState returned z=0 and 6
20/10:03:21|do_GetState returned z=0 and 6
20/10:03:33|do_GetState returned z=0 and 6
20/10:03:44|do_GetState returned z=0 and 6
20/10:03:56|do_GetState returned z=0 and 6
20/10:04:07|do_GetState returned z=0 and 6
20/10:04:19|do_GetState returned z=0 and 6
20/10:04:30|do_GetState returned z=0 and 6
20/10:04:42|do_GetState returned z=0 and 6
20/10:04:53|do_GetState returned z=0 and 6
20/10:05:04|do_GetState returned z=0 and 6
20/10:05:16|do_GetState returned z=0 and 6
20/10:05:27|do_GetState returned z=0 and 6
20/10:05:39|do_GetState returned z=0 and 6
20/10:05:50|do_GetState returned z=0 and 6
20/10:06:02|do_GetState returned z=0 and 6
20/10:06:12|executing generic doReset
20/10:06:21|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:06:22|ResetC:
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:34|do_GetState returned z=0 and 6
20/10:06:45|do_GetState returned z=0 and 6
20/10:06:57|do_GetState returned z=0 and 6
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
20/10:07:15|[  OK  ]
20/10:07:15|Stopping rpcbind: [  OK  ]
20/10:07:17|Shutting down kernel logger: [  OK  ]
20/10:07:19|Shutting down system logger: [  OK  ]
20/10:07:19|Stopping xinetd: [  OK  ]
20/10:07:19|[  OK  ]
20/10:07:19|Sending all processes the TERM signal...
20/10:07:20|xaida: release:
20/10:07:20|Sending all processes the KILL signal...
20/10:07:25|Saving random seed:
20/10:07:25|Syncing hardware clock to system time xaida: open:
20/10:07:25|xaida: release:
20/10:07:25|xaida: open:
20/10:07:25|xaida: ioctl: called with command 536899587
20/10:07:25|xaida: ioctl: called with command 1076129801
20/10:07:25|RTCxaida: release:
20/10:07:25|_RD_TIME: Inappropriate ioctl for device
20/10:07:25|ioctl() to /dev/rtc to read the time failed.
20/10:07:25|
20/10:07:25|Please stand by while rebooting the system...
Sj�5õÅ07:27|Restarting system.
20/10:07:29|ÉU�×®­Ë
                   R�   $T,W¤']Z�¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/10:07:29|
Found 0 errors checking kernel image
20/10:07:30|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/10:07:30|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/10:07:31|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/10:07:31|SPIconfig done now to set up the LMK3200 registers
20/10:07:31|LMK #0 : regInit[0]=0x80000000
20/10:07:31|LMK #0 : regInit[1]=0x10070600
20/10:07:31|LMK #0 : regInit[2]=0x60601
20/10:07:31|LMK #0 : regInit[3]=0x60602
20/10:07:31|LMK #0 : regInit[4]=0x60603
20/10:07:31|LMK #0 : regInit[5]=0x70624
20/10:07:31|LMK #0 : regInit[6]=0x70605
20/10:07:31|LMK #0 : regInit[7]=0x70606
20/10:07:31|LMK #0 : regInit[8]=0x70627
20/10:07:31|LMK #0 : regInit[9]=0x10000908
20/10:07:31|LMK #0 : regInit[10]=0xA0022A09
20/10:07:31|LMK #0 : regInit[11]=0x82800B
20/10:07:31|LMK #0 : regInit[12]=0x28C800D
20/10:07:31|LMK #0 : regInit[13]=0x830020E
20/10:07:31|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/10:07:32|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/10:07:32|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/10:07:32|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/10:07:32|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/10:07:35|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xd7 0xcc
20/10:07:42|Using Xilinx Virtex440 machine description
20/10:07:42|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/10:07:42|Zone PFN ranges:
20/10:07:42|  DMA      0x00000000 -> 0x00007000
20/10:07:42|  Normal   0x00007000 -> 0x00007000
20/10:07:42|Movable zone start PFN for each node
20/10:07:42|early_node_map[1] active PFN ranges
20/10:07:43|    0: 0x00000000 -> 0x00007000
20/10:07:43|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/10:07:43|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/10:07:43|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/10:07:43|PID hash table entries: 512 (order: 9, 2048 bytes)
20/10:07:43|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/10:07:43|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/10:07:43|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/10:07:43|Kernel virtual memory layout:
20/10:07:43|  * 0xffffe000..0xfffff000  : fixmap
20/10:07:43|  * 0xfde00000..0xfe000000  : consistent mem
20/10:07:43|  * 0xfde00000..0xfde00000  : early ioremap
20/10:07:43|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/10:07:43|NR_IRQS:512
20/10:07:43|clocksource: timebase mult[a00000] shift[22] registered
20/10:07:43|Console: colour dummy device 80x25
20/10:07:43|Mount-cache hash table entries: 512
20/10:07:43|NET: Registered protocol family 16
20/10:07:43|PCI: Probing PCI hardware
20/10:07:43|bio: create slab <bio-0> at 0
20/10:07:44|NET: Registered protocol family 2
20/10:07:44|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/10:07:44|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/10:07:44|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/10:07:44|TCP: Hash tables configured (established 4096 bind 4096)
20/10:07:44|TCP reno registered
20/10:07:44|NET: Registered protocol family 1
20/10:07:44|ROMFS MTD (C) 2007 Red Hat, Inc.
20/10:07:44|msgmni has been set to 214
20/10:07:44|io scheduler noop registered
20/10:07:44|io scheduler anticipatory registered
20/10:07:44|io scheduler deadline registered
20/10:07:44|io scheduler cfq registered (default)
20/10:07:44|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/10:07:44|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/10:07:44|console [ttyS0] enabled
20/10:07:44|brd: module loaded
20/10:07:44|loop: module loaded
20/10:07:44|Device Tree Probing 'ethernet'
20/10:07:44|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:d7:cc
20/10:07:44|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/10:07:44|XLlTemac: DCR address: 0x80
20/10:07:44|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/10:07:45|XLlTemac: Allocating DMA descriptors with kmalloc
20/10:07:45|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/10:07:45|XTemac: PHY detected at address 7.
20/10:07:45|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/10:07:45|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45|Using buffer write method
20/10:07:45|cfi_cmdset_0001: Erase suspend on write enabled
20/10:07:45|cmdlinepart partition parsing not available
20/10:07:45|RedBoot partition parsing not available
20/10:07:45|Creating 5 MTD partitions on "fc000000.flash":
20/10:07:45|0x000000000000-0x000000500000 : "golden_firmware"
20/10:07:45|0x000000500000-0x000000800000 : "golden_kernel"
20/10:07:45|0x000000800000-0x000000d00000 : "user_firmware"
20/10:07:46|0x000000d00000-0x000000fe0000 : "user_kernel"
20/10:07:46|0x000000fe0000-0x000001000000 : "env_variables"
20/10:07:46|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/10:07:46|SPI: XIlinx spi: bus number now 32766
20/10:07:46|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/10:07:46|SPI: XIlinx spi: bus number now 32765
20/10:07:46|mice: PS/2 mouse device common for all mice
20/10:07:46|Device Tree Probing 'i2c'
20/10:07:46| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/10:07:46|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/10:07:46|TCP cubic registered
20/10:07:46|NET: Registered protocol family 17
20/10:07:46|RPC: Registered udp transport module.
20/10:07:46|RPC: Registered tcp transport module.
20/10:07:46|eth0: XLlTemac: Options: 0x3fa
20/10:07:47|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/10:07:47|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/10:07:47|eth0: XLlTemac: speed set to 1000Mb/s
20/10:07:49|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/10:07:49|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/10:07:49|Sending DHCP requests ., OK
20/10:07:50|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.5
20/10:07:50|IP-Config: Complete:
20/10:07:50|     device=eth0, addr=192.168.11.5, mask=255.255.255.0, gw=255.255.255.255,
20/10:07:50|     host=aida05, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/10:07:50|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05
20/10:07:50|Looking up port of RPC 100003/2 on 192.168.11.99
20/10:07:51|Looking up port of RPC 100005/1 on 192.168.11.99
20/10:07:51|VFS: Mounted root (nfs filesystem) on device 0:12.
20/10:07:51|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/10:07:51|Starting sysinit...
20/10:07:51|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/10:07:52|            Press 'I' to enter interactive startup.
20/10:07:52|Setting clock  (utc): Tue Apr 20 10:07:52 BST 2021 [  OK  ]
20/10:07:53|Building the cache [  OK  ]
20/10:07:53|Setting hostname aida05:  [  OK  ]
20/10:07:54|Mounting local filesystems:  [  OK  ]
20/10:07:55|Enabling /etc/fstab swaps:  [  OK  ]
20/10:07:57|Finishing sysinit...
INIT: Entering runlevel: 3
20/10:08:00|Entering non-interactive startup
20/10:08:01|FATAL: Module ipv6 not found.
20/10:08:02|Bringing up loopback interface:  [  OK  ]
20/10:08:04|FATAL: Module ipv6 not found.
20/10:08:04|Starting system logger: [  OK  ]
20/10:08:05|Starting kernel logger: [  OK  ]
20/10:08:05|Starting rpcbind: [  OK  ]
20/10:08:06|Mounting NFS filesystems:  [  OK  ]
20/10:08:07|Mounting other filesystems:  [  OK  ]
20/10:08:07|Starting xinetd: [  OK  ]
20/10:08:08|Starting midas:  Starting MIDAS Data Acquisition for aida05
20/10:08:08|xaida: device parameters: base=0x81000000 size=0x200000
20/10:08:14|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/10:08:14|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/10:08:14|xaida: driver assigned major number 254
20/10:08:14|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/10:08:20|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/10:08:20|AIDAMEM: aidamem: driver assigned major number 253
20/10:08:20|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/10:08:26|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/10:08:27|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/10:08:27|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/10:08:27|Computer Name = aida05; Temp Directory = /tmp/tcl356
20/10:08:30|package limit is not available: can't find package limit
20/10:08:33|Running with default file descriptor limit
20/10:08:33|package setuid is not available: can't find package setuid
20/10:08:35|Running as user 0 group 0
20/10:08:35|
20/10:08:36|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting

From ttyUSB6 system console log

20/07:07:10|kernel BUG at mm/slab.c:2974!
20/07:07:10|Oops: Exception in kernel mode, sig: 5 [#1]
20/07:07:10|PREEMPT Xilinx Virtex440
20/07:07:10|Modules linked in: aidamem xdriver xh_spidev_register
20/07:07:10|NIP: c009211c LR: c00920b0 CTR: c0038e70
20/07:07:10|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)
20/07:07:10|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000
20/07:07:10|TASK = c036e318[0] 'swapper' THREAD: c0390000
20/07:07:10|GPR00: 00000001 c0391ca0 c036e318 c680daf0 00000020 00000001 c6940000 00000000
20/07:07:11|GPR08: fffffff4 00010003 c680e400 c0390000 24008042 00005aa8 c03a0000 00000020
20/07:07:11|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200
20/07:07:11|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400
20/07:07:11|NIP [c009211c] cache_alloc_refill+0x130/0x608
20/07:07:11|LR [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:11|Call Trace:
20/07:07:11|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
20/07:07:11|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:11|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:11|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:11|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:11|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:11|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:11|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:11|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:11|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:11|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:11|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:12|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:12|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:12|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:12|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:12|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:12|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:12|Instruction dump:
20/07:07:12|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010
20/07:07:12|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840
20/07:07:12|Kernel panic - not syncing: Fatal exception in interrupt
20/07:07:12|Call Trace:
20/07:07:12|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/07:07:12|[c0391a80] [c00345bc] panic+0x94/0x168
20/07:07:12|[c0391ad0] [c000bd44] die+0x178/0x18c
20/07:07:12|[c0391af0] [c000c000] _exception+0x164/0x1b4
20/07:07:12|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c
20/07:07:12|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:12|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:12|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:12|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:13|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:13|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:13|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:13|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:13|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:13|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:13|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:13|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:13|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:13|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:13|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:13|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:13|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:13|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:13|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/07:10:14|
Found 0 errors checking kernel image
20/07:10:15|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/07:10:15|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/07:10:15|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/07:10:15|SPIconfig done now to set up the LMK3200 registers
20/07:10:15|LMK #0 : regInit[0]=0x80000000
20/07:10:15|LMK #0 : regInit[1]=0x10070600
20/07:10:15|LMK #0 : regInit[2]=0x60601
20/07:10:15|LMK #0 : regInit[3]=0x60602
20/07:10:16|LMK #0 : regInit[4]=0x60603
20/07:10:16|LMK #0 : regInit[5]=0x70624
20/07:10:16|LMK #0 : regInit[6]=0x70605
20/07:10:16|LMK #0 : regInit[7]=0x70606
20/07:10:16|LMK #0 : regInit[8]=0x70627
20/07:10:16|LMK #0 : regInit[9]=0x10000908
20/07:10:16|LMK #0 : regInit[10]=0xA0022A09
20/07:10:16|LMK #0 : regInit[11]=0x82800B
20/07:10:16|LMK #0 : regInit[12]=0x28C800D
20/07:10:16|LMK #0 : regInit[13]=0x830020E
20/07:10:16|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/07:10:16|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/07:10:16|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/07:10:16|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/07:10:17|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/07:10:20|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xee
20/07:10:26|Using Xilinx Virtex440 machine description
20/07:10:27|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/07:10:27|Zone PFN ranges:
20/07:10:27|  DMA      0x00000000 -> 0x00007000
20/07:10:27|  Normal   0x00007000 -> 0x00007000
20/07:10:27|Movable zone start PFN for each node
20/07:10:27|early_node_map[1] active PFN ranges
20/07:10:27|    0: 0x00000000 -> 0x00007000
20/07:10:27|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/07:10:27|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/07:10:27|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/07:10:27|PID hash table entries: 512 (order: 9, 2048 bytes)
20/07:10:27|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/07:10:27|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/07:10:28|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/07:10:28|Kernel virtual memory layout:
20/07:10:28|  * 0xffffe000..0xfffff000  : fixmap
20/07:10:28|  * 0xfde00000..0xfe000000  : consistent mem
20/07:10:28|  * 0xfde00000..0xfde00000  : early ioremap
20/07:10:28|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/07:10:28|NR_IRQS:512
20/07:10:28|clocksource: timebase mult[a00000] shift[22] registered
20/07:10:28|Console: colour dummy device 80x25
20/07:10:28|Mount-cache hash table entries: 512
20/07:10:28|NET: Registered protocol family 16
20/07:10:28|PCI: Probing PCI hardware
20/07:10:28|bio: create slab <bio-0> at 0
20/07:10:28|NET: Registered protocol family 2
20/07:10:28|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/07:10:28|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/07:10:28|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/07:10:28|TCP: Hash tables configured (established 4096 bind 4096)
20/07:10:28|TCP reno registered
20/07:10:28|NET: Registered protocol family 1
20/07:10:28|ROMFS MTD (C) 2007 Red Hat, Inc.
20/07:10:28|msgmni has been set to 214
20/07:10:29|io scheduler noop registered
20/07:10:29|io scheduler anticipatory registered
20/07:10:29|io scheduler deadline registered
20/07:10:29|io scheduler cfq registered (default)
20/07:10:29|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/07:10:29|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/07:10:29|console [ttyS0] enabled
20/07:10:29|brd: module loaded
20/07:10:29|loop: module loaded
20/07:10:29|Device Tree Probing 'ethernet'
20/07:10:29|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ee
20/07:10:29|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/07:10:29|XLlTemac: DCR address: 0x80
20/07:10:29|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/07:10:29|XLlTemac: Allocating DMA descriptors with kmalloc
20/07:10:29|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/07:10:29|XTemac: PHY detected at address 7.
20/07:10:29|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/07:10:29|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/07:10:29| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30|Using buffer write method
20/07:10:30|cfi_cmdset_0001: Erase suspend on write enabled
20/07:10:30|cmdlinepart partition parsing not available
20/07:10:30|RedBoot partition parsing not available
20/07:10:30|Creating 5 MTD partitions on "fc000000.flash":
20/07:10:30|0x000000000000-0x000000500000 : "golden_firmware"
20/07:10:30|0x000000500000-0x000000800000 : "golden_kernel"
20/07:10:30|0x000000800000-0x000000d00000 : "user_firmware"
20/07:10:30|0x000000d00000-0x000000fe0000 : "user_kernel"
20/07:10:30|0x000000fe0000-0x000001000000 : "env_variables"
20/07:10:30|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/07:10:30|SPI: XIlinx spi: bus number now 32766
20/07:10:30|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/07:10:30|SPI: XIlinx spi: bus number now 32765
20/07:10:30|mice: PS/2 mouse device common for all mice
20/07:10:31|Device Tree Probing 'i2c'
20/07:10:31| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/07:10:31|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/07:10:31|TCP cubic registered
20/07:10:31|NET: Registered protocol family 17
20/07:10:31|RPC: Registered udp transport module.
20/07:10:31|RPC: Registered tcp transport module.
20/07:10:31|eth0: XLlTemac: Options: 0x3fa
20/07:10:31|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/07:10:31|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/07:10:31|eth0: XLlTemac: speed set to 1000Mb/s
20/07:10:33|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/07:10:34|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/07:10:34|Sending DHCP requests ., OK
20/07:10:35|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
20/07:10:35|IP-Config: Complete:
20/07:10:35|     device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
20/07:10:35|     host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/07:10:35|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
20/07:10:35|Looking up port of RPC 100003/2 on 192.168.11.99
20/07:10:35|Looking up port of RPC 100005/1 on 192.168.11.99
20/07:10:35|VFS: Mounted root (nfs filesystem) on device 0:12.
20/07:10:35|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/07:10:36|Starting sysinit...
20/07:10:36|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/07:10:37|            Press 'I' to enter interactive startup.
20/07:10:37|Setting clock  (utc): Tue Apr 20 07:10:37 BST 2021 [  OK  ]
20/07:10:38|Building the cache [  OK  ]
20/07:10:38|Setting hostname aida09:  [  OK  ]
20/07:10:39|Mounting local filesystems:  [  OK  ]
20/07:10:40|Enabling /etc/fstab swaps:  [  OK  ]
20/07:10:43|Finishing sysinit...
INIT: Entering runlevel: 3
20/07:10:46|Entering non-interactive startup
20/07:10:46|FATAL: Module ipv6 not found.
20/07:10:47|Bringing up loopback interface:  [  OK  ]
20/07:10:49|FATAL: Module ipv6 not found.
20/07:10:50|Starting system logger: [  OK  ]
20/07:10:50|Starting kernel logger: [  OK  ]
20/07:10:50|Starting rpcbind: [  OK  ]
20/07:10:52|Mounting NFS filesystems:  [  OK  ]
20/07:10:53|Mounting other filesystems:  [  OK  ]
20/07:10:53|Starting xinetd: [  OK  ]
20/07:10:54|Starting midas:  Starting MIDAS Data Acquisition for aida09
20/07:10:54|xaida: device parameters: base=0x81000000 size=0x200000
20/07:11:00|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/07:11:00|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/07:11:00|xaida: driver assigned major number 254
20/07:11:00|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/07:11:06|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/07:11:06|AIDAMEM: aidamem: driver assigned major number 253
20/07:11:06|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/07:11:12|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/07:11:12|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/07:11:12|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/07:11:13|Computer Name = aida09; Temp Directory = /tmp/tcl361
20/07:11:17|package limit is not available: can't find package limit
20/07:11:20|Running with default file descriptor limit
20/07:11:20|package setuid is not available: can't find package setuid
20/07:11:21|Running as user 0 group 0
20/07:11:21|
20/07:11:22|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
20/07:11:22|
20/07:11:22|Built without pthreads
20/07:11:22|xaida: open:
20/07:11:22|
20/07:11:22|Creating NetAIDAMEM: aidamem_open:
20/07:11:22|Vars
20/07:11:22|Output buffer length = 65504; format option = 4; transfer option = 3
20/07:11:22|EB transfer option = 3
20/07:11:22|NetVars created and initialised
20/07:11:22|Statistics thread starting
20/07:11:22|Statistics thread created
20/07:11:22|Stat/Rate creation thread starting
20/07:11:22|Data Acquisition task has PID 375
20/07:11:22|Stat/Rate creation thread created
20/07:11:22|Hit/Rate creation thread starting
20/07:11:22|Hit/Rate creation thread created
20/07:11:22|AIDA Heartbeat thread starting
20/07:11:22|Heartbeat thread created
20/07:11:22|Installing signal handlers
20/07:11:22|Done
20/07:11:22|ModuleNum = 0
20/07:11:22|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706
20/07:11:22|Spectra table initialised
20/07:11:22|AIDA Data Acquisition now all ready to start
20/07:11:23|SIGBUS, SIGSEGV and SIGPIPE traps setup
20/07:11:23|/debug user "debug" password "5tvq-4iyxl 4"
20/07:11:30|httpd started on port 8015
20/07:11:30|
20/07:11:30|Cannot use /MIDAS/config/TclHttpd/aida09@8015/startup.tcl
20/07:11:30|Custom startup from /MIDAS/config/TclHttpd/aida09/startup.tcl
20/07:11:30|XAIDA Access package 1.0
20/07:11:32|/XAIDAAccessServer
20/07:11:32|XAD9252 Access package 1.0
20/07:11:32|/XAD9252AccessServer
20/07:11:32|/DataBaseAccessServer
20/07:11:32|[  OK  ]
20/07:11:32|/NetVarService
20/07:11:32|/SigTaskService
20/07:11:32|Loaded MemSasAccess
20/07:11:33|/SpectrumService
20/07:11:33|loading tcl/AIDARunControl.tcl for namespace ::
20/07:11:33|/DataAcquisitionControlServer
20/07:11:33|DefineMessage unknown
20/07:11:33|
20/07:11:33|Run Control Server Implementation for AIDA
20/07:11:33|DENX ELDK version 4.2 build 2008-04-01
20/07:11:33|Linux 2.6.31 on a ppc
20/07:11:33|
20/07:11:33|aida09 login: RunControlServer loaded
20/07:11:33|loading Html/RunControl/implementation.tcl
20/07:11:33|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
20/07:11:34|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
20/07:11:34|Created UI registers
20/07:11:34|RunControl loaded
20/07:11:34|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
20/07:11:34|AIDA RunControl loaded
20/07:11:35|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
20/07:11:35|do_GetState returned z=0 and 8
20/07:12:03|do_GetState returned z=0 and 8
20/07:12:09|do_GetState returned z=0 and 8
20/07:12:21|xaida: open:
20/07:12:35|do_GetState returned z=0 and 8
20/07:12:36|

From aida01 /var/log/messages

Apr 20 21:04:06 aida01 kernel: BUG: scheduling while atomic: tclsh8.5/398/0x00000201
Apr 20 21:04:08 aida01 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 20 21:04:09 aida01 kernel: Call Trace:
Apr 20 21:04:11 aida01 kernel: [c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
Apr 20 21:04:12 aida01 kernel: [c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
Apr 20 21:04:14 aida01 kernel: [c6941aa0] [c02a8020] schedule+0x2f4/0x34c
Apr 20 21:07:53 aida01 syslogd 1.4.2: restart.

From ttyUSB0 system console log

20/21:04:06|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:06|Call Trace:
20/21:04:06|[c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:06|[c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:07|[c6941aa0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:07|[c6941ae0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:07|[c6941af0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:07|[c6941b10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:07|[c6941b50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:07|[c6941b90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:07|[c6941ba0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:07|[c6941bd0] [c013da58] nfs3_rpc_wrapper+0x48/0x84
20/21:04:07|[c6941bf0] [c013e734] nfs3_proc_access+0xcc/0x1ac
20/21:04:07|[c6941cc0] [c0129644] nfs_do_access+0xb8/0x438
20/21:04:07|[c6941d00] [c0129a6c] nfs_permission+0xa8/0x1c4
20/21:04:07|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:07|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:07|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:07|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:07|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:07|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:07|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:07|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:07|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:08|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:08|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:08|Call Trace:
20/21:04:08|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:08|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:08|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:08|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:08|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:08|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:08|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:08|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:08|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:08|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:08|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:08|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:08|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:08|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:09|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:09|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:09|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:09|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:09|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:09|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:09|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:09|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:09|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:09|Call Trace:
20/21:04:09|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:09|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:09|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:09|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:09|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:09|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:09|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:09|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:09|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:09|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:10|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:10|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:10|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:10|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:10|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:10|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:10|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:10|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:10|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:10|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:10|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:10|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:10|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:10|Call Trace:
20/21:04:10|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:10|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:10|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:10|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:10|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:10|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:11|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:11|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:11|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:11|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:11|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:11|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:11|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:11|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:11|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:11|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:11|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:11|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:11|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:11|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:11|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:11|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:11|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:11|Call Trace:
20/21:04:11|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:11|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:12|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:12|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:12|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:12|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:12|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:12|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:12|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:12|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:12|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:12|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:12|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:12|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:12|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:12|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:12|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:12|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:12|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:12|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:12|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:12|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:13|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:13|Call Trace:
20/21:04:13|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:13|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:13|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:13|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:13|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:13|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:13|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:13|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:13|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:13|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:13|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:13|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:13|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:13|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:13|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:13|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:13|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:13|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:14|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:14|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:14|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:14|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:14|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:14|Call Trace:
20/21:04:14|[c6941950] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:14|[c6941990] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:14|[c69419a0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:14|[c69419e0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:14|[c69419f0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:14|[c6941a10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:14|[c6941a50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:14|[c6941a90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:14|[c6941aa0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:14|[c6941ad0] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:14|[c6941b00] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:14|[c6941bc0] [c012bd70] nfs_lookup_revalidate+0x3c0/0x464
20/21:04:14|[c6941d00] [c00a0f7c] do_lookup+0x54/0x20c
20/21:04:14|[c6941d30] [c00a308c] __link_path_walk+0x57c/0xf7c
20/21:04:15|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:15|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:15|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:15|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:15|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:15|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:15|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:15|Page fault in user mode with in_atomic() = 1 mm = c6178c80
20/21:04:15|NIP = fb85a30  MSR = 2d000
20/21:04:15|Oops: Weird page fault, sig: 11 [#1]
20/21:04:15|PREEMPT Xilinx Virtex440
20/21:04:15|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:15|NIP: 0fb85a30 LR: 0ff85338 CTR: c01288f4
20/21:04:15|REGS: c6941f50 TRAP: 0401   Not tainted  (2.6.31)
20/21:04:15|MSR: 0002d000 <EE,PR,ME,CE>  CR: 44884044  XER: 00000000
20/21:04:15|TASK = c608e0c0[398] 'tclsh8.5' THREAD: c6940000
20/21:04:15|GPR00: 0ff85334 bf9a3870 4802b0a0 ffffffff 0fc094e0 54884044 fffffff2 0fc08d3c
20/21:04:15|GPR08: 0002d000 480240a8 00000000 bf9a3870 c6940000
20/21:04:15|NIP [0fb85a30] 0xfb85a30
20/21:04:15|LR [0ff85338] 0xff85338
20/21:04:16|Call Trace:
20/21:04:16|Kernel panic - not syncing: Fatal exception in interrupt
20/21:04:16|Call Trace:
20/21:04:16|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:16|[c6941e20] [c00345bc] panic+0x94/0x168
20/21:04:16|[c6941e70] [c000bd44] die+0x178/0x18c
20/21:04:16|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
20/21:04:16|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
20/21:04:16|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/21:07:16|
Found 0 errors checking kernel image
20/21:07:18|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/21:07:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/21:07:18|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/21:07:18|SPIconfig done now to set up the LMK3200 registers
20/21:07:18|LMK #0 : regInit[0]=0x80000000
20/21:07:18|LMK #0 : regInit[1]=0x10070600
20/21:07:18|LMK #0 : regInit[2]=0x60601
20/21:07:18|LMK #0 : regInit[3]=0x60602
20/21:07:18|LMK #0 : regInit[4]=0x60603
20/21:07:18|LMK #0 : regInit[5]=0x70624
20/21:07:18|LMK #0 : regInit[6]=0x70605
20/21:07:18|LMK #0 : regInit[7]=0x70606
20/21:07:18|LMK #0 : regInit[8]=0x70627
20/21:07:18|LMK #0 : regInit[9]=0x10000908
20/21:07:18|LMK #0 : regInit[10]=0xA0022A09
20/21:07:18|LMK #0 : regInit[11]=0x82800B
20/21:07:18|LMK #0 : regInit[12]=0x28C800D
20/21:07:18|LMK #0 : regInit[13]=0x830020E
20/21:07:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/21:07:19|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/21:07:19|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
ADCs calibrated
20/21:07:19|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
20/21:07:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/21:07:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xba 0x8a
20/21:07:29|Using Xilinx Virtex440 machine description
20/21:07:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/21:07:30|Zone PFN ranges:
20/21:07:30|  DMA      0x00000000 -> 0x00007000
20/21:07:30|  Normal   0x00007000 -> 0x00007000
20/21:07:30|Movable zone start PFN for each node
20/21:07:30|early_node_map[1] active PFN ranges
20/21:07:30|    0: 0x00000000 -> 0x00007000
20/21:07:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/21:07:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/21:07:30|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/21:07:30|PID hash table entries: 512 (order: 9, 2048 bytes)
20/21:07:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/21:07:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/21:07:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/21:07:30|Kernel virtual memory layout:
20/21:07:30|  * 0xffffe000..0xfffff000  : fixmap
20/21:07:31|  * 0xfde00000..0xfe000000  : consistent mem
20/21:07:31|  * 0xfde00000..0xfde00000  : early ioremap
20/21:07:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/21:07:31|NR_IRQS:512
20/21:07:31|clocksource: timebase mult[a00000] shift[22] registered
20/21:07:31|Console: colour dummy device 80x25
20/21:07:31|Mount-cache hash table entries: 512
20/21:07:31|NET: Registered protocol family 16
20/21:07:31|PCI: Probing PCI hardware
20/21:07:31|bio: create slab <bio-0> at 0
20/21:07:31|NET: Registered protocol family 2
20/21:07:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/21:07:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/21:07:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/21:07:31|TCP: Hash tables configured (established 4096 bind 4096)
20/21:07:31|TCP reno registered
20/21:07:31|NET: Registered protocol family 1
20/21:07:31|ROMFS MTD (C) 2007 Red Hat, Inc.
20/21:07:31|msgmni has been set to 214
20/21:07:31|io scheduler noop registered
20/21:07:31|io scheduler anticipatory registered
20/21:07:31|io scheduler deadline registered
20/21:07:31|io scheduler cfq registered (default)
20/21:07:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/21:07:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/21:07:32|console [ttyS0] enabled
20/21:07:32|brd: module loaded
20/21:07:32|loop: module loaded
20/21:07:32|Device Tree Probing 'ethernet'
20/21:07:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ba:8a
20/21:07:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/21:07:32|XLlTemac: DCR address: 0x80
20/21:07:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/21:07:32|XLlTemac: Allocating DMA descriptors with kmalloc
20/21:07:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/21:07:32|XTemac: PHY detected at address 7.
20/21:07:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/21:07:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33|Using buffer write method
20/21:07:33|cfi_cmdset_0001: Erase suspend on write enabled
20/21:07:33|cmdlinepart partition parsing not available
20/21:07:33|RedBoot partition parsing not available
20/21:07:33|Creating 5 MTD partitions on "fc000000.flash":
20/21:07:33|0x000000000000-0x000000500000 : "golden_firmware"
20/21:07:33|0x000000500000-0x000000800000 : "golden_kernel"
20/21:07:33|0x000000800000-0x000000d00000 : "user_firmware"
20/21:07:33|0x000000d00000-0x000000fe0000 : "user_kernel"
20/21:07:33|0x000000fe0000-0x000001000000 : "env_variables"
20/21:07:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/21:07:33|SPI: XIlinx spi: bus number now 32766
20/21:07:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/21:07:33|SPI: XIlinx spi: bus number now 32765
20/21:07:33|mice: PS/2 mouse device common for all mice
20/21:07:33|Device Tree Probing 'i2c'
20/21:07:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/21:07:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/21:07:33|TCP cubic registered
20/21:07:33|NET: Registered protocol family 17
20/21:07:34|RPC: Registered udp transport module.
20/21:07:34|RPC: Registered tcp transport module.
20/21:07:34|eth0: XLlTemac: Options: 0x3fa
20/21:07:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/21:07:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/21:07:34|eth0: XLlTemac: speed set to 1000Mb/s
20/21:07:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/21:07:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/21:07:36|Sending DHCP requests ., OK
20/21:07:38|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.1
20/21:07:38|IP-Config: Complete:
20/21:07:38|     device=eth0, addr=192.168.11.1, mask=255.255.255.0, gw=255.255.255.255,
20/21:07:38|     host=aida01, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/21:07:38|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01
20/21:07:38|Looking up port of RPC 100003/2 on 192.168.11.99
20/21:07:38|Looking up port of RPC 100005/1 on 192.168.11.99
20/21:07:38|VFS: Mounted root (nfs filesystem) on device 0:12.
20/21:07:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/21:07:39|Starting sysinit...
20/21:07:39|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/21:07:40|            Press 'I' to enter interactive startup.
20/21:07:40|Setting clock  (utc): Tue Apr 20 21:07:40 BST 2021 [  OK  ]
20/21:07:40|Building the cache [  OK  ]
20/21:07:40|Setting hostname aida01:  [  OK  ]
20/21:07:42|Mounting local filesystems:  [  OK  ]
20/21:07:43|Enabling /etc/fstab swaps:  [  OK  ]
20/21:07:46|Finishing sysinit...
INIT: Entering runlevel: 3
20/21:07:49|Entering non-interactive startup
20/21:07:49|FATAL: Module ipv6 not found.
20/21:07:51|Bringing up loopback interface:  [  OK  ]
20/21:07:53|FATAL: Module ipv6 not found.
20/21:07:53|Starting system logger: [  OK  ]
20/21:07:54|Starting kernel logger: [  OK  ]
20/21:07:54|Starting rpcbind: [  OK  ]
20/21:07:56|Mounting NFS filesystems:  [  OK  ]
20/21:07:57|Mounting other filesystems:  [  OK  ]
20/21:07:57|Starting xinetd: [  OK  ]
20/21:07:58|Starting midas:  Starting MIDAS Data Acquisition for aida01
20/21:07:58|xaida: device parameters: base=0x81000000 size=0x200000
20/21:08:04|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/21:08:04|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/21:08:05|xaida: driver assigned major number 254
20/21:08:05|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/21:08:10|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/21:08:10|AIDAMEM: aidamem: driver assigned major number 253
20/21:08:10|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/21:08:17|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/21:08:17|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/21:08:17|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/21:08:17|Computer Name = aida01; Temp Directory = /tmp/tcl361
20/21:08:22|package limit is not available: can't find package limit
20/21:08:25|Running with default file descriptor limit
20/21:08:25|
20/21:08:26|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting

https://docs.google.com/spreadsheets/d/1CyL5L-WLsh9fsS7CN989GC3ktqiPQOkG8YIDJrrGxYA/edit#gid=0

I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again

Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)

System Wide Checks
Clocks: OK
FADC: OK
WR: 
		 Base 		Current 	Difference
aida05 fault 	 0x31a2 : 	 0x31a2 : 	 0  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22e8 : 	 1  
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK

aida05 ASIC maybe didn't complete setup properly (WR 0x10?)

Timestamps: OK (1)
Temps: OK

Will start gzipping S460 runs for migration to lustre & tape

Beam has now stopped (end of expriment) and background measurements starts.

Some screenshot of spectra attached

ucesb is back - see attachment

Quick system checks after crash

Temperature ok.

Still cannot connect remotely to /ucesb with my gsi web login and password (?).

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida05 fault      0x31a2 :      0x31a2 :      0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      9      3      4      2      3      2      1      4      2      3      7   : 39804
aida02 :     29      4      2      3      0      3      1      4      2      3      7   : 39828
aida03 :     20      6      1      2      2      3      1      4      2      3      7   : 39888
aida04 :     19      5      4      2      1      3      1      4      2      3      7   : 39860
aida05 :     27      5      4      1      2      3      2      3      1      4      7   : 40692
aida06 :     17     10      3      2      1      3      1      4      2      3      7   : 39876
aida07 :     22      7      2      2      2      3      1      4      2      3      7   : 39920
aida08 :     12      6      5      1      3      2      1      4      2      3      7   : 39824
aida09 :      0      9      1      3      1      3      1      4      2      3      7   : 39800
aida10 :      2      2      2      3      3      2      3      2      2      3      7   : 39256
aida11 :     21      7      4      3      0      3      2      3      2      3      7   : 39596
aida12 :     18      8      3      0      2      3      2      3      2      3      7   : 39608

Looks like the aida-gsi window manager/login session crashed.

On site S460 shift restarted AnyDesk. Upon re-connection *all* browsers, terminals, applications were gone
I have re-started the basics but BNC PB-5 pulser, CAEN NIM bin, MBS rate monitor, system console monitor etc
still need to be re-started. 

Extract from /var/log/messages

Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal cpuset=/ mems_allowed=0
Apr 24 03:19:01 aidas-gsi kernel: Pid: 18955, comm: gnome-terminal Not tainted 2.6.32-696.el6.x86_64 #1
Apr 24 03:19:01 aidas-gsi kernel: Call Trace:
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131550>] ? dump_header+0x90/0x1b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8123c1dc>] ? security_real_capable_noaudit+0x3c/0x70
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811319d2>] ? oom_kill_process+0x82/0x2a0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131911>] ? select_bad_process+0xe1/0x120
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131e10>] ? out_of_memory+0x220/0x3c0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113e7ec>] ? __alloc_pages_nodemask+0x93c/0x950
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81015533>] ? native_sched_clock+0x13/0x80
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff810aaea1>] ? lock_hrtimer_base+0x31/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81177a3a>] ? alloc_pages_current+0xaa/0x110
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113bb4e>] ? __get_free_pages+0xe/0x50
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1944>] ? __pollwait+0xb4/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81348684>] ? n_tty_poll+0x194/0x1d0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff813440aa>] ? tty_poll+0x8a/0xa0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1db5>] ? do_sys_poll+0x2d5/0x5b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1890>] ? __pollwait+0x0/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1745>] ? poll_select_set_timeout+0x95/0xb0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b2281>] ? sys_poll+0x71/0x100
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Apr 24 03:19:01 aidas-gsi kernel: Mem-Info:
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 172
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  45
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd:  97
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 142
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  49
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd:  56
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 179
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd:  67
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 107
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 170
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  54
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 118
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 196
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  36
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  37
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd: 128
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 173
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd: 161
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 177
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 178
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  99
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 183
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 162
Apr 24 03:19:01 aidas-gsi kernel: active_anon:3408671 inactive_anon:401328 isolated_anon:128
Apr 24 03:19:01 aidas-gsi kernel: active_file:22995 inactive_file:61739 isolated_file:0
Apr 24 03:19:01 aidas-gsi kernel: unevictable:0 dirty:59856 writeback:22 unstable:0
Apr 24 03:19:01 aidas-gsi kernel: free:35843 slab_reclaimable:15134 slab_unreclaimable:65491
Apr 24 03:19:01 aidas-gsi kernel: mapped:129529 shmem:249820 pagetables:24586 bounce:0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA free:15752kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 874 16024 16024
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 free:64204kB min:3680kB low:4600kB high:5520kB active_anon:369500kB inactive_anon:377436kB active_file:10988kB 
inactive_file:26828kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:895084kB mlocked:0kB dirty:26856kB writeback:88kB mapped:35896kB shmem:39156kB 
slab_reclaimable:4224kB slab_unreclaimable:4344kB kernel_stack:3344kB pagetables:2032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:65792 all_unreclaimable? 
no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 15150 15150
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal free:63416kB min:63836kB low:79792kB high:95752kB active_anon:13265184kB inactive_anon:1227876kB active_file:80992kB 
inactive_file:220128kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15513600kB mlocked:0kB dirty:212568kB writeback:0kB mapped:482220kB 
shmem:960124kB slab_reclaimable:56312kB slab_unreclaimable:257620kB kernel_stack:17600kB pagetables:96312kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:17824 all_unreclaimable? no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 0 0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32: 373*4kB 559*8kB 428*16kB 296*32kB 189*64kB 111*128kB 43*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 64204kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal: 16100*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64408kB
Apr 24 03:19:01 aidas-gsi kernel: 357788 total pagecache pages
Apr 24 03:19:01 aidas-gsi kernel: 22930 pages in swap cache
Apr 24 03:19:01 aidas-gsi kernel: Swap cache stats: add 3720478, delete 3697548, find 5026852/5228898
Apr 24 03:19:01 aidas-gsi kernel: Free swap  = 1826100kB
Apr 24 03:19:01 aidas-gsi kernel: Total swap = 8200188kB
Apr 24 03:19:01 aidas-gsi kernel: 4194303 pages RAM
Apr 24 03:19:01 aidas-gsi kernel: 97868 pages reserved
Apr 24 03:19:01 aidas-gsi kernel: 578727 pages shared
Apr 24 03:19:01 aidas-gsi kernel: 3846484 pages non-shared
Apr 24 03:19:01 aidas-gsi kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 24 03:19:01 aidas-gsi kernel: [ 1388]     0  1388     2819      130   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 3927]     0  3927     7442      128  12     -17         -1000 auditd
Apr 24 03:19:01 aidas-gsi kernel: [ 3961]     0  3961    62914      755  15       0             0 rsyslogd
Apr 24 03:19:01 aidas-gsi kernel: [ 4029]     0  4029     4604      141   7       0             0 irqbalance
Apr 24 03:19:01 aidas-gsi kernel: [ 4054]    32  4054     4746      124  13       0             0 rpcbind
Apr 24 03:19:01 aidas-gsi kernel: [ 4074]    81  4074     8194      440   6       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 4089]     0  4089    26490     2647   4       0             0 NetworkManager
Apr 24 03:19:01 aidas-gsi kernel: [ 4096]     0  4096    14534      328  12       0             0 modem-manager
Apr 24 03:19:01 aidas-gsi kernel: [ 4112]    29  4112     5839       97   0       0             0 rpc.statd
Apr 24 03:19:01 aidas-gsi kernel: [ 4151]     0  4151    47297      117  15       0             0 cupsd
Apr 24 03:19:01 aidas-gsi kernel: [ 4171]     0  4171    11251       75   9       0             0 wpa_supplicant
Apr 24 03:19:01 aidas-gsi kernel: [ 4180]     0  4180     2280      162   9       0             0 dhclient
Apr 24 03:19:01 aidas-gsi kernel: [ 4210]     0  4210     1020       82   8       0             0 acpid
Apr 24 03:19:01 aidas-gsi kernel: [ 4222]    68  4222     9968      569  15       0             0 hald
Apr 24 03:19:01 aidas-gsi kernel: [ 4223]     0  4223     5100      184  12       0             0 hald-runner
Apr 24 03:19:01 aidas-gsi kernel: [ 4271]     0  4271     5630      157  12       0             0 hald-addon-inpu
Apr 24 03:19:01 aidas-gsi kernel: [ 4272]    68  4272     4502      120   0       0             0 hald-addon-acpi
Apr 24 03:19:01 aidas-gsi kernel: [ 4306]     0  4306    96540      161   0       0             0 automount
Apr 24 03:19:01 aidas-gsi kernel: [ 4492]     0  4492    26831       15   9       0             0 rpc.rquotad
Apr 24 03:19:01 aidas-gsi kernel: [ 4497]     0  4497     5450      213   4       0             0 rpc.mountd
Apr 24 03:19:01 aidas-gsi kernel: [ 4542]     0  4542     5776       42   4       0             0 rpc.idmapd
Apr 24 03:19:01 aidas-gsi kernel: [ 4567]     0  4567    16573      113   3     -17         -1000 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 4578]     0  4578     5429      117   7       0             0 xinetd
Apr 24 03:19:01 aidas-gsi kernel: [ 4589]    38  4589     7686      213  10       0             0 ntpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4675]     0  4675    20260      147   5       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [ 4682]    89  4682    20323      141   5       0             0 qmgr
Apr 24 03:19:01 aidas-gsi kernel: [ 4696]     0  4696    27153      253   0       0             0 ksmtuned
Apr 24 03:19:01 aidas-gsi kernel: [ 4709]     0  4709    29222      137  10       0             0 crond
Apr 24 03:19:01 aidas-gsi kernel: [ 4724]     0  4724     5277       50  12       0             0 atd
Apr 24 03:19:01 aidas-gsi kernel: [ 4741]     0  4741   172796     2328   0       0             0 virt-who
Apr 24 03:19:01 aidas-gsi kernel: [ 4762]     0  4762   256599      407  10       0             0 libvirtd
Apr 24 03:19:01 aidas-gsi kernel: [ 4858]     0  4858    16126      150  10       0             0 certmonger
Apr 24 03:19:01 aidas-gsi kernel: [ 4903]    99  4903     3223      102   2       0             0 dnsmasq
Apr 24 03:19:01 aidas-gsi kernel: [ 4918]     0  4918     4539      223   5       0             0 dhcpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4920]     0  4920    33509      312  11       0             0 gdm-binary
Apr 24 03:19:01 aidas-gsi kernel: [ 4927]     0  4927     1016       61  11       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4929]     0  4929     1016       61  12       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4931]     0  4931     1016       61   9       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4933]     0  4933     1016       61  15       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4935]     0  4935     1016       61   1       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4944]     0  4944     3148      440   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 4945]     0  4945     3148      438   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 5050]     0  5050  1029198      377   3       0             0 console-kit-dae
Apr 24 03:19:01 aidas-gsi kernel: [ 5126]     0  5126    12508      365   3       0             0 devkit-power-da
Apr 24 03:19:01 aidas-gsi kernel: [ 5164]     0  5164    45435      581   1       0             0 polkitd
Apr 24 03:19:01 aidas-gsi kernel: [ 5175]   499  5175    42114      146   0       0             0 rtkit-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5702]     0  5702    14009      514   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5725]     0  5725    11306       74   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18391]     0 18391    41240      293   8       0             0 gdm-simple-slav
Apr 24 03:19:01 aidas-gsi kernel: [18393]     0 18393  3476635  2412883  13       0             0 Xorg
Apr 24 03:19:01 aidas-gsi kernel: [18427]    42 18427     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18464]     0 18464    47926      269   0       0             0 gdm-session-wor
Apr 24 03:19:01 aidas-gsi kernel: [18510]   500 18510    57281      424   1       0             0 gnome-keyring-d
Apr 24 03:19:01 aidas-gsi kernel: [18520]   500 18520    64493      751   0       0             0 gnome-session
Apr 24 03:19:01 aidas-gsi kernel: [18528]   500 18528     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18529]   500 18529     8211      379   8       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18604]   500 18604    33379     1113  13       0             0 gconfd-2
Apr 24 03:19:01 aidas-gsi kernel: [18612]   500 18612   139831     1984   9       0             0 gnome-settings-
Apr 24 03:19:01 aidas-gsi kernel: [18614]   500 18614    71643      696  12       0             0 seahorse-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18616]   500 18616    34351      298  13       0             0 gvfsd
Apr 24 03:19:01 aidas-gsi kernel: [18636]   500 18636   112427     2321   4       0             0 metacity
Apr 24 03:19:01 aidas-gsi kernel: [18637]   500 18637   203008     6568  10       0             0 gnome-panel
Apr 24 03:19:01 aidas-gsi kernel: [18638]   500 18638   326834    16727  10       0             0 nautilus
Apr 24 03:19:01 aidas-gsi kernel: [18640]   500 18640   141364      341  14       0             0 bonobo-activati
Apr 24 03:19:01 aidas-gsi kernel: [18642]   500 18642    65558      741  10       0             0 bluetooth-apple
Apr 24 03:19:01 aidas-gsi kernel: [18645]   500 18645    68687      968   4       0             0 gnome-power-man
Apr 24 03:19:01 aidas-gsi kernel: [18646]   500 18646     7601      812  12       0             0 restorecond
Apr 24 03:19:01 aidas-gsi kernel: [18648]   500 18648    28879      401   5       0             0 im-settings-dae
Apr 24 03:19:01 aidas-gsi kernel: [18649]   500 18649   132278     1323   4       0             0 gnome-volume-co
Apr 24 03:19:01 aidas-gsi kernel: [18654]   500 18654    75318     2282  10       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18657]   500 18657   119660     1244   2       0             0 trashapplet
Apr 24 03:19:01 aidas-gsi kernel: [18659]   500 18659    88087     3906   5       0             0 wnck-applet
Apr 24 03:19:01 aidas-gsi kernel: [18660]   500 18660    62988      653   7       0             0 vino-server
Apr 24 03:19:01 aidas-gsi kernel: [18661]   500 18661    72490      958   0       0             0 polkit-gnome-au
Apr 24 03:19:01 aidas-gsi kernel: [18662]   500 18662    82733     1441   5       0             0 nm-applet
Apr 24 03:19:01 aidas-gsi kernel: [18666]   500 18666   126400      635   1       0             0 gvfs-gdu-volume
Apr 24 03:19:01 aidas-gsi kernel: [18667]   500 18667    74660     1042   5       0             0 gpk-update-icon
Apr 24 03:19:01 aidas-gsi kernel: [18669]   500 18669    64033      850   0       0             0 gdu-notificatio
Apr 24 03:19:01 aidas-gsi kernel: [18676]   500 18676    81107     2344   6       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [18693]   500 18693   131702     2471   3       0             0 pulseaudio
Apr 24 03:19:01 aidas-gsi kernel: [18823]   500 18823    58170      269   4       0             0 gvfs-afc-volume
Apr 24 03:19:01 aidas-gsi kernel: [18836]   500 18836    37789      300   8       0             0 gvfs-gphoto2-vo
Apr 24 03:19:01 aidas-gsi kernel: [18844]   500 18844    36051      437   0       0             0 gvfsd-trash
Apr 24 03:19:01 aidas-gsi kernel: [18868]   500 18868    10275      287   6       0             0 gconf-im-settin
Apr 24 03:19:01 aidas-gsi kernel: [18869]   500 18869    84055     4872   3       0             0 gnome-screensav
Apr 24 03:19:01 aidas-gsi kernel: [18870]   500 18870     2057       88   3       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18871]   500 18871    27123      198   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18894]   500 18894    98681     1288   0       0             0 gdm-user-switch
Apr 24 03:19:01 aidas-gsi kernel: [18896]   500 18896    72800      911   3       0             0 notification-ar
Apr 24 03:19:01 aidas-gsi kernel: [18897]   500 18897   143660     1846   2       0             0 clock-applet
Apr 24 03:19:01 aidas-gsi kernel: [18900]   500 18900    23718      242   3       0             0 gconf-helper
Apr 24 03:19:01 aidas-gsi kernel: [18918]   500 18918    34358      265   0       0             0 gvfsd-burn
Apr 24 03:19:01 aidas-gsi kernel: [18932]   500 18932    33897      307   4       0             0 gvfsd-metadata
Apr 24 03:19:01 aidas-gsi kernel: [18955]   500 18955    82196     7037   7       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18957]   500 18957     2057       97   7       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18977]   500 18977    27123      114   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18991]     0 18991    47361      130  10       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [18992]     0 18992    27524      290  15       0             0 iptraf
Apr 24 03:19:01 aidas-gsi kernel: [21973]   500 21973  1058234   192783   0       0             0 firefox
Apr 24 03:19:01 aidas-gsi kernel: [21987]   500 21987    68608      284   3       0             0 at-spi-bus-laun
Apr 24 03:19:01 aidas-gsi kernel: [22221]   500 22221   731047    34381   0       0             0 Privileged Cont
Apr 24 03:19:01 aidas-gsi kernel: [22248]   500 22248   705574    10970   0       0             0 WebExtensions
Apr 24 03:19:01 aidas-gsi kernel: [22335]   500 22335   960920   166074   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22407]   500 22407  1023120    77418   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22810]   500 22810    27123      114   2       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [22826]   500 22826    15487      185   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [22830]   500 22830    27123      114   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26234]   500 26234   839272    43450   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [26311]   500 26311    27123      282   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26884]   500 26884    27123      115   5       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19388]   500 19388    27123      293   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11683]   500 11683   791783    18526   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [ 4876]   500  4876    54985      304   5       0             0 gvfsd-computer
Apr 24 03:19:01 aidas-gsi kernel: [17248]   500 17248    29595      100   1       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17249]   500 17249    29631      157  10       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17250]   500 17250    27124      114   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17266]   500 17266    15489      181   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [19879]     0 19879    25531      154   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19883]   500 19883    25531      163   2       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19884]   500 19884    27638      106  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19907]   500 19907    29595       97   5       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [14850]   500 14850    56156      327   2       0             0 gvfsd-network
Apr 24 03:19:01 aidas-gsi kernel: [14855]   500 14855    36479      318   3       0             0 gvfsd-dnssd
Apr 24 03:19:01 aidas-gsi kernel: [22615]   500 22615    27123      117   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [13867]     0 13867    25531      161   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13880]   500 13880    25645      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13881]   500 13881    14400      138   8       0             0 sftp-server
Apr 24 03:19:01 aidas-gsi kernel: [20258]   500 20258   776888    93602   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [25910]   500 25910    15050      209   3       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [26010]   500 26010     2879       95  15       0             0 tinyproxy
Apr 24 03:19:01 aidas-gsi kernel: [26513]     0 26513    25530      204  12       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [26517]   500 26517    25530      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29203]     0 29203    25016      164  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29207]   500 29207    25534      158   6       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 1105]   500  1105    27123      143   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17163]     0 17163    25016      190  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [17167]   500 17167    25534      185   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18690]     0 18690    25530      183   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18695]   500 18695    25530      138   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18696]   500 18696    27123      127   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [20802]     0 20802    25530      173   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20806]   500 20806    25530      142   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20807]   500 20807    27123      172   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 9697]     0  9697    25530      187   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9701]   500  9701    25563      184  10       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9703]   500  9703    27123      132   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [21737]   500 21737    29350      360  10       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [21860]   500 21860   166837     1623   5       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21861]   500 21861   208277     3415   0       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21868]   500 21868   470786     1884   2       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [20297]   500 20297   731145    44730   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [23452]     0 23452    47361      305   5       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [23453]     0 23453    25240       86   2       0             0 tail
Apr 24 03:19:01 aidas-gsi kernel: [23841]   500 23841    27123      243   3       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [  345]   500   345    27156      311  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 4825]   500  4825   682055    25908   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [16950]   500 16950    26921      257   1       0             0 watch
Apr 24 03:19:01 aidas-gsi kernel: [10822]   500 10822  1060110   432573   3       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [29109]   500 29109    27186      482   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 3754]   500  3754    27123      375   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11083]   500 11083    28030      216   3       0             0 HTTPD@8115
Apr 24 03:19:01 aidas-gsi kernel: [11087]   500 11087    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11089]   500 11089    36731     3211  12       0             0 tclsh8.5_copy2
Apr 24 03:19:01 aidas-gsi kernel: [11097]   500 11097    28030      207   1       0             0 TapeServer
Apr 24 03:19:01 aidas-gsi kernel: [11104]   500 11104     5398     1567  10       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [11105]   500 11105    25231      134  12       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11106]   500 11106     4323      187   6       0             0 stats
Apr 24 03:19:01 aidas-gsi kernel: [11107]   500 11107     4918      219   5       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11108]   500 11108     4917      194   9       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11109]   500 11109     4917      194  12       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11110]   500 11110     4630      214   0       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11111]   500 11111     4630      182   9       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11114]   500 11114    28030      217  12       0             0 NewMerger
Apr 24 03:19:01 aidas-gsi kernel: [11124]   500 11124   404629     8857   0       0             0 master64
Apr 24 03:19:01 aidas-gsi kernel: [11125]   500 11125    25231      135   9       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11126]   500 11126    19761    17318  13       0             0 statrate64
Apr 24 03:19:01 aidas-gsi kernel: [11127]   500 11127    35968    11059   2       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11128]   500 11128    35968    11498  11       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11129]   500 11129    35968    11581  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11130]   500 11130    35968    12366   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11131]   500 11131    35968    10031   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11132]   500 11132    35968    10689   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11133]   500 11133    35968    11896   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11134]   500 11134    35968    10510  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11135]   500 11135    35968    15679  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11136]   500 11136    35968    14623  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11137]   500 11137    35968    12790  15       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11138]   500 11138    35968    11683   0       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11139]   500 11139    35968     8773   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11140]   500 11140    35968     8774  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11141]   500 11141    35968     8772  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11142]   500 11142    35968     8773  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11152]   500 11152   307392    47667   5       0             0 merge64
Apr 24 03:19:01 aidas-gsi kernel: [11200]   500 11200   723346    37902   5       0             0 file:// Content
Apr 24 03:19:01 aidas-gsi kernel: [11338]   500 11338    28030      216   0       0             0 HTTPD
Apr 24 03:19:01 aidas-gsi kernel: [11342]   500 11342    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11344]   500 11344    46965    14788   3       0             0 tclsh8.5
Apr 24 03:19:01 aidas-gsi kernel: [12835]   500 12835   766232    71066   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [24071]   500 24071    27123      409   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [24347]   500 24347    28030      231   4       0             0 datarelaymbs
Apr 24 03:19:01 aidas-gsi kernel: [24351]   500 24351     7751     1313  10       0             0 DataRelayMBS
Apr 24 03:19:01 aidas-gsi kernel: [  407]    89   407    20280      470  13       0             0 pickup
Apr 24 03:19:01 aidas-gsi kernel: [ 1586]     0  1586     4238      156  13       0             0 anacron
Apr 24 03:19:01 aidas-gsi kernel: [ 1876]     0  1876    25231      144  10       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: Out of memory: Kill process 18393 (Xorg) score 523 or sacrifice child
Apr 24 03:19:01 aidas-gsi kernel: Killed process 18393, UID 0, (Xorg) total-vm:13906540kB, anon-rss:9558800kB, file-rss:92712kB
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11471" (uid=500 pid=1884 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11473" (uid=500 pid=1887 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:45 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:19:47 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1961 of process 1961 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Apr 24 03:19:48 aidas-gsi pulseaudio[1961]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1965 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1966 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1967 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11485" (uid=500 pid=1981 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11487" (uid=500 pid=1983 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: DNS-SD initialization failed: Daemon not running
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: init gpgme version 1.1.8
Apr 24 03:30:44 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2243 of process 2243 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:45 aidas-gsi pulseaudio[2243]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2301 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2302 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2303 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2325 of process 2325 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2325]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2326 of process 2326 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2326]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11506" (uid=500 pid=2328 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2418 of process 2418 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2418]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11508" (uid=500 pid=2420 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:51 aidas-gsi pulseaudio[2243]: ratelimit.c: 25 events suppressed
Apr 24 03:30:51 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr 24 03:33:11 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11521" (uid=500 pid=3005 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11524" (uid=500 pid=3012 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11525" (uid=500 pid=3015 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11527" (uid=500 pid=3018 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11529" (uid=500 pid=3022 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11531" (uid=500 pid=3024 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:13 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11533" (uid=500 pid=3100 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:37 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:40:46 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:41:22 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:19 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:43:31 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:47:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.7 (192.168.11.99) from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.4 (192.168.11.99) from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.3 (192.168.11.99) from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.8 (192.168.11.99) from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.2 (192.168.11.99) from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.1 (192.168.11.99) from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.6 (192.168.11.99) from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.5 (192.168.11.99) from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:816 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:952 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:884 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:895 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:892 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:839 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:897 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.8
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.1
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.7
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.3
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.4
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.2
Apr 24 03:51:04 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.5
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:29 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:06 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.9 (192.168.11.99) from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.10 (192.168.11.99) from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.11 (192.168.11.99) from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.12 (192.168.11.99) from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:814 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:852 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:694 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:770 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.9
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.10
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.12
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.11
Apr 24 03:52:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)

AIDA PC crashe completely at ~3:30am CET.

Tom was called and he Restarted the AIDA system at ~4am with Run 52.

Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.

system wide checks:

Clock status:
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit

         Base         Current     Difference
aida06 fault      0x679f :      0x67a4 :      5  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error page not working !!

Memory information:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     15      4      6      4      3      2      4      2      3      6   : 36420
aida02 :     36     25     24      6      0      4      2      3      2      3      4   : 28056
aida03 :     17      7      4      3      3      5      2      2      3      3      6   : 36444
aida04 :      4      5      4      5      3      4      2      3      3      3      6   : 36824
aida05 :     23      9      4      2      3      3      1      3      3      3      6   : 36452
aida06 :     26      8      5      1      3      4      2      2      3      3      6   : 36312
aida07 :     21      9      3      3      4      5      1      2      3      3      6   : 36268
aida08 :     27      9      2      2      2      4      1      3      3      3      6   : 36500
aida09 :     13      4      3      3      1      2      4      2      3      3      6   : 36388
aida10 :      1     10      0      3      1      2      3      3      2      3      6   : 35572
aida11 :     23     13      8      3      3      4      3      3      2      3      6   : 36196
aida12 :     19      7      4      3      2      3      2      3      3      3      6   : 36644

21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378

16:00 taking over from Muneerah

16:15 - temp, ucesb, database, stats etc. look ok.

16:41 - Nic restarting MBS

17:05 - temp, ucesb, database, stats etc. look ok.

17:43 - temp, ucesb, database, stats etc. look ok.

18.10 - system wide checks.  All same as previous checks - eg elog 264 + 266

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
 

Check FPGA Timestamp Error - HTML errors

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      7      4      7      4      3      2      4      2      3      6   : 36388
aida02 :      1      2      5      2      0      4      3      4      3      4      3   : 27044
aida03 :     28      4      4      4      4      4      2      2      3      3      6   : 36432
aida04 :      5      5      2      6      2      3      3      3      3      3      6   : 36892
aida05 :     24      4      7      3      3      3      2      2      3      3      6   : 36240
aida06 :     21      6      6      3      3      4      2      2      3      3      6   : 36356
aida07 :     15      7      4      3      3      4      2      2      3      3      6   : 36308
aida08 :     23      6      7      1      3      4      1      3      3      3      6   : 36572
aida09 :      1      8      2      1      1      2      4      2      3      3      6   : 36292
aida10 :      3      4      6      2      2      2      3      3      2      3      6   : 35660
aida11 :     15      7      4      1      2      3      2      4      2      3      6   : 36052
aida12 :     25     13      5      2      2      3      2      3      3      3      6   : 36700
 

18:45 - temp, ucesb, database, stats etc. look ok.

19:03 - temp, ucesb, database, stats etc. look ok.

19:26  - temp, ucesb, database, stats etc. look ok.

TapeData (df -h) - 7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7

19:30 - beam being shared with R3B, implantation rates fluctuation by~  50% back to normal 19:40

19:50 system checks - all as previous

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

    
         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Check FPGA Timestamp Error - HTML errors

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     25      6      4      7      4      3      2      4      2      3      6   : 36404
aida02 :     20      7      5      1      0      4      2      3      2      3      4   : 27384
aida03 :     28      6      3      5      3      5      2      2      3      3      6   : 36528
aida04 :     19      8      4      6      2      4      4      2      3      3      6   : 36876
aida05 :      7      3      8      3      4      3      1      3      3      3      6   : 36500
aida06 :     25      8      4      3      3      4      2      3      3      3      6   : 36868
aida07 :     18      7      3      4      3      4      2      2      3      3      6   : 36336
aida08 :     16     10      6      2      3      3      1      3      3      3      6   : 36464
aida09 :      0      3      1      1      1      3      4      2      3      3      6   : 36360
aida10 :      8      1     10      0      2      2      2      4      2      3      6   : 35912
aida11 :      2      2      2      3      3      4      3      3      2      3      6   : 35928
aida12 :     12      7      4      1      3      3      2      3      3      3      6   : 36616