10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11

08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.

      They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
      They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID

08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2

09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
      Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
      Data rate 16.3MB/s
      ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.

09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage

09:47 Beam back turned off no storage. On file R1_10

11:04 Now that we have implants on the detector we get a better look at the strips
      From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
      This adaptor board corresponds to FEE11

12:17 Leakage currents screenshot 512.png

13:30 Beam off. Expected back at 15:00...

14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
      They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
      They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
      Rates are as before. No better, no worse.

15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...

08:00 OH Takes over

      Statistics still look good from yesterday. Rate is still down.
      Dead time in FEEs 2 and 4 is 2.04%

      System wide checks clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x500 : 	 38  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      *Baseline reset*
      FPGA check ok
      Stats - attachment 1
      Temp - attachment 2
      Bias - attachment 3
      

08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips.
      I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4
      The results of the test appear to indicate that it is the network link that we are saturating causing dead time
***	Before		                  After	
FEE	DT (s)  Dead time %	           DT (s)	Dead time %
0	0.013	0.004184451	           0.026	0.010025681
1	6.352	2.04458693	           4.097	1.57981599
2	0.295	0.09495484	           0.292	0.112596112
3	6.349	2.043621288	           8.255	3.183153771
4	0.021	0.006759497	           0.03	        0.011568094
5	0.285	0.091736032	           3.388	1.306423377
6	0.02	0.006437616	           0.149	0.057454865
7	1.219	0.392372712	           2.133	0.822491459
8	1.137	0.365978485	           0.457	0.176220627
9	0	0	                       0	0
10	1.633	0.525631369	           1.545	0.595756823
11	0.061	0.01963473	           0.166	0.064010118
12	0.059	0.018990968	           0.085	0.032776265
13	0	0	                       0	0
14	0.016	0.005150093	           0.023	0.008868872
15	0.282	0.09077039	           0.168	0.064781324


      Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased

09:00 They are stopping the run to decrease the bias on the bPlast detector
09:25 No noticable change in AIDA
      They will now begin to change the spill structure.

09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change

10:21 Beam is back on AIDA

10:39 System wide checks. 
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x506 : 	 6  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

11:00 
      Stats - attachment 4
      Temp - attachment 5
      Bias - attachment 6

12:46 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50b : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

14:36 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50f : 	 15  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

DSSSD bias OFF. DAQ STOPped

Attachment 6 FEE64 temps OK

Attachment 1-5 system wide checks OK 

Re-assembled triple snout re-installed on the AIDA stand - DSSSD ribbon cables and grounds re-connected.
FEE64 adaptor PCB LKs per https://elog.ph.ed.ac.uk/DESPEC/409

Attachments 1 & 2 1.8.L spectra
  p+n FEE64 aida10 pulser peak width 97 ch FWHM
  n+n FEE64 aida04 pulser peak width 550 ch FWHM

Attachment 3 rate spectra
  missing channels aida01, aida07 & aida11 - misaligned ERNI and/or Samtec connectors?

Attachments 4 & 5 1.8.W spectra 

Attachment 6 ADC data item rates
 aida10 OK, all others c. 300-400k

Attachment 7 FEE64 temperatures OK

Attachment 8 DSSSD bias & leakage currents OK

08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
      It is currently sending data pause requests and data push requests but nothing else - attachment 1
      telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
      FEE appears to have hung. - attachment 2
      Still passes all system wide checks.
      When performing a check load can see the tclsh process take up CPU usage

09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
      AIDA14 recovered but ASIC2 still missing
    
      Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
      Given access would change the adapter board on AIDA05.
      Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
      Would retain the option of a speedybacktrack

      Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
      A threshold of 0x4 reduced the rate in the HEC channels considerably
      A threshold of 0x6 removes entirely

      Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned

09:59 Had access so replaced the adapter card on AIDA05
      Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
      This could also have been the cause


10:09 Restarted DAQ and there is no change to the behavior of AIDA05
      Still see events in HEC
      Again raising threshold to 0x5 in ASIC1 largely removes them. 

13.10 Attachments 3-5
      DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
      ASIC check load
      ASIC slow comparator all FEE64s all ASICS 0xa
      ADC data item stats

13.17 Attachments 6-9
      DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
      ADC data item stats
      NewMerger/Tape Server

18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
      It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2

      To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
      After powering on it was found there was no improvement and possible worse noise than before

      We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14

ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2

 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6

Required packages not included as default

• csh (For running MIDAS scripts)
• xinetd (Runs the rdate server for the FEEs)
• Telnet (Communication with the FEEs)
• dhcp

Configuration changes

• FEEs uses NFS2 which is no longer enabled by default
  • Must edit config in /etc/nfs.conf to uncomment the following lines:
    • [nfsd]
    • vers2=y
• /etc/xinetd.d/time-stream must be edited by uncommenting

  • disable        = no

• Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

  • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

  • Likewise can check status with `systemctl status $SERVICE_NAME`

• Also need to check the firewall status of the ethernet port connected to the FEEs.

  • Easiest solution is to set the port to be trusted

Optional extras

• For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
  • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
  • Can also add screenshot to this - Attachment 2
• Gnome tweaks also allows the number of workspaces to be changed
• Also recommend setting workspaces to span displays

An interesting find with the new sum inverter is that with the pulser at default settings the pulser peak on the n+n side is above the range of the ADC.
We have checked signals from the sum inverter and pulser and both are equal in magnitude.
We therefor determine it is over range as the charge is spread over 4 FEEs vs 12 on the p+n side.
This possibly wasn't an issue with the previous inverter if it produced a lower amplitude.

For tests going forward p+n voltage will be 2V. n+n voltage will be 0.5V - Attachment 1
Going to look into a 20dB attenuator

Seems there were issues with the lemo cables in the n+n pulser chain

Pulser spectra

• Changes to the nn pulser circuit yesterday have improved the peak widths on the p+n side
• p+n peak widths averact around 100 channels
• n+n around 1000 at 0xa threshold
• n+n 1.8.L spectra - attachment 1
• p+n 1.8.L spectra attachment 2

p+n peak widths

n+n peak widths

Daily checks

• Fee temperatures ok - attachment 3
• Bias and leakage currents ok - attachment 4
• Stats spectra with current grounding - attachment 5
  • Current grounding is just a cable going between each adaptor board and its respective FEE
  • Interfee grounding done by power supply and also pulser circuit
• Rates for each of the asics - attachment 6
• p+n waveforms - attachment 7
• n+n waveforms - attachment 8
  • Lots of noise evident on them

Plans for the day

• Connect FEEs up to the copper earth bars installed
• Powercycle and check noise levels
• If n+n remains significantly higher investigate the n+n pulser circuit further
• Investigate the effects of different jumper connetions At present n+n have all LK jumpers on and p+n have lk2-4

Grounding implemented

• Each FEE given direct connection to heavy duty earth bar via adaptor board lemo ground
• Earth bars are then connected to platform ground on AIDA frame
• Photo of grounding - attachment 1
• Improvements not as large as expected from previous experience though
  • Particularly in the n+n side where widths still appear awful
• While implementing the grounding it was noticed the ground lemo socket on FEE7 was damaged and no longer gripped onto a lemo cable
  • It was noted that it would need changing at some point but continued with it for the time being
• Jumper configuration was n+n LK1-4 all on, p+n LK2-4 all on

p+n widths

• Compared to the width with no grounding the FEEs for the first DSSD (First 6) range from no change to slightly worse
• 1.8.L spectra - attachment 3

n+n widths

• Widths still very large
• Width for 8 is around 500
• 1.8.L attachment 2

Additional info

• Statistics - attachment 4

Jumper optimisation

• Went to our previously found optimal jumper configuration for the trips
• n+n LK1 only
• p+n LK 2 and 4 on all FEE
• p+n LK3 on non isolated cables (PCB ground on lower middle cable and n+n side bias on top middle cable)
• While changing the jumpers we also swapped out the damaged adaptor board on aida07 for a new one
• Saw an improvement for most FEEs

• Unfortunately saw no improvement for the n+n
• We event tested having the pulser only going into a single FEE (aida08)
• Still have a width of 622 for that FEE
• To compare FEEs we have layout 1 - attachment 5
• Layouot 1 on a common timescale - attachment 6
  • Note that the rate is typically dominated by a single strip or a couple of strips within a FEE
  • Many channels are just showing the pulser rate 25Hz
• p+n  1.8.L attachment 7
• p+n waveforms attachment 8
• n+n waveforms attachment 9
• Testing the noise on the n+n strips we observe noise with slow comparator set to 200 (2MeV)
• Meanwhile the p+n strips are silent at 0x64 for the most part.
• DAQ left running overnight with 0x64 on ASIC settings to see if we observe any alphas

Plan for the day

• Test daisy chaining the n+n across to the other n+n strips
  • Currently the n+n braid is only plugged into either aida06 or aida08 (depending on DSSD)
  • All of the jumpers are however grounded
  • Will still test
  • May investigate also trying this on LK3 on adaptors for aida01 and aida03
• Observe the output of the new sum inverter on the scope to check for variation in amplitude

Tests performed

• After leaving overnight noticed the pulser was not fully inserted into aida09
  • Little change in rates observed in 9 but otherwise nothing else
• Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
  • No change observed in rates
• Also tested covering the snout in light tight material
  • No Changes
• Unplugged the system monitor from 6 and 8, no change
• Connected the snout to ground with copper tape
  • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
  • Removed the ground and powercycled and rates restored to where previously
• Changed the bias supply jumper inside to have the bias line connected to the rack ground
  • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
    • Recrimped and inserted again
  • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
  • Pulser widths however larger than yesterday

p+n width

n+n width

• Connected up entire pulser circuit to n+n FEEs

Switching bias supply back

• because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
• DAQ was then left overnight as had to leave to catch the bus
• Rates are observed to be back at their higher values - attachment 2
• n+n 1.8.L spectra - attachment 3
• p+n 1.8.L spectra - attachment 4

p+n widths

n+n widths

• The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

n+n

• Don't see a pulser peak in 4 or 2
• Will replace cable between 4 and 2 and 2 and 6
• Statistics at 0x64 with pulser on - atachment 5
• Now see peaks - attachment 6
• Statistics at 0xa - attachment 7

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

• Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
• 0x64 stats - attachment 7
• 0xa statistics - attachment 8
• p+n 1.8.L - attachment 9
• p+n waveform - attachment 10
• n+n waveform - attachment 11
  • Same high frequency component in both
  • 35 channel spacing corresponds to a frequency of ~1.4MHz

Next tests

• Returned braid to FEE 4 and 8
  • Recovered earlier rates and performance
• Re-added the ground cable to the snout
  • Much worse noise performance across all FEEs
  • p+n rates in 200k
• Removed the ground from the snout
  • Got really good rates originally - attachment 12
  • Noticed 1 and 5 failed ADC calibration
  • Calibrated them and there rates increased massively - attachment 13
• Notice that the pulser peak in FEE2 and 6 keeps dropping out
  • As the cables have been changed multiple times thought it could be that an adaptor board was broken
  • 4 seemed the most likely suspect with its high rates and triple peaks
  • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
  • Pulser peaks were still muissing
  • Issue could be then with adaptor board 2 possibly

Conclusions

• Rates on the FEEs are in a better place now than they were at the start of the week
  • p+n in particular have shown a large amount of improvement
• There are still however issues with the n+n FEEs
• I think there is likely an issue with one of the adaptor boards
  • Possible aida02 at the pulser connection point
  • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
  • Would recomend swapping out aida02 as a test next week
• Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
  • Nic would like to use something other than clingfilm in future
  • One suggestion would be mylar which could be held in place with strips of double sided tape
  • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
• Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
• I am at a slight loss of what else to try

• I have left the daq running at 0x64 overnight to check for alphas
• Histograms and statistics zeroed

09:59 Noticed that the right hand ribbon cable for aida07 was offset vertically by 1. i.e. Only 1 of the 2 rows of pins in. Unbiassed detectors and reconnected.
      Rebiassed detectors - leakage current on DSSD2 increased to 9.96uA - Attachment 1

      Power cycling FEEs

      Following restart the FEEs show some improvement in rates - attachment 2
      Rate spectra - attachment 3
      Still issues with aida11 in the rate spectra (Likely a misaligned ERNI connector)



      Pulser on rate spectra - attachment 4
      Pulser off rate spectra - attachment 5
      We can see that with the pulser on there are events in the channels in aida01 and aida11
      With the pulser off these channels disappear.
      ERNI connector is then likely aligned the issue is more likely the DSSD ribbon cable is misaligned.

      Taking bias off to investigate.


10:39 Could not observe anything out of alignement about either the connectors or the adaptor boards
      Note that after pushing on the connector AIDA01 had largely recovered

13:30 Top of snout taken off while mounted in S4. It was observed that one of the ribbon cables for aida11 was off by 1 row.
      This was corrected in situ and the snout was put back together and made light tight again.
      Oddly leakage current still the same - DSSD2 has come down though - attachment 6
      FEEs powercycled for new tests

      Half the channels in aida11 had appeared again
      Took apart once more and found the other header also off by one row which has been corrected

      FEEs powercycled for new test

      All of aida11 now showing - attachment 7
      Statistics at 0xa shown in attachment 8
      All p+n for DSSD1 are good (aida01, 03, 09, 10, 11 and 12)
      p+n for DSSD2 are so so (aida05, 07, 13, 14, 15 and 16)
      All n+n 2, 4, 6 and 8 are poor still


14:50 p+n waveforms - attachment 9
      n+n waveforms - attachment 10

      Current jumper configuration
      LK1 (Bias ground) on FEEs 6 and 8
      LK2&4 on all p+n
      LK3 on aida03 and aida07 (Bottom middle)

15:00 Investigated whether there was a light leak. Placed black material over snout.
      Leakage on DSSD2 dropped to 5.6 - attachment 11
      There was a light leak
      Stats have decreased slightly - attachment 12

16:00 Move raspberry pis atop 5mm alminimum plate to keep away from FEE64 PSUs
      No change

16:30 Remove pulser network from all FEEs 
      Rates get noticably worse in many FEEs... why?
      Stats - attachment 13

      Possible ideas include braid touching things it shouldn't (Adapter PCBs?) 

Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23

09:51 Taken over from Tom following the night shift
      Experiment is still running smoothly
      Compression of the files is complete up to the start of R5.
      Have started compression of files in R5 which should run up to R5_499
      Currently on R5_528

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      System wide checks:
          		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc594 : 	 87  
      aida08 fault 	 0xf1be : 	 0xf271 : 	 179  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 	Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Current merger rate is 2E6-4E6 events per second
      Current tapeserver rate is 5800 kB/s
      Free HDD space is 1.1 TB
      At current rates will last 54 hours which should see out the experiment
      Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
      
      Analysis of R5_528 - attachment 4
      Deadtime of AIDA02 currently around 15%

11:56 Statistics ok - attachment 5
      Temps ok - attachment 6
      Bias and leakage currents ok - attachment 7
      System wide checks:
      Clocks all ok 
      
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59c : 	 95  
      aida08 fault 	 0xf1be : 	 0xf27a : 	 188  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of file R5_458 - attachment 8
      Deadtime in AIDA02 only 11.5% in this file
      TapeServer rate still 5.5 MB/s

14:36 Has been no beam for the last while or so.
      With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
      Stats- attachment 9
      Temp - attachment 10
      Bias and leakage currents ok - attachment 11
      System wide checks:
      Clock ok
      	
        		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59d : 	 96  
      aida08 fault 	 0xf1be : 	 0xf27c : 	 190  
      White Rabbit error counter test result: Passed 6, Failed 2
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3c : 	 18  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

13:30 Beam taken to change an ion source

14:57 Beam back

16:21 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14

      System wide checks - ASIC clocks ok 
	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5a3 : 	 102  
      aida08 fault 	 0xf1be : 	 0xf285 : 	 199  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3e : 	 20  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Currently on file R5_592
      Analysis of R5_591 - Attachment 15
      Deadtime of AIDA02 sitting at 17%

[NH taking over for OH so he can get home]

18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
        Statitics ok - attachement 16
        Temps ok - attachement 17
        Bias & leakage ok - attachement 18

        System wide checks - 
        Clocks OK
        ADC Calibration N/A 
	WR Decoder -
        		 Base 		Current 	Difference
        aida07 fault 	 0xc53d : 	 0xc5ac : 	 111  
        aida08 fault 	 0xf1be : 	 0xf28c : 	 206  
        White Rabbit error counter test result: Passed 6, Failed 2
        FPGA -
         Base 		Current 		Difference
        aida07 fault 	 0x2a : 	 0x3f : 	 21  
        PLL OK

        Currently on file R5_621
        Analysis of R5_620 - Attachement 19
        aida02 deadtime only 5%, all others negligible 

20:20 Stats ok - attachment 20
      Temps ok - attachment 21
      Bias and leakage currents ok - attachment 22
      System wide checks:
      Clock ok
    	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ad : 	 112  
      aida08 fault 	 0xf1be : 	 0xf293 : 	 213  
      White Rabbit error counter test result: Passed 6, Failed 2

     	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3f : 	 21  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R6_637 - attachment 23

22:13 Statistics ok - attachment 24
      Temperatures ok - attachment 25
      Bias and leakage currents ok - attachment 26
      ASIC clock check ok

	
		         Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ba : 	 125  
      aida08 fault 	 0xf1be : 	 0xf2a2 : 	 228  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x40 : 	 22  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_658 - attachment 27

11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?

08:19 System wide checks all ok
      Baselines reset on WR and FPGA checks for the start of the day

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3

09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
      No corruptions have been noted since

10:10 System check, clcock and ADC ok

	 Base 		Current 	Difference
aida07 fault 	 0xfb3a : 	 0xfb3d : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA and memory check ok

      Statistics ok - attachment 4
      Temperatures ok - attachment 5
      Bias and leakage currents ok - attachment 6

10:39 Fission fragments were punching through AIDA
      This was even with the maxmimum degraders in place
      They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.

11:38 They have found the issue with the degrader and have corrected it.
      We now see next to no implants in AIDA as expect.
      Histograms reset at around 11:35
      Layout 2 shows very few implants in 5 minutes - attachment 7

12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
      This was a large amount of dose going into the detectors which we should try to avoid

13:25 system check, clock and ADC ok

	 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics ok - attachment 9
      Temperatures ok - attachment 10
      Bias and leakage currents ok - attachment 11
FPGA and memory check ok


16:10 system check, clock and ADC ok

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


      Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14
FPGA and memory check ok

07:50 Pulser settings - attachment 1

07:52 System wide checks all ok
      Statistics ok - attachment 2
      Bias ok - attachment 3

09:30 The intensity of fragments is being increased
      They are trying to do a factor of 10 increase first which should take us to around 100Hz

10:00 System wide checks all ok
      Statistics - attachment 4
      Layout 1 showing implants - attachment 5
      Fee temperatures ok - attachment 6

12.00 System wide checks all okay, no failures
      Statistics (screenshot7)
      Spectrum rate (screenshot8)
      FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal 
      (screenshot10)
      Leakage currents okay, increase from previous values written to sheets (screenshot12)
      
      Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer 
than 
      usual, another reload shows all FEE temps as normal.
      
      Merger and tape server okay

14.00 System wide checks all okay, no failures
      Statistics (screenshot13)
      Spectrum rate (screenshot14)
      FEE temps normal, no issue with reload this time (screenshot15)
      Leakage currents okay still increasing, written to sheets (screenshot16)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)

16.00 System wide checks all okay, no failures
      Statistics (screenshot17)


      Spectrum rate (screenshot18)      
      FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
      Leakage currents added to sheet (screenshot21)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

17.17 no beam, preparing for the 190Ta setting

17:44 OH Takes over
      System wide checks all ok
      Statistics - attachment 22
      Temp ok - Attachment 23
      Bias and leakage currents ok - Attatchment 24

      While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
      The Directory for the experiment was then setup S452
      The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.

      ASIC Check all ok
      

18:10 Started writing data to file
      File directory S452
      Run number R1_0
      Seems to have skipped a few files First full file R1_10
      Currently writing to /media/SecondDrive/TapeData
      Current free space 4211084180 kB
      Current write speed 46163 kB/s
      Time remaining until full 91222 seconds
      25 hours remaining space

18:34 DESPEC starting new MBS file
      AIDA Currently on file R1_41

19:00 DESPEC Closing file
      AIDA Currently on R1_75

19:05 Comparing last years statistics to this years

FEE	Last year	This year    Factor increase
1	60960	        183111       3.00
2	183494	        201256       1.09
3	91506	        196647       2.15
4	73914	        208290       2.81
5	31979	        63571        1.99
6	54084	        144638       2.67
7	35621	        137605       3.86
8	38626	        95858        2.48
9	264436	        225790       0.85
10	209709	        198896       0.94
11	88443	        133899       1.51
12	183493	        204996       1.12

Sum     1316265	        1994557      1.51

19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
      AIDA on R1_104

      Looking int to high write  rate to see if we can do anything to reduce it
      It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24

19:46 Much of the rate is coming from a small number of channels - attachment 25

19:17 WR Timestamp error.
      Mentioned to Nic and he said that sometimes small glitches like this are observed
		 Base 		Current 	Difference
aida01 fault 	 0x9e8b : 	 0x9e8d : 	 2  
aida02 fault 	 0x9a8f : 	 0x9a91 : 	 2  
aida03 fault 	 0x29e9 : 	 0x29eb : 	 2  
aida04 fault 	 0x7f66 : 	 0x7f68 : 	 2  
aida05 fault 	 0x1f4f : 	 0x1f51 : 	 2  
aida06 fault 	 0x8faa : 	 0x8fac : 	 2  
aida07 fault 	 0x53f0 : 	 0x53f2 : 	 2  
aida08 fault 	 0x7066 : 	 0x7068 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     All other checks passed

20:52 DESPEC Stopping file
      AIDA on R1_221


20:53 DESPEC Stopped file because UCESB crashed
      AIDA on file R1_237

21:20 All system wide  checks ok
      Statistics DISC info- Attachment 27
      FEE Temp ok - Attachment 28
      Bias and leakage current ok - Attachment 29
      Statistics good events - Attachment 30

20:51 Started compressing data with command nice -n 10 gzip -v R1_*
      Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
      I am not sure if MBS can keep up with this rate
      We are still writing to file though.

23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us

08:16 System wide checks all ok
      Statistics - attachment 1
      Temperatures - attachment 2
      bias - attachment 3
      Merger running 4.5e6 events per second
      Tape data writing at 42MB per second

      Currently no beam

09:45 While there is no beam we will perform a longer test of the new merger.
      Run R20 stopped and merger changed to neew version
      Started R21.

      The now usual behaviour of additional files at start of merge observed.
      N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was 
set to going
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18

     TapeData rate of 13280 kB/sec
     No errors observed in ucesb
     Correlations seen between time machine in AIDA and Ge in online
     Pulser rate in the online makes sense
     With the current data rate remaining HDD space on current drive will last around 89 hours.

09.20 analysis of file R21_27
      ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data 


10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
    (procedure "XAIDAAccessClient__Get" line 1)
    invoked from within
"XAIDAAccessClient__Get $Addr"

     aida02 restarted itself during the reset process
     Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
     System wide checks following the restart all ok.
     When restarting the MBS relay errors observed until a timestamp was observed:
     Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000

10:55 Statistics - attachment 5
      Temperature - attachment 6
      Bias - attachment 7

11:03 Time machine correlation spectra with new merger
      AIDA - FATIMA - Attachment 8
      AIDA - Ge - Attachment 9

11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the 
layout to determine which FEE/ASIC caused the events before checking ASIC control

12:12 System wide checks all ok *except adc calibration which is same as before
      Statistics - attachment 10
      Temperature - attachment 11
      Bias and leakage currents ok - attachment 12

      A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.

      After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
      They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon 
cable from the VME scalers.
      Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of 
DSSD3.
    
12:50 We have waveforms for some FEES
      Layout 7 attachment -14
      Layout 8 - attachment 15

14.00 (LS)
      Still no beam
      System wide checks okay except same as before:
      **FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module**
      
      Statistics (attachment16)
      Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
      aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
      aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
 
      FEE Temps (attachment17)
      Leakage currents written to sheets(attachment18)
      Merger ~ 45M items/s
      TapeServer~ 14MB/s
     
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file 
      R22_98, rate spectra attached (attachment19)

14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw 
      no timewarps so we are not losing anything.
      Restarted MBS relay, errors have not reappeared so far

16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE Temps (attachment22)
      Leakage currents written to sheets(attachment23)
      Merger ~ 4.3M items/s
      TapeServer~ 14MB/s  
      Current file R22_148   

16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from 
S452f115
      Around R22_150

18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
      Rates of R22_122 (attachment25):
      First DSSSD (fee0-fee3) ~66/s
      Second DSSSD (fee4-fee7) ~49/s
      Third DSSSD (fee8-fee11) ~22/s

      Rates of R22_178 (attachment26):
      First DSSSD (fee0-fee3) ~122/s
      Second DSSSD (fee4-fee7) ~94/s
      Third DSSSD (fee8-fee11) ~49/s


18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment27)
      Rate spectra (attachment28)
      FEE Temps (attachment29)
      Leakage currents written to sheets(attachment30), look to be on the way down
      Merger ~ 4.5M items/s
      TapeServer~ 15MB/s  

18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier

19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.

20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre 
reboot)
      We are borrowing a HDD from the SHIP group

20:30 System wide checks all ok
      Statistics - attachment 31
      Temperature - attachment 32
      Bias - attachment 33

22:10 MBS DAQ has crashed
      Merger terminal is now showing a large number of bad merge events

      It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.
      

22:42 System wide checks all ok
      Statistics - attachment 34
      Temperature - Attachment 35
      Bias and leakage currents ok - attachment 36

23:33 System wide checks all ok - attachment 37
      Temperatures - attachment 38
      Bias and leakage currents ok - attachment 39