> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom

Status before grounding checks

Test with shielded HV cables for both detectors.

The FWHM for ch1 is now 184 ch.

Yesterday cables for cards 5, 6, 10, 11, 12, 14, 16 were partially connected (only 1 out of 2 flat cables was in), now both are connected.

1) Click on item "Merger MK2" in top panel
   This kills any existing Merger existence and loads a new copy.

2) Start (if needed) browser conection to localhost:8115.

3) Click on "NewMerger Control"

4) In NewMerger Control, click "Reload"

5) If MergerState = IDLE, click "SETUP".

6) Wait until state = STOPed

7) If links shown are correct, click "GO"

8) Wait until popup about "Resume Merger" appears and click OK

9) when writing to disk, click "Forward Output to Data"

Some screenshot of spectra attached

The AIDA snout (with single DSSD and plastic) has been mounted in S4 cave again. DSSD was connected to the outer 4 FEEs again.

-160 V Bias applied and DSSD holds voltage with leakage current of around 1.3 uA (was dropping as only biased briefly)

Bias off and FEEs offline again now as day nearly done. MBS testing will be performed shortly, as well as resolution checks.

On the 18th of September we installed DSSD 3208-14

Was found to have problems including exponential leakage behavior

  V      Ic
 -5   15.4uA
 -10  53uA
 -15  Tripped at over 100uA

Cables were double checked, bond wires were checked and no problems could be observed.

DSSD replaced with 3208-15

Bias at -100V Ic 10.2uA GP/grnd Jumpers removed,
                         rising upwards, possible short circuit as well.

Jumpers replaced

Bias -100V Ic 1.6uA
Bias -160V Ic 1.62uA

Possibility that mechanical sample 3208-17 and 3208-14 have been swapped. (To check at later date)

FEEs powered up and Adapter boards and all grounding installed with DSSD going to 9,10,11 and 12

Rates shown - attachment 1
Waveforms - attachment 2-4
1.8.l Spectra - attachment 5
FEEs 10 and 11 around 60FWHM and 9 and 12 around 80-90FWHM
Stats page - attachment 6

Aluminium foil used to make sure all ports are light tight shows a stats drop of around half - attachment 8

New peak widths

aida09 - 79.65
aida10 - 67.37
aida11 - 59.99
aida12 - 97.83

Going to begin a systematic study of shaping time

                              FWHM
Shaping Time   aida09   aida10   aida11   aida12
   
     8          79.65   67.37    59.99     97.83
     7          97.55   70.84    62.67     115.34
     6         100.72   70.04    62.77     120.05
     4         147.47   80.64    71.34     167.23
     2         217.62   114.98   133.35    232.47

Shaping time 8us 1.8.L Shots 
Shaping time 4us Stats appear obviously worse - attachment 7
                 1.8.L Widths - attachment 8
Shaping time 2us Stats appear worse again - attachment 9
                 1.8.L Widths - attachment 10
Shaping time 6us Stats are better than 2 and 4 - attachment 11
                 1.8.L widths - attachment 12
Shaping time 7us Stats are better than 6 - attachment 13
                 1.8.L widths - attachment 14


8us is the optimum shaping time.
Taken back to 8us and similar shaping times are observed.

Connected the ground of all 4 adapter boards with direct cables. Observed improvements in both FEEs 9 and 12
however the noise in 11 went
through the roof >500kHz - attachment 15

Tested just having 9,10 and 12 connected the rates are comparable to when disconnected - attachment 16

All 4 grounded together with a ground running to the snout as well stats No difference to all 4 together -
attachment 17
              1.8.L Widths - attachment 17
aida09 104.76
aida10 84.58
aida11 281.77
aida12 82.58

Tested just having 10 and 12 directly connected. Same side of the DSSD: Noise is the same as when disconnected.
Stats - attachment 19
1.8.L spectra - attachment 20

aida09 107.13
aida10 67.02
aida11 66.78
aida12 131.51

Tested having 11 and 12 connected directly. This is the two adapter boards responsible for the bias with 11
carrying the core and 12 the braid.
Massively reduced noise in 9 and 12, comparable noise in 10 but much more noise in 11. Stats - attachment 21
aida09 63.96
aida10 76.64
aida11 125.96
aida12 68.05
1.8.L spectra - attachment 23



18:39 Alpha run started in /TapeData/Sep18/R2
      Writing at 96kb/s
      11 Was fast at start bus asic control brought it down




                 

Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 3)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- IGNORE FOR TIME BEING Options database monitor (Top middle terminal screen 2, workspace 3)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 5)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 3)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 3)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 3)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 3)

- Perform the five system wide checks (Web browser tab screen 1, workspace 3)
     - Ignore the system wide checks for ADC calibration (ADCs are turned off so they will all fail)
     - Check memory information also fails.
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 5)

Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 2)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- Options database monitor (Top middle terminal screen 2, workspace 2)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 4)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 2)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 2)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 2)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 2)

- Perform the five system wide checks (Web browser tab screen 1, workspace 2)
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 4)

4:13

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Screenshot 1: statistics

Screenshot 2: temperatures

Screenshot 3: scalers

Screenshot 4: Bias and leakage current

6:36

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Screenshot 5: statistics

Screenshot 6: temperatures

Screenshot 7: scalers

Screenshot 8: Bias and leakage current

Attachment1: AIDA05.1.8L when AIDA05 connected, biased at 100 V with not-shielded cables, CAEN HT,  after the base of the snout (screws) has been connected to the ground via cupper sheets

FWHM: 76 channels (to be compared with the 2 previous elogs 73 channels and 75 channels)

Conclusion: no differences can be observed.

- Initialised the Merger and Tape server

- loading of settings: EXPERIMENTS/AIDA/2025Oct29-16.11.06

- aida16 very noisy (attachments 3,4), slow comparator thresholds was risen to 500 keV, until reducing its deadtime to 0

- y FEEs disabled in the discriminator (example in attachment 5)

- Starting data taking with following details:

MIDAS data in: Bi207Centre

MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq')

Data taking and DAQ stopped now (17.40)

11.12 System wide checks ok (attachment 1)
      FEE temperatures ok (attachment 2)
      bias/leakage currents ok (attachment 3)

11.13 1.8.W spectra for naida 1,9,10,11,12 (attachment 4)
      1.8.W spectra expanded in Y direction (attachment 5)
      pulser peaks (attachment 6)
      
11.20 pulser peak widths:
      
      Aida10 -> 98.82

      Aida09 -> 109.17

      Aida11 -> 71.72

      Aida12 -> 92.82

      Aida01 -> 24.05

11.23 hit rates (attachment 7)

11.41 pictures of setup with scintillator mounted (attachment 8-14)

24:00 Joined in with TD to debug the dropout of AIDA05 - See https://elog.ph.ed.ac.uk/DESPEC/193
      After both powercycles and telnet reboots we were still seeing 0 in the statistics but did note that the counter was > 0
      The last check we did was a restart of the merger server. This solved the issue.
      It is likely that the link to the FEE was dropped but not re-established upon the resets.
      There was no error message that this was seen. From now on I recommend refreshing the statistics ever 30 minutes.

01:29 System wide check
N.B. May not have reset baselines after reset
WR fault
		 Base 		Current 	Difference
aida01 fault 	 0x1ad : 	 0x1af : 	 2  
aida02 fault 	 0x4434 : 	 0x4436 : 	 2  
aida03 fault 	 0xcf95 : 	 0xcf99 : 	 4  
aida04 fault 	 0x25aa : 	 0x25ae : 	 4  
aida05 fault 	 0x4b4 : 	 0x4b7 : 	 3  
aida06 fault 	 0x834 : 	 0x837 : 	 3  
aida07 fault 	 0x146d : 	 0x1472 : 	 5  
aida08 fault 	 0xf4a8 : 	 0xf4ab : 	 3  
aida09 fault 	 0x4f6f : 	 0x4f73 : 	 4  
aida10 fault 	 0x7504 : 	 0x7506 : 	 2  
aida11 fault 	 0x26fa : 	 0x26fc : 	 2  
aida12 fault 	 0x2858 : 	 0x285b : 	 3  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Check
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

01:42 Statistics - attachment 1
      Temperature - attachment 2
      Bias and leakage current - attachment 3

03:57 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1473 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x4 : 	 0xc : 	 8  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics - attachment 4
      Temp - attachment 5
      Bias - attachment 6

05:27 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1474 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x4 : 	 0x16 : 	 18  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


     Statistics - attachment 7
     Temp - attachment 8
     Bias - attachment 9

05:51 Looking into the bad timestamp messages in the merger e.g.
MERGE Data Link (28348): bad timestamp  5 3 0xc14f8415 0x00eea280 0x0000cd2e70eea280 0x166bcd2e70eea280 0x166bcd2e716c1f80

Looking in the merger source for the bad timestamp message:

            if (op->Time < LastTimeStamp) {

//     invalid time stamp 

               (*StatsMem[TSSEQERR])++;

               (*StatsMem[TSSEQERR+((LinkNum+1)*MAXCOUNTERS)])++;

               sprintf(message_buffer, "bad timestamp  %d %d 0x%08lx 0x%08lx 0x%016llx 0x%016llx 0x%016llx",LinkNum, link_table[LinkNum]->link_state, op->Data, op->Timestamp, INFO4, op->Time, LastTimeStamp);
      
                report_message(MSG_WARNING);   /***************************/

//              LastTimeStamp = op->Time;


So the message is generated when the merger detects a timewarp.

Took the first warning a data block of errors (The first instance of that particular `LastTimeStamp` and c alculated the time difference between the new timestamp and the last timestamp
The time and LastTimestamp were 0x166bccef3f424a50 0x166bccef4f4232e0 respectively
The time difference between them is 268429456 which is 268ms which seems quite a large difference


06:39 AIDA Crashed
      This time all FEEs were responsive but not showing any stats
      When stopping the DAQ all FEEs except aida06 stopped - attachment 10
      Did a reset of the DAQ and all recovered but no stats on aida06 - attachment 11
      Regained DAQ with a powercycle and a complete restart of the AIDA:8115 Merger and TaperServer

      It is worth noting that aida06 is connected to link 5 the data link which had been producing the bad merge messages overnight.
      We have now had it in aida05, aida6 and aida07. Could it be to do with the correlation scaler rate going into these FEES?

      Going through the var/log/messages on aida-3 aida06 rebooted itself at 06:37
      Mar 13 06:37:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Mar 13 06:37:18 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Mar 13 06:37:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)

Looking in /var/log/messages on aida06 no evidence of a reason why:
Mar 12 23:30:56 aida06 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: driver assigned major number 253
Mar 12 23:31:13 aida06 kernel: xaida: open:
Mar 12 23:31:14 aida06 kernel: AIDAMEM: aidamem_open:
Mar 12 23:33:06 aida06 kernel: xaida: open:
Mar 13 05:37:30 aida06 syslogd 1.4.2: restart.
Mar 13 05:37:30 aida06 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Mar 13 05:37:30 aida06 kernel: Using Xilinx Virtex440 machine description
Mar 13 05:37:30 aida06 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Mar 13 05:37:30 aida06 kernel: Zone PFN ranges:

09.54 FEE64 temperatures OK - attachment 1
      ADC data item stats OK - attachment 2
       p+n FEE64s 11x <100k, 1x <200k
      Rate spectra - attachments 3 & 4
       before and after ASIC check load
      1.8.L spectra - attachments 5 & 6
      1.8.W spectra 20us FSR - attachments 7 & 8
      Disc, WR-28-47, pause, resume info data items - attachments 9-12
       note large number pause/resume for aida13 - legacy numbers since last restart/zero?
       attachment 16 for pause stats since stats zero at 10.02 - looks OK
      Grafana DSSSD bias & leakage currents for most recent 48h OK - atatchment 13
      DSSSD bias & leakage currents - attachment 14
      Merger stats/merger/tape server messages - attachment 15

10.02 All histograms and stats zero'd
      
10.06 All system wide checks OK *except*

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 06 15:29:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.25 Pulser peak width aida01 97 ch FWHM, aida02 350 ch FWHM
       peak widths are increasing cf. earlier this week - see https://elog.ph.ed.ac.uk/DESPEC/291 , https://elog.ph.ed.ac.uk/DESPEC/298

14.09 aida02 WR stall
      DAQ STOP
      restart Merger, Mrger SETUP, Merger GO
      DAQ GO

      all histograms & stats zero'd

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 4

      WR timestamp before/after Merger restart - attachments 3 & 5

      all system wide checks OK *except* attachments 6-7

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 8-10
       ADC data items rates high - source installed?

      per FEE64 Rate spectra - attachments 11-13

      per FEE64 1.8.W spectra - 20us FSR - attachments 14-15

      per p+n FEE64 1.8.L spectra - attachment 16
       aida09 pulser peak width 60 ch FWHM

      Merger, TapeServer etc - attachments 17-19
       data file S181/R2_249 switch TapeServer to No Storage mode

      per p+n FEE64 1.8.W spectra - 20us FSR - attachment 20

20.33 DSSSD bias & leakage current OK - attachments 21

      FEE64 temperatures OK - attachment 22

      ADC data item stats - attachment 23

      per FEE64 Rate spectra - attachment 24

Attachments 1 & 2 per FEE64 1.8.L spectra 
 pulser peak widths aida01 121 ch FWHM, aida02 517 ch FWHM

Attachments 3 & 4 per FEE64 1.8.W spectra 20us FSR

Attachment 5 per FEE64 rate spectra

Attachment 6 - grafana DSSSD bias/leakage current for previous 7 days showing effect of day/night temp variations

Attachments 7 & 8 - merger & tape server c. 14Mb/s to disk - no errors reported by servers

Attachments 9-12 - system wide checks - all OK

Attachment 13 ADC data item stats

Attachment 14 FEE64 temps OK

Attachment 15 DSSSD bias & leakage currents OK

16.15 aida05 zero ADC data items stats
      merger restart, DAQ STOP/Go cycle to recover


      activity is S4 area near AIDA DSSSDs as install of DEGAS detectors commences

19.50 DEGAS install continues

      DSSSD bias and FEE64 power OFF
 

07:50 Pulser settings - attachment 1

07:52 System wide checks all ok
      Statistics ok - attachment 2
      Bias ok - attachment 3

09:30 The intensity of fragments is being increased
      They are trying to do a factor of 10 increase first which should take us to around 100Hz

10:00 System wide checks all ok
      Statistics - attachment 4
      Layout 1 showing implants - attachment 5
      Fee temperatures ok - attachment 6

12.00 System wide checks all okay, no failures
      Statistics (screenshot7)
      Spectrum rate (screenshot8)
      FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal 
      (screenshot10)
      Leakage currents okay, increase from previous values written to sheets (screenshot12)
      
      Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer 
than 
      usual, another reload shows all FEE temps as normal.
      
      Merger and tape server okay

14.00 System wide checks all okay, no failures
      Statistics (screenshot13)
      Spectrum rate (screenshot14)
      FEE temps normal, no issue with reload this time (screenshot15)
      Leakage currents okay still increasing, written to sheets (screenshot16)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)

16.00 System wide checks all okay, no failures
      Statistics (screenshot17)


      Spectrum rate (screenshot18)      
      FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
      Leakage currents added to sheet (screenshot21)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

17.17 no beam, preparing for the 190Ta setting

17:44 OH Takes over
      System wide checks all ok
      Statistics - attachment 22
      Temp ok - Attachment 23
      Bias and leakage currents ok - Attatchment 24

      While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
      The Directory for the experiment was then setup S452
      The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.

      ASIC Check all ok
      

18:10 Started writing data to file
      File directory S452
      Run number R1_0
      Seems to have skipped a few files First full file R1_10
      Currently writing to /media/SecondDrive/TapeData
      Current free space 4211084180 kB
      Current write speed 46163 kB/s
      Time remaining until full 91222 seconds
      25 hours remaining space

18:34 DESPEC starting new MBS file
      AIDA Currently on file R1_41

19:00 DESPEC Closing file
      AIDA Currently on R1_75

19:05 Comparing last years statistics to this years

FEE	Last year	This year    Factor increase
1	60960	        183111       3.00
2	183494	        201256       1.09
3	91506	        196647       2.15
4	73914	        208290       2.81
5	31979	        63571        1.99
6	54084	        144638       2.67
7	35621	        137605       3.86
8	38626	        95858        2.48
9	264436	        225790       0.85
10	209709	        198896       0.94
11	88443	        133899       1.51
12	183493	        204996       1.12

Sum     1316265	        1994557      1.51

19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
      AIDA on R1_104

      Looking int to high write  rate to see if we can do anything to reduce it
      It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24

19:46 Much of the rate is coming from a small number of channels - attachment 25

19:17 WR Timestamp error.
      Mentioned to Nic and he said that sometimes small glitches like this are observed
		 Base 		Current 	Difference
aida01 fault 	 0x9e8b : 	 0x9e8d : 	 2  
aida02 fault 	 0x9a8f : 	 0x9a91 : 	 2  
aida03 fault 	 0x29e9 : 	 0x29eb : 	 2  
aida04 fault 	 0x7f66 : 	 0x7f68 : 	 2  
aida05 fault 	 0x1f4f : 	 0x1f51 : 	 2  
aida06 fault 	 0x8faa : 	 0x8fac : 	 2  
aida07 fault 	 0x53f0 : 	 0x53f2 : 	 2  
aida08 fault 	 0x7066 : 	 0x7068 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     All other checks passed

20:52 DESPEC Stopping file
      AIDA on R1_221


20:53 DESPEC Stopped file because UCESB crashed
      AIDA on file R1_237

21:20 All system wide  checks ok
      Statistics DISC info- Attachment 27
      FEE Temp ok - Attachment 28
      Bias and leakage current ok - Attachment 29
      Statistics good events - Attachment 30

20:51 Started compressing data with command nice -n 10 gzip -v R1_*
      Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
      I am not sure if MBS can keep up with this rate
      We are still writing to file though.

23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us