15:55 Took over from OH

16:00
Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System wide checks
All OK, except ADC all fail (as before) and WR

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_691. Attach 4. Dead time <5%

16:34 MBS changes file.


17:00
Stats OK - attach 5
Temps OK - as before
System wide checks as before
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


17:55
Stats OK - attach 6
Temps OK - as before

Leakage current approaching 10 uA - attach 7

System-wide checks - as before except
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x568 : 	 104  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



18:01 MBS switches to new file



18:52
Stats OK - attach 8
Temps OK - as before

System-wide checks as before

Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9

19:36 Stats OK - attach 10
Bias - attach 11

19:48 Shift handed over to LS

16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before

18.25 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - OK

WR timestamp control - attachments 4 & 5

merger & TapeServer OK - attachments 6-8

rate spectra - attachment 9

1.8.L spectra - attachments 10 & 11
  aida03 72 ch FWHM

20:36 DSSSD biases & leakage currents OK - attachment 12
      FEE64 temps OK - attachment 13
      good events statistics OK - attachment 14

      all system wide checks ok

21.50 beam starts back up
      DAQ running, TapeServer now writing to file R7_0
      data rate ~ 25 Mb/s
      data forwarding to MBS ok
      merger ok, data item rate ~ 2.7 million items/sec
      spill time length is now 3 seconds
      1500 count rate per spill, ~ 300 Hz implant rate in AIDA

22.35 FEE64 temps OK - attachment 15
      good event statistics ok - attachment 16
      detector bias/ leakage currents ok - attachment 17

      all system wide checks ok

22.47 attachments 18 & 19 - high energy spectra for all FEE64 (1.8.H)

      attachment 20 - hit rate spectra for all FEE64

23.02 beam down
      on file R7_55

23.11 beam back online
      on file R7_61

23.15 system wide checks all ok

      detector bias/ leakage currents ok - attachment 21

      good event statistics ok - attachment 22

      FEE64 temperatures ok - attachment 23      

Attachment 1 - 1.8.W waveform spectra with pulser signal

Attachment 2 - 1.8.L pulser peak spectra

widths: aida01 - 16.06
       
        aida04 - 20.68

        aida05 - 14.03

        aida08 - 73.02

        aida09 - 669.29

        aida11 - 405.35

        aida12 - 397.6

Attachment 3 - good event statistics
               fast comparator threshold (LEC) at 0xff
               FEEs 8,9,11,12 running hot

Attachment 4 - FEE64 temperatures - all ok

System wide checks ok, except aida02 fails ADC calibration

Leak found in seal on cooling pipe, switched power off

Pictures of leaking cooling pipe

MACB switch settings

AIDA mount - needs rotated

Attachment 11 - AIDA@DESPEC MACB settings - 100MHz and WR

Post White Rabbit update

Attachment 1 - 1.8.W pulser waveform spectra for all FEE64
             - pulser signals observed in all FEE64

Attachment 2 - good event statistics - aida08 has significant noise

Attachment 3 - FEE64 temperatures - all ok, but firmware version not displayed

Attachment 4 - 1.8.L pulser peak spectra

             - widths:

             aida01 - 23.66
             aida02 - 19.76
             aida03 - 20.43
             aida04 - 18.82
             aida05 - 20.05
             aida06 - 41.04
             aida07 - 21.24
             aida08 - 205.02 (noisy FEE64)
             aida09 - 24.81
             aida10 - 22.85
             aida11 - 13.98
             aida12 - 19.90

Attachment 5 - New Merger statistics

Attachment 6 - Raw data stream

10.30 AIDA setup and DAQ start complete
      writing to file TapeData/NULL/R1

      FEE64 Temperatures and Good event statistics ok (attachments 1 and 2)

11.00 results of analyser - shows significant number of timewarps in aida07, and a few in aida01-04

11.24 compared timestamps between mbs and AIDA, agree to 8 digits (attachment 3)

12.49 DAQ and Merger stopped and restarted

      Raised slow comparator threshold from 0xa to 0x64

      Raised fast comparator threshold (LEC/MEC) from 0xf to 0xff

12.51 Good event statistics: aida08 running at a normal rate

      pulser peak (1.8.L) no longer has noise shoulder (attachment 4)

      writing to file TapeData/NULL/R2


13.58 stopped writing to file. Pulser frequency increased to 1 kHz

      writing to file TapeData/NULL/R3

15.00 file R4 - DAQ startup

17.58 trimmed and re-crimped ground wire on aida08
      lowered pulser freq back to 50Hz
      reduced slow comparator threshold to 0xa, fast comparator threshold (LEC) to 0xf

      good event stats ok, aida08 no longer running hot (attachment 6)
      1.8.L and 1.8.W spectra of aida08 (attachment 5)
      pulser peak width now at 55.71

      FEE64 temperatures ok (attachment 7)

AIDA@DESPEC MACB time switch settings before and after WR are as follows.

Attachment 1 - MACB 1-4 from left to right

MACB number:                           1        2        3        4

100MHZ clock setting (before WR):      0        2        2        2

White Rabbit setting (after WR):       3        3        3        3

09.30: AIDA still running, not writing to file
       Still running ok
       No water leaks

09.45: FEE64 temperatures ok (attachment 1)
       Good event statistics ok (attachment 2)

10.00: Results of System Wide Checks
       ADC Calibration check (attachment 3)
       Clock Status (attachment 4)
       Sync error counter (attachment 5)
       White Rabbit Decoder status (attachment 7)
       Memory Information (attachment 8) 
       
10.30: Clock ReSync performed (attachment 6)
       Start writing to file TapeData/NULL/R7_0

11.50: DAQ stopped, TapeServer stopped
       
       TapeServer restarted, DAQ restarted

       Writing to file TapeData/NULL/R9_0 with Clock ReSync at start.

       Wasn't writing to disk, restarting (DAQ stop, TapeServer stop, TapeServer start TO DISK, DAQ Start)

       Writing to file TapeData/NULL/R10_0

12.04: DAQ stopped, TapeServer stopped, merger stopped
       FEE64 power relay switched off
       Raspberry PIs switched off and disconnected
       water cooler stopped
       Mains IEC cables removed from relay

       
        

Ribbon cables fitted with kapton PCB couplers -> 3 isolated + 1 non-isolated for each cable length/DSSSD

Ribbon cable lengths:

without kapton PCB coupler (cm) - 40, 47, 51.5

with kapton PCB coupler (cm) - 50.5, 57.5, 62

gap between plastic and dsssd - 15 mm

length between top of snout and dsssd - 5cm

12.51 - detector bias and leakage currents - detector 3 has no current
                                           - other detectors ok

13.13 - tried changing from HV3 cables to HV4 - still open circuit 

      - tried different permutations of moving HV-4 braid and core cables (high voltage to ground connections)

       (dssd3)

      (all directions defined as if looking upstream)

      - core -> bottom, braid -> left     open - circuit

      - core -> top, braid -> right       trips at 8V

      - core -> bottom, braid -> right     open - circuit

      - core -> top, braid-> left          trips at 8V

      - removed jumpers - still trips

13.36 - testing continuity of bond wires on dsssd 3208-14 & 3208-15

      - observe 20 ohm resistance on p+p and p+n bond wires 

19:41 - dsssd 1 and 2 not biasing, 3 ok.

      - changed bias configuration -> HV core 1 & 2 now on top, braid on right

      - ribbon cables for DSSSD #1 & #2 p+n junction strips down were misaligned

      - all detectors biased succesfully

HV cable configuration (looking upstream): HV-1 Core -> top-inner (FEE 1)
                        
                                           HV-2 Core -> top-middle (FEE 5)

                                           HV-3 Core -> bottom-outer (FEE 11)

                                           HV-1 Braid -> right-inner (FEE 2)

                                           HV-2 Braid -> right-middle (FEE 6)

                                           HV-3 Braid -> left-outer (FEE 12)

Attachment 3 - rough sketch of HV cable configuration, looking upstream.
       

10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
      Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%

10.15 DSSSD stack as follows

      DSSSD  serial  thickness  depletion 
      #              (um)       voltage (V)
      1      3208-7  1019       125              upstream
      2      3208-13 1020       120
      3      3208-14 1020       130              downstream

11.30 AIDA power relays sequenced on
      
      DAQ start

      system wide checks ok

11.33 attachment 2 - fee temperatures - ok

      attachment 3 - good event statistics with slow comparator threshold = 100

      attachment 4 - good event statistics with slow comparator threshold = 50

      attachment 5 - good event statistics with slow comparator threshold = 20

      attachment 6 - good event statistics with slow comparator threshold = 10
     
13.51 attachment 7 - Rates
                   - aida09 1 ASIC not working (no data)

      attachment 8 - good event statistics

      attachments 9/10 - 1.8.L spectra
                       - no peak observed in aida09

      attachments 11/12 - 1.8.W spectra
                        - waveform only observed in naida 2,4,7,8,12

      pulser peak widths -   FEE         width(ch)
                              1             71
                              2             96
                              3             66
                              4             79
                              5             74
                              6             96
                              7             89
                              8             83
                              9           no peak
                              10            82
                              11            153
                              12            120

17.26 FATIMA moved upstream around AIDA

      attachment 13 - good event statistics

      ASIC check-load performed

      attachment 14 - good event statistics following checkload

11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding

3x MSL type BB18(DS)-1000 installed 

detector bias -160V, leakage current c. 1uA @ +21 deg C for all 3x DSSSDs

all FEE64 good event rates (slow comparator 0xa, LEC fast comparator 0xff) c. 120k, or less, typically 30-50k, overall merge rate c. 1.6M data item/s

See https://elog.ph.ed.ac.uk/DESPEC/70



Outstanding issues

- occasional loss of bits 48-63 of WR timestamp for aida09 - can be recovered by power cycle - replace HDMI cable?

- aida09 asic 1 not producing ADC or disc data - replace aida09? 
  ASIC Check works OK
  aida09 asic temp c. 30deg C < other asic temps

- evidence of c. 1MHz extrinsic noise for most FEE64s

- acquisition of waveform data not robust - most ASIC channels do not produce data or quickly stop producing data  - PJCS to review firmware rev for Jan/Feb 2020

- require spare DSSSDs, FEE64s, HDMI cables etc

>
> 
> - evidence of c. 1MHz extrinsic noise for most FEE64s


Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom

ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:09 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

08:14 FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias/leakage current ok - attachment 3

08:22 Merger 4.3M items/s

      Tapeserver 14 MB/s

08:24 DAQ ok, writing to file R43_45

08:39 rate spectra - attachment 4

      aida09 spectrum not showing, however continues to collect statistics ok

10:03 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

      FEE64 temperatures ok - attachment 5

      good event statistics ok - attachment 6

      detector bias/leakage current ok - attachment 7

10:10 Merger 4.5M items/s

      Tapeserver 14 MB/s

      data forwarding to MBS ok

10:11 writing to file R43_91

12.00 (LS)
      System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      		         Base 		Current 	Difference
      aida07 fault 	 0x829e : 	 0x829f : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1
      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      no FGPA timestamp errors all passed

      Statistics (attachment8)
      Spectra rate (attachment9), aida09 still does not show histogram, but on run control are enabled for all fees. still 
      showing good events and other information (temperatures etc)
      FEE temps (attachment10)
      Leakage currents written to sheets (attachment11)
      Merger~4.6M items/s
      Tapeserver~14MB/s

12.36 bunch of bad timestamp errors in the new merger terminal (attachment 12) around file R43_152
      this error is still occurring at 12.45 will continue to monitor, all other system checks are reporting as normal


13.50 beam down file R43_183
      beam back a couple of minutes later

14.00 System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      no white rabbit or FPGA errors all passed

      Statistics (attachment13)
      Spectra rate (attachment14)
      FEE temps (attachment15)
      Leakage currents written to sheets (attachment16)
      Merger~4.6M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay, still seeing bad timestamp errors in the new merger terminal continuing to monitor and update 
      statistics tab every 20 to 30 minutes 

14.12 Analysed files R43_185,186,187 from timestamp error in ucesb, but see no timewarps  

14.30 have not seen any bad timestamp errors in new merger terminal for a while, unsure if something has been changed which 
      has stopped them

15.25 beam down
15.53 beam back

16.00 moved back to writing to /media/SecondDrive which should last until the end of the experiment
      now writing R46

16.10 System wide checks okay except:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment17)
      Spectra rate (attachment18)
      FEE temps (attachment19)
      Leakage currents written to sheets (attachment20)
      Merger~4.7M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay

11.12 System wide checks ok (attachment 1)
      FEE temperatures ok (attachment 2)
      bias/leakage currents ok (attachment 3)

11.13 1.8.W spectra for naida 1,9,10,11,12 (attachment 4)
      1.8.W spectra expanded in Y direction (attachment 5)
      pulser peaks (attachment 6)
      
11.20 pulser peak widths:
      
      Aida10 -> 98.82

      Aida09 -> 109.17

      Aida11 -> 71.72

      Aida12 -> 92.82

      Aida01 -> 24.05

11.23 hit rates (attachment 7)

11.41 pictures of setup with scintillator mounted (attachment 8-14)

09.28 System wide checks - aida06 global clock failure (check clock status)

                         - aida02 , aida04, aida06 fail calibration (ADC calibration)

                         - SYNC error counter ok

                         - SYNC pulse check ok

                         - 'collect memory information from FEE64 Linux' still returns scripting error

                         - WR decoder test passes

09.33 good event statistics ok, except aida09, aida10 running hot. (attachment 1)

      FEE temperatures ok (attachment 2)

      detector bias/leakage current ok (attachment 3)

09.38 screenshots of NewMerger tab, Merger/Tape HTTPd and NewMerger terminal (attachments 4, 5, 6)

10.05 1.8.L pulser peak spectra (attachment 7)

10.13 1.8.W waveform spectra (attachment 8)

21.13 Beam on

      AIDA DAQ running - writing to file R9 in directory 290319

      System wide checks - same as above

      good event statistics ok - although rates on 9,10,11,12 have dropped (attachment 10)

      ASIC check load performed

      FEE temperatures ok (attachment 9)

      detector bias leakage currents ok (attachment 11)

      Scintillator 41 -> approx 1000 ions/s, 11 second super cycle (beam on 10s, beam off 1s)
      c. 130-150MeV/u 107Ag ions incident on plastic scintillator + AIDA DSSSD

21.36 High energy spectra for aida09 - (attachment 12)

      High energy spectra for aida11 - (attachment 13)

      Low energy spectra - (attachment 14)

      1D hit pattern - (attachment 15)

21.53 ASIC control settings - see for slow comparator threshold, fast comparator threshold (HEC) and fast
      comparator threshold (LEC) (attachment 16)

      NewMerger screenshot - working (attachment 17)

22.00 ASIC options - including directory path to saved ASIC settings (attachment 18)

22.09 beam stop
      TapeServer stopped, restarted not writing to storage
      one complete file produced R9_0 (2.0 GB)
      plus R9_1 (154 MB)

*NOTE FROM 02/04/19 - DEGRADER in front of detector, silver beam not observed, just unidentified light ions.

On Save/Restore Module Settings Tab - "inquire save state" returns an error. See attachment 1.