00:23 206Hg setting - No beam in AIDA yet

      AIDA not writing to file

      Data being forwarded to MBS

      Fragments expected tonight (?)

00:45 GSI DAQ restart

      FEE64 Temps ok - attachment 1

      All system wide checks ok

01:00 Good event statistics - attachment 2

01:48 detector biases/ leakage currents - ok - attachment 3

02:10 beam on, plug removed

      beam delivered to AIDA

      rates spectra - attachment 4

      peaks at ~ 10 counts/channel in DSSD1 c.2000Hz

02:20 beam stop

02:26 beam restart - rate spectra - attachment 5

02:43 high energy channel spectra - attachment 6

      saved the high energy spectra to layout 2 in spectrum browser

03:00 beam stop

03:08 temperatures ok - attachment 7

      statistics ok - attachment 8

      bias/leakage currents ok - attachment 9

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

      all system wide checks ok

04:24 beam is back

      rate spectra - attachment 10

04:51 206Hg implanting in DSSD2

      Rate spectra - attachment 11

      HEC spectra - attachment 12

05:08 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias/leakage currents ok - attachment 15

      All system wide checks ok

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok 

05:45 beam stopped

05:50 beam back

06:45 beam running, but no implants in AIDA

      possible issues with beam delivery to S4

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

07:10 temperatures ok - attachment 16

      statistics ok - attachment 17

      detector bias/leakage currents ok - attachment 18

      all system wide checks OK














      

      

      

16:30 good event statistics ok, with 9,10,11,12 running faster (detector FEE64s) (attachment 1)

      FEE temperatures ok (attachment 2)

      detector bias/leakage currents ok (attachment 3)

      system wide checks ok *except* FEE64 Linux memory information check causes script error,
                                     WR decoder status returns errors

      DAQ & Merger running ok

      slow comparator threshold raised to 0x64

   

To start the MBS Data Relay, click on the top rightmost of the MIDAS icons (MBS relay, Data Relay => MBS) in
Desktop 3. (attachment 1)

This will bring up the Data Relay MBS terminal.

Ensure the path defined in the datarelaymbs is set to the correct directory (shown in attachment 2) 

10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   

00:05 DESPEC tuning beam, changing spill length

00:21 DAQ continues ok
      Merger/TapeServer ok

00:25 still no beam in AIDA

00:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 1

00:52 beam in AIDA - rates spectra - attachment 2

01:05 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 3

      beam is off again

01:30 beam back

01:32 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 4

01:37 beam off - FRS increasing spill length to 5s

01:48 beam back with new spill setting - 5s on spill, 2s off spill

02:07 system wide checks ok except;

WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb42 : 	 8  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check :

			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:10 FEE64 Temps ok - attachment 5

02:20 good event statistics ok - attachment 6

02:27 detector bias / leakage currents ok - attachment 7

02:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok

      beam off

03:00 stats/ucesb/database checks ok

03:30 stats/ucesb/database checks ok

      still no beam in AIDA

05:11 system is checked, beam is back with low intensity 

05:13 FEE64 Temps ok - attachment 8

05:13 good event statistics ok - attachment 9

05:13 detector bias / leakage currents ok - attachment 10

05:33 stats/ucesb/database checks ok

06:00 AIDA crashes, lose connection to aida09

06:20 aida08 does not respond on powercycle and reset

06:30 Called TD 

06:50 successfully reset AIDA. Writing to file R30

      synchronise ASIC clocks:
	
FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0

Database options file size check:


If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

System wide checks ok *except*

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

re-calibration of these FEE modules unsuccessful in FADC align and control.

07:05 FEE64 Temps ok - attachment 12

      good event statistics ok - attachment 13

      detector bias / leakage currents ok - attachment 14

07:10 stats/ucesb/database checks ok

07:30 stats/ucesb/database checks ok


      
      

17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1

17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2

17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3

17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

07:46 all system wide checks ok

      good event statistics ok - attachment 1
  
      FEE64 temperatures ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:48 merger ok ~ 3 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

08:50 AIDA writing to file SecondDrive/TapeData/S480/R12_1524

09:56 all system wide checks ok

      good event statistics ok - attachment 4

      detector bias/leakage currents ok - attachment 5
  
      FEE64 temperatures ok - attachment 6

12:09 all system wide checks ok


12:13 all system wide checks ok

      good event statistics ok - attachment 7
  
      FEE64 temperatures ok - attachment 8

      detector bias/leakage currents ok, but ramping up - attachment 9


13:32 merger ok ~ 3 million data items/s

      TapeServer ok ~ 23 MB/s

      data forwarding to MBS ok

      Beam seems to be off

13:33 AIDA writing to file R12_1743

*update* checked DESPEC S480 Elog, issues with UNILAC, beam lost at 13:29

13:55 beam back

13:57 AIDA writing to file R12_1759

14:35 all system wide checks ok

      FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11
  
      detector bias/leakage currents ok, but ramping up - attachment 12

15:18 merger ok ~ 2.7 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

      AIDA writing to file R12_1283

16:00 1.8.H high energy spectra - attachments 13 & 14

      hit-rate spectra - attachment 15
 
16.22 all system wide checks okay 
      FEE temperatures okay (attachment 16)
      leakage currents okay and recorded to spreadsheet (attachment 17)
      good event stats okay (attachment 18)
      merger running at 3M items/sec
      tape service running at 26MB/sec

18.00 began compressing R7 files with command: nice -n 10 gzip -v R7_* &
      
      all system wide checks okay 
      FEE temperatures okay (attachment 19)
      leakage currents okay and recorded to spreadsheet (attachment 20)
      good event stats okay (attachment 21)
      merger running at 3M items/sec
      tape service running at 27MB/sec    

20.10 all system wide checks okay 
      FEE temperatures okay (attachment 22)
      leakage currents okay and recorded to spreadsheet, detectors 1 and 2 still rising (attachment 23)
      good event stats okay (attachment 24)
      merger running at 3M items/sec
      tape service running at 25MB/sec

      rates histogram (attachment 25)
      low energy histograms (attachment 26 and 27)
      high energy histograms (attachments 28 and 29)

      
22.07 all system wide checks okay 
      FEE temperatures okay (attachment 30)
      leakage currents okay and recorded to spreadsheet (attachment 31)
      good event stats okay (attachment 32)
      merger running at 3M items/sec
      tape service running at 25MB/sec
      currently on R12_2097

23.56 all system wide checks okay 
      FEE temperatures okay (attachment 33)
      leakage currents okay and recorded to spreadsheet (attachment 34)
      good event stats okay (attachment 35)
      merger running at 3M items/sec
      tape service running at 26MB/sec
      
finishing up now. night shift group have been informed how to check up on AIDA, there is a sheet on the table with relevant information
      

08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:30 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

      DAQ continues to run - R1_1148

      Merger ok - 4.6M data items/s

      TapeServer ok - 43MB/s

08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots

09:19 zeroed all histograms

      writing to file R1_1204

      FRS on file 72

      DESPEC to add AIDA XY hit patterns to online analysis histograms

09:35 beam stop

09:56 beam back 

      writing to file R1_1249

      rate spectra - attachment 5

09:30 AIDA DAQ crashed - lost connection to aida05

      power cycle and reset

09:40 AIDA ok now, writing to file R2

09:50 LS takes over

11.10 (German time)
      System wide checks produces some fails: 
      FEE64 module aida06 global clocks failed, 6 (screenshot6)
      This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there

      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2 (screenshot7)
      Tried a recalibration for the two FEEs in FADC align & control tab but error still there

      Rest of the checks pass
      Statistics (screenshot8)
      FEE temps okay (screenshot9)
      Leakage currents written to sheets (screenshot10)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

13.00 System wide checks produces same fails as last time: 
      FEE64 module aida06 global clocks failed, 6
      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2
      Rest of the checks are okay

      Statistics (screenshot11)
      Spectra rate (screenshot12)
      FEE temps okay (screenshot13)
      Leakage currents written to sheets (screenshot14)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s
      There is 1.5TB left on current drive

14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold
      Slow comparators in FEE9-12 raised to 0xd
      

      

      
      
      



        

08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude 1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

08:20 Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

      Writing to file R6_1148

09:00 Beam off for optimisation - writing to file R6_1212

      Expected back at 09:30

09:30 Beam will be off for a bit longer - an hour or two

09:34 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
      Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds 
to finish
      Something confusing, will look at data and report to VP
      Reverted to original merger and back... 45 MB/s
      R9

12.00 (LS)
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment7)
      FEE temps (attatchment8)
      Rate spectra (attachment9), beam still down
      Leakage Currents written to sheets (attachment10)
      Merger ~45M items/s
      Tapeserver ~43MB/s

14.00 Still no beam but is expected to return soon
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment11)
      FEE temps (attatchment12)
      Leakage Currents written to sheets (attachment13)
      Merger ~45M items/s
      Tapeserver ~44MB/s

14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)

15.00 Beam is back (attachment15)
      Still writing to R9 (roughly around R9_386)

15.28 No beam (attachment16)

15.41 Beam back (attachment17)
      Roughly R9 (roughly around R9_439)

16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
 
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment19)
      Spectra Rate (attachment20)
      FEE temps (attatchment21)
      Leakage Currents written to sheets (attachment22), 
      still rising but looking on Grafana 7 day period should start to lower around 17.00
      Merger ~45M items/s
      Tapeserver ~43MB/s

16.55 Beam down (attachment23), back a couple minutes later

15.51 beam down (attachment24)
      

08:00 DAQ continues ok - writing to file R13_120

      Merger ok - 4.6M data items/s

      TapeServer ok - 43 MB/s

08:03 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

08:08 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

08:32 CA briefly lost network connection - back now

08:42 DESPEC on file 102

09:24 Rate spectra - attachment 4

09:32 HEC spectra - attachment 5/6

      beam off - writing to file R13_243

09:42 merger bad timestamp errors - attachment 7

      all system wide checks ok

10:30 NH - Beam was down, I try the new merger from Vic again (R14)
           I made a mistake and had to powercycle fees as they got stuck! aida10 seemed to reboot itslef
           After fixing my mistakes, all running with new merger and rate 15 MB/s (R14)
           It all looked OK even going to MBS/Go4
           I reverted back to original hot merger, R15 and look at the data.
           Beam is now back and I return AIDA to shifters :)

11.30  System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Reran calibration for both FEEs  but still fail
       Statistics (attachment8)
       Rate spectra (attachment9)
       FEE temps (attachment10)
       Leakage currents written to sheet (attachment11)
       Tapeserver ~33MB/s
       Merger ~ 3.4M items/s
       AIDA file R15_92

11.55 Beam off for a couple minutes but back now

12.00 New Merge terminal disappeared so stopped daq and restarted, and also restarted MBS back working
      Now writing to R16
      Merger ~ 4.5M items/s
      Tapeserver ~44MB/s

12.41 no beam

13.00 DAQ crashed (aida03 no responding), AIDA recovered from power cycle while beam is down
      Beam is back now, forwarding to MBS
      Merger ~ 45M items/s
      Tapeserver ~44MB/s
      Writing to R17

13.20 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment12)
       Rate spectra (attachment13)
       FEE temps (attachment14)
       Leakage currents written to sheet (attachment15)
       Tapeserver ~45MB/s
       Merger ~ 4.4M items/s

14.27 beam off
      file R17_115
      back a couple minutes later

14.40 no beam again back at 14.46

15.40 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment16)
       Rate spectra (attachment17)
       FEE temps (attachment18)
       Leakage currents written to sheet (attachment19)
       Tapeserver ~44MB/s
       Merger ~ 4.4M items/s
       
15.52 beam down R17_221, back at 15.58 R17_229

16.07 Beam down
       
16:20 Removed S480 files from /media/1e... and /media/SecondDrive
      Set merger to no output and stopped tapedata.
      Changed symbolic link of /TapeData to /media/1e.....
      Restarted tapedata and the merger with outputting to file R20
      Local DESPEC was made aware of this before carrying out and stopped the run.
      Beam back

17.12 beam down for a couple of minutes R20_90

17.40 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment20)
       Rate spectra (attachment21)
       FEE temps (attachment22)
       Leakage currents written to sheet (attachment23)
       Tapeserver ~45MB/s
       Merger ~ 4.5M items/s

      

15:30
  TD found the merger was running at ~43 Mb/s
  The odd-numbered FEE64s all had significant (>200k) disk rates
  ALL LEC/MEC fast discriminators and increased slow comparator from 0xa to 0x14
  Merger rate dropped to a more nominal value like 2.3 Mb/s
  Settings not saved

40Ar beam 300 MeV/u expected, but SIS problem at the moment.

23:00
  Leak currents are stable as previously (20-21 Nov) and entered to Google spreadsheet
  S4 vault is closed.
  Presently AIDA is running without tape storage
  System wide checks: 
    Master clock / aida01 sync reports error as expected; 
    All white rabbit decoders fail, but they are just off by 1 bit (no problem)
    Everything else okay.
  Stats look okay, but keep in mind above discriminator threshold -- see attachment 1
  Temperatures fine -- see attachment 2
  MBS looks okay -- we see a possible hint for the 1 bit difference with an "unexpected data word" -- see attachment 3 
  Data rate is 1.7 Mb/s, merger okay -- see attachment 4

23:30 
  Now we wait for the beam

00:05
  Attachments 5-11: results of system wide checks after collecting sync errors and WR errors from baseline - all ok

00:25
  Histograms zeroed at 00:25.

00:35

  1.8.L pulser peak widths

FEE64       width(ch)
  1            92
  2            131
  3            75
  4            97
  5            89
  6            117
  7            90
  8            87
  9            223
 10            58
 11            112
 12            126 

no peak observed on aida09

For comparison, pulser peaks measured on 13/11/19:

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

2:00
  40Ar beam tuning is finished.  Intensity at S4 is about 3 kHz per spill.  1 spill per / 10 seconds
  MBS system cannot be used for online merging tonight, because NeuLAND group left their connection open
  We can merge offline using the White Rabbit timestamps
  Remove the beam blocking steel flange from in front of AIDA et al.
  Writing to file Dec19/R1

03.20 Rate, Stat, 1.8.L, 1.8.H, 1.8.W spectra - attachments 13-20
      Observe
       light ions in 1.8.L spectra
       low energy (<1.5GeV) heavy ions in 1.8.H spectra with most heavy ions stopping in DSSSDs 1 & 2
       Stat hit patterns indicate broad (c. 5cm FWHM) centred heavy ion peak

03.24 Detector biases & leakage currents OK - attachment 21
      FEE64 temperatures OK - attachment 22
      Good event statistics (12 of 12 < 50k) - attachment 23

      System wide checks
      List All Sync Error Counters - attachment 24
      List all Sync Received Counters - attachment 25

      Baseline invoked earlier c. 00.05
      Collect all SYNC Error Counters for Baseline
      Collect All White Rabbit status Error Counters for Baseline 

04:07 2.6 mm Al degrader. 300Hz beam. 300 MeV/u 40Ar beam.

04.14 pulser peak seen in aida09 after performing ASIC check-load earlier in shift - width = 219 ch

04.45 1.8.H energy spectra (attachment 26/27)

06:08 attachments 28-30: detector bias/leakage currents ok
                         temperatures ok
                         good event statistics ok
                         all system wide checks ok

08:49 DAQ stop/start. TapeServer writing to No Storage. End of beam.


      

21.35 AIDA DAQ currently running, TapeService on, not writing to storage

      System wide checks ok

      Attachments 1-3: temperatures, good event stats, detector bias/leakage currents ok

22.04 waiting for beam. Possibility of fragments, further news to come

22.43 expecting fragments with 300 MeV/u 40Ar primary beam

      histograms zeroed

00.31 still waiting for beam

      detector bias/leakage currents ok (see google spreadsheet)

      FEE64 temps and good event statistics ok

02:02 still awaiting beam - issues with interlock

03:17 beam tuning - centering primary beam on target

03:43 beam reached AIDA, fragmentation beam (see elog entry by DK for PID)

      primary beam 300 Mev/u 40Ar, fragments in region of 34Si

      started writing to file Dec19/R2 at rate ~ 4000 kb/s

      histograms zeroed

03.49 system wide checks all ok

      attachments 4-6:

      temperatures ok

      bias/leakage currents ok

      good event statistics ok, except aida09 running slightly fast

04:03 attachments 7-8 - 1.8.H energy spectra

                      - no high energy events seen in aida10 -> checked ASIC control

                      - still no high energy events in aida10

04:09 attachment 9 - rate spectra for all FEE modules

04.16 attachment 10 - 1D hits for all FEE modules

04.19 beam stop - file R2_3

04:52 beam back on - file R2_6 - rate of ~ 5000 ions per spill

06:13 temperatures ok

      good event statistics ok (aida09 still running fast) - attachment 11

      detector bias/ leakage currents ok - see google spreadsheet

      all system wide checks ok

      1.8.H spectra - attachments 12/13

06:20 attachment 14 - rate spectra for all FEE modules

      attachment 15 - 1D hits for all FEE modules

      still no high energy events in aida10


      

Arrived at 09:54 - sequenced all FEE64 on

FEE64 not mounted properly - something happened to network overnight, see attachment 1,2 for AIDA system log

10.23 - FEE64 still not mounted properly, power cycling FEE64

10.30 - sequenced all FEE64 on

10.43 - FEE64 mounted successfully

      - DAQ reset and setup - looks ok

      - histogram enabled on aida12

      - clock ReSYNC

      - system wide checks - all FEE64 pass clock status check
           
                           - aida2 and aida12 fail ADC calibration

                           - SYNC pulse and SYNC error checks ok

                           - WR decoder check returns errors (attachment 3)


10.53 - FEE64 temperatures ok (attachment 4)

      - detector bias/leakage current ok (attachment 5)

11.04 - TapeServer start - not writing to storage

      - ASIC check load performed

      - DAQ start

      - ASIC control tab (attachment 6)

      - Run control keeps reloading - has returned to 'reset' screen with aida11 showing "undefined" (attachment 7)

      - good event statistics - all FEE64 showing 0 *except* aida09 (attachment 8)

11.20 - stopped tape server

      - reset daq - returns error (attachment 9)

11.41 DAQ off, detector bias off, FEE64 off

13.30 detector bias on, FEE64 sequenced on

      all browser tabs reopened

      merger set back up - ok

      mbs data relay set back up - ok

      DAQ setup and started, initially ok

      good event statistics (attachment 10)

16.25 FEE64 restarted, along with Merger, TapeServer, DAQ (not writing to file)

      thresholds (slow comparator, fast comparator HEC, fast comparator LEC) raised to 0xff for testing.

      aida10 stats appear reasonable, however wave events still reported despite being deactivated (see report entry
on DAQ stall) (attachment 11)

17.00 DAQ, TapeServer, Merger all stopped

      Detector bias off

      FEE64s switched off

      coolant water still running (cannot be disabled due to lack of access)

      AIDA in safe state

aida10 runs very fast for a period, then stops.

Have to restart Merger and TapeServer to get it running again.

As shown in attachment 1, aida10 records very high number of bad events.

Waveform for all FEE64s are disabled (see for example attachment 4) *BUT* note that 'wave events' are reported
in aida10 stats (attachment 1)

19 November

Waiting for 40Ar beam to be delivered for tests.
Objective today is to assess degrader thickness suitable to stop 40Ar in 2nd DSSD layer.
Energy expected at 300 MeV/A, around 1E3-4 beam intensity - to be determined.

Tomorrow fragments could be expected.

DSSD / FEE64 pairing as follows

DSSD3 (most downstream): FEEs 9,10,11,12
DSSD2 (middle): FEEs 5,6,7,8
DSSD1 (most upstream): FEEs 1,2,3,4
+plastic scintillator furthest upstream

New AIDA Google spreadsheet created with leakage current, to monitor them during beamtime. First entry added. Bias attached.
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023

DESPEC platform has already been moved. No effect on rates - see attached. FEE1 is missing one channel. FEE9, 10 noisy as expected. See attached.


System wide checks *all OK* except
Master clock failed (normal)
ADC calibration fails on 6 and 12. Recalibrated via FADC Align. All OK now.

MBS transfer OK. See attached.

Good event statistics OK. See attached.

FEE64 temperatures OK. See attached.

Merger and data rate OK. See attached.

Waiting for beam.



20 November

01:07 - Still no beam. 

System wide checks OK. 
Stats OK. 
Temps OK. 
MBS OK. 
Merger and data rate OK.


02:35  - Beam is expected soon
Tried enabling writing data to disk. Merger crashed. FEE64 stopped responding to commands.
Reset Merger. FEE64 now respond to commands.
Tried enabling writing to disk again. Merger crashed once more. Reset merger.
Reset Tape Server interface. Now writing to disk.

No data currently being sent to MBS, but operators say they "are adding a new detector". May be normal.

DAQ & Merger going OK. Writing to Nov19/R1

No beam yet.

Temperatures OK
System wide checks OK
Stats OK. See attached.
MBS not working.
Merger and data rate OK. See attached.

~02:48 Beam on. 40Ar. 300 MeV/A, 500 pps. Degrader settings unclear.

03:54 Degrader settings have been changing between 4.5, 1 and 6 ug/cm2
Currently on 6 g/cm2.
AIDA seen through MBS shows no HE on detector 3, and Det 1 and 2 show no counts on positive y (FEE6 and 1). That is not the case from MIDAS.

04:30 The aim of the tests above was to have no 40Ar at all in DSSD3. It does not appear that 6 g/cm2 is sufficient, as we still see events. The reduction between 1 & 2 vs. 3 is about a factor of 2-10 depending on ADC channel. See attached.

~04:30 Moved to 6.2 g/cm2 degrader. See attached.

~04:45 Moved to 5.8 g/cm2. See attached.

~05:09 Moved to 6.4 g/cm2. See attached.

05:31 High energy channel spectra (1.8.H) for all FEE64 w/ 6.4 g/cm2 degrader - See attachments 13 & 14

05:38: Moved to 1.0 g/cm2. See attached. Difference is not huge which may indicate these spectra are misleading, as partly expected. A more detailed analysis will be carried out off line.

06:00: Beam stopped. ~R77. AIDA keeps running background over the day. Beam should restart around 11 pm later today.

21:35 (NH) - A few merger timestamp errors were appearing - Decided to powercycle all FEE64s, all checks OK.
 No Screenshot due to forgetting before I closed merger 
 FEEs were often not stopping properly and needing merger reset as well.

Beam Plan:

40Ar 300 MeV/u Primary Beam
and/or
34Si Fragment from Ar

22:45

FEES were restart near these timestamps in the TapeData, evidently
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:17 R1_361
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:36 R1_362

Run number was kept the same (Tape Server was not stopped @ FEE restart?)

All system wide checks are okay (except of course the master clock fails)
Temps look okay (attachment 1, dated 21/Nov by mistake)
Leak currents all around 0.9 uA as before (attachment 2)
Stats are behaved. We don't have beam yet (attachment 3)
Merger and tape server happy (see attachment 4)
Data rate histos while beam is still off (see attachment 5)

==Nov 21==

0:00
TD noted that the data rates have gone up at some point (about double).  
Now we increased the slow comparator thresholds from 0xa to 0xb (data rate drops from 24 Mb/s to 16 Mb/s).
We increased it again from 0xb to 0xc, but near that time the beam was also turned on, and the data rate was about 25 Mb/s.

Beam was turned on near R1_472 or a bit earlier.

TD requested waveforms, but at the moment they are not appearing.  Will enable WFs after a condition change.

Beam seen in all three DSSSDs, see attachment 6.  Beam was turned off before I could get more screencaps.

Degrader change ... near R1_481

00:32 waveforms now enabled under Run Control.  They also appear to be enabled on ASIC1 undre LED and Waveform controls.  However, we are unable to see the waveforms in the relevant histograms.

00:44 3 g/cm^2 inserted R1_488

00:45 3.2 g/cm^2 degrader in R1_489.

1:00 Several degraders and combinations are being tested since last note.

1:10 It seems 1 g/cm^2 of energy loss is not accounted for by LISE++.  Trying to understand the reason.

1:28 Satisfied with a total degrader thickness of 2.6 g/cm^2 -- seems to calculate that fragments should implant mainly in DSSSD2?  Issue seems to be resolved updating the LISE++ file.
  Near R1_507

Can see the implant samples attached as #7 and #8

1:43 According to the stats, most of the beam energy is being deposited in DSSSD1.  See #9.
  Present beam burst repetition is around 0.1 Hz (each ~10 sec)

2:00 better example of stats histos (see #10)

2:45 tuning the FRS degraders (ones for RIB separation rather than AIDA implantation depth).

3:17 Just started fragments.  Disregard previous filenames with "34Si".  Finally no degrader in the FRS was needed to get Z separation.
  R1_559
  As the DAQs are not synchronized, a quick call to the BNC PB-5 pulser shifiting from 2 -> 50 -> 2 Hz was used to make a signal burst.
  S4 degrader is 5.2 g/cm^2

3:22 Updated biases in Google Sheets.

Having trouble seeing the fragments in the AIDA histograms.  Since we just started fragments, zero the histos.

3:35 Attachment 11, with 34Si fragments as Stats.  Difficult to click Update on the Rates at the right time.  
   From the beam monitors, fragments should be at around 3000 cps.

3:45 (Up to now, some optics tuning with slits, etc)

3:46 Optics settings accepted, AIDA at R1_572, accept this condition for data runs on ~34Si fragments.

3:47 Sent a pulser burst 2 -> 50 -> 2 Hz

3:50 zero histograms again.
   Temperatures attachment 12
   Biases attachment 13
   Statistics attachment 14
   Merger / Tape Server attachment 15

3:56   
   StatHistos attachment 16

4:09 FRS starts new run #56, AIDA R1_583.  Pulser 2-> 50 -> 2 Hz

4:56 Stat histograms in attachment 17.  Not zeroed since noted at 3:50

/MIDAS/Linux/startup/datarelaymbs 

18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

18:24 Rate spectra - attachment 4

18:30 Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

19:28 beam back after a brief outage - AIDA on file R9_736

      FRS having DAQ issues

19:57 FRS DAQ/Scalars back

20:12 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

      Merger ok - 4.5M data items/s

      TapeServer ok - 46MB/s

      Data forwarding to MBS ok

20:33 No beam - AIDA on file R9_821

21:08 rates spectra - attachment 8

22:35 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 9
      FEE Temperatures - attachment 10
      Bias and leakage currents ok - attachment 11

      Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

      temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload.

23:07 media/ThirdDrive at 65% capacity

      writing to file R9_1021

      file compression currently up to R1_686

23:36 beam stable ~1e9 pps

      rate spectra - attachment 12




      

00:13 beam back, but some spills 'missing'

      DESPEC keeping beam as is, but may stop later if it worsens

00:15 DAQ continues OK - file R20_622

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:20 System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2

      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1

     If any of these counts are reported as in error
     The ASIC readout system has detected a timeslip.
     That is the timestamp read from the time FIFO is not younger than the last

00:24 still no bad timestamp errors in NewMerger since 16:30 UTC

00:30 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3

      Merger OK - 4.2M data items/s
      TapeServer OK - 45 Mb/s

01:30 rate spectra - attachment 4

      HEC spectra - attachment 5 & 6

      note aida04 spectrum still not showing

02:07 beam off - file R20_760

02:11 beam back - file R20_764

02:44 System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2


      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1


      still no bad timestamp errors in NewMerger since 16:30 UTC

      FEE64 Temperatures OK - attachment 7
      Good event statistics OK - attachment 8
      Detector bias & leakage currents OK - attachment 9

      Merger OK - 4.2M data items/s
      TapeServer OK - 44 Mb/s

04:16 bad timestamp errors in NewMerger terminal, first since 16:30 UTC - attachment 10

      System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2


      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1


      FEE64 Temperatures OK - attachment 11
      Good event statistics OK - attachment 12
      Detector bias & leakage currents OK - attachment 13

05:03 Merger OK - 4.5M data items/s
      TapeServer OK - 43 Mb/s

      no bad timestamp errors for ~40 mins, data forwarding to MBS ok at usual rate

06:11 rates spectra - attachment 14

06:14 System wide checks

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2

      Check WR decoder status
      Base 		Current 	Difference
      aida05 fault 	 0x1591 : 	 0x1593 : 	 2  
      aida06 fault 	 0xe23c : 	 0xe23e : 	 2  
      aida07 fault 	 0x743 : 	 0x745 : 	 2  
      aida08 fault 	 0xf84e : 	 0xf850 : 	 2  
      aida09 fault 	 0x5449 : 	 0x544a : 	 1  
      White Rabbit error counter test result: Passed 7, Failed 5

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
   

      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1

      still no further bad timestamp errors in NewMerger terminal

      collected all WR and FPGA errors from baseline, system wide checks ok

      FEE64 Temperatures OK - attachment 15
      Good event statistics OK - attachment 16
      Detector bias & leakage currents OK - attachment 17

06:24 AIDA writing to file R20_1080

      Merger OK - 4.5M data items/s
      TapeServer OK - 46 Mb/s

06:26 another burst of bad timestamp errors - attachment 18

      performed system wide checks again - all ok aside from aida07/aida10 calibration

07:32 HEC spectra - attachments 19/20

07:35 no beam

      


      

00:10 DAQ continues OK - file R30_372

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:17 System wide checks all OK

      FEE64 Temperatures ok - attachment 1

      Good event statistics ok - attachment 2

      detector bias and leakage currents ok - attachment 3

00:20 DESPEC on run 145

00:24 Merger ok ~45M items/s
      Tapeserver ok ~15MB/s

01:00 aida07 crashed

      recovered, but all FEE64 crash shortly after

      called OH - stop DAQ, TapeService, Merger

01:20 all FEE64 powercycled, AIDA restart

01:30 AIDA recovered, DAQ now running - writing to R33

      system wide checks ok *except*

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

01:52 FEE64 Temperatures ok - attachment 4

      -had to reload a few times to work, but otherwise ok

      Good event statistics - attachment 5

      -aida05 and aida08 running faster than before

      detector bias and leakage currents ok - attachment 6

02:04 writing to file R33_34

      Data forwarding to MBS ok

      AIDA ASIC settings ok

02:11 beam off

02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails 

02:32 beam back - writing to file R33_48

04:30 DESPEC having issues with Go4 crashing

      ucesb reports AIDA/FATIMA/bplast timewarp events

      System wide checks:

      Clock error:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Calibration:
      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      White Rabbit:
       Base 		Current 	Difference
       aida07 fault 	 0xcdd1 : 	 0xcdd2 : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 Temperatures ok - attachment 7

      Good event statistics ok - attachment 8

      detector bias and leakage currents ok - attachment 9

04:52 analyzer output for R33_113 - attachment 10

      no timewarps

      aida06 dead time? (ignore idle time and rates)

05:02 Merger 5M data items/s

      TapeServer 17 MB/s

05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash

      They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now

05:25 rates spectra - attachment 11

05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12

06:29 system wide checks:

      Clock error:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Calibration:
      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      White Rabbit:
                         Base 		Current 	Difference
      aida05 fault 	 0x7a56 : 	 0x7a58 : 	 2  
      aida06 fault 	 0x65bb : 	 0x65bd : 	 2  
      aida07 fault 	 0xcdd1 : 	 0xcdd4 : 	 3  
      aida08 fault 	 0x2ab3 : 	 0x2ab5 : 	 2  
      White Rabbit error counter test result: Passed 8, Failed 4

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      rest ok

06:32 FEE64 Temperatures ok - attachment 13

      Good event statistics ok - attachment 14

      detector bias and leakage currents ok - attachment 15

06:41 Merger 5.1M data items/s

      TapeServer 17 MB/s

      Data forwarding to MBS ok

08:54 restart MBS relay, requested by NH