00.30 UNILAC problem - no beam

00:39 DAQ continues R1_518
      FRS 190W setting 

      ASIC settinfs 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

01:07 all system wide checks OK

      detector biases & leakage currents OK - see attachment 1
      FEE64 temperatures OK - see attachment 2
      good event statistics - OK - see attachment 3
      Merger OK - see attachment 4
      TapeServer OK - see attachment 5

01:20 zero'd all histograms
      ASIC check load

      rate spectra - see attachment 6

01:48 beam return

      run74 (R1_608)

03.05 all system wide checks OK

      detector biases & leakage currents OK - see attachment 7
      FEE64 temperatures OK - see attachment 8
      good event statistics - OK - see attachment 9
      Merger OK - see attachment 10
      TapeServer OK - see attachment 11

     rate spectra - see attachment 12

04.46 run74 ends (R1_841)

05.00 run75 (R1_859)

05:40 detector biases & leakage currents OK - see attachment 13
      FEE64 temperatures OK - see attachment 14
      good event statistics - OK - see attachment 15
      Merger OK - see attachment 16
      TapeServer OK - see attachment 17

     rate spectra - see attachment 18

07:29 run76 (R1_1055)

07:50 Pulser settings - attachment 1

07:52 System wide checks all ok
      Statistics ok - attachment 2
      Bias ok - attachment 3

09:30 The intensity of fragments is being increased
      They are trying to do a factor of 10 increase first which should take us to around 100Hz

10:00 System wide checks all ok
      Statistics - attachment 4
      Layout 1 showing implants - attachment 5
      Fee temperatures ok - attachment 6

12.00 System wide checks all okay, no failures
      Statistics (screenshot7)
      Spectrum rate (screenshot8)
      FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal 
      (screenshot10)
      Leakage currents okay, increase from previous values written to sheets (screenshot12)
      
      Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer 
than 
      usual, another reload shows all FEE temps as normal.
      
      Merger and tape server okay

14.00 System wide checks all okay, no failures
      Statistics (screenshot13)
      Spectrum rate (screenshot14)
      FEE temps normal, no issue with reload this time (screenshot15)
      Leakage currents okay still increasing, written to sheets (screenshot16)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)

16.00 System wide checks all okay, no failures
      Statistics (screenshot17)


      Spectrum rate (screenshot18)      
      FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
      Leakage currents added to sheet (screenshot21)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

17.17 no beam, preparing for the 190Ta setting

17:44 OH Takes over
      System wide checks all ok
      Statistics - attachment 22
      Temp ok - Attachment 23
      Bias and leakage currents ok - Attatchment 24

      While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
      The Directory for the experiment was then setup S452
      The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.

      ASIC Check all ok
      

18:10 Started writing data to file
      File directory S452
      Run number R1_0
      Seems to have skipped a few files First full file R1_10
      Currently writing to /media/SecondDrive/TapeData
      Current free space 4211084180 kB
      Current write speed 46163 kB/s
      Time remaining until full 91222 seconds
      25 hours remaining space

18:34 DESPEC starting new MBS file
      AIDA Currently on file R1_41

19:00 DESPEC Closing file
      AIDA Currently on R1_75

19:05 Comparing last years statistics to this years

FEE	Last year	This year    Factor increase
1	60960	        183111       3.00
2	183494	        201256       1.09
3	91506	        196647       2.15
4	73914	        208290       2.81
5	31979	        63571        1.99
6	54084	        144638       2.67
7	35621	        137605       3.86
8	38626	        95858        2.48
9	264436	        225790       0.85
10	209709	        198896       0.94
11	88443	        133899       1.51
12	183493	        204996       1.12

Sum     1316265	        1994557      1.51

19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
      AIDA on R1_104

      Looking int to high write  rate to see if we can do anything to reduce it
      It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24

19:46 Much of the rate is coming from a small number of channels - attachment 25

19:17 WR Timestamp error.
      Mentioned to Nic and he said that sometimes small glitches like this are observed
		 Base 		Current 	Difference
aida01 fault 	 0x9e8b : 	 0x9e8d : 	 2  
aida02 fault 	 0x9a8f : 	 0x9a91 : 	 2  
aida03 fault 	 0x29e9 : 	 0x29eb : 	 2  
aida04 fault 	 0x7f66 : 	 0x7f68 : 	 2  
aida05 fault 	 0x1f4f : 	 0x1f51 : 	 2  
aida06 fault 	 0x8faa : 	 0x8fac : 	 2  
aida07 fault 	 0x53f0 : 	 0x53f2 : 	 2  
aida08 fault 	 0x7066 : 	 0x7068 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     All other checks passed

20:52 DESPEC Stopping file
      AIDA on R1_221


20:53 DESPEC Stopped file because UCESB crashed
      AIDA on file R1_237

21:20 All system wide  checks ok
      Statistics DISC info- Attachment 27
      FEE Temp ok - Attachment 28
      Bias and leakage current ok - Attachment 29
      Statistics good events - Attachment 30

20:51 Started compressing data with command nice -n 10 gzip -v R1_*
      Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
      I am not sure if MBS can keep up with this rate
      We are still writing to file though.

23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us

aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S452 shift checks - attachment 1

S452 shift plan - PDF attachment 2 - Google doc https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1745844766

Mattermost - https://mattermost.gsi.de/despec/channels/experiment-s452-nearline-sort

DESPEC Zoom meeting

Join Zoom Meeting
https://gsi-fair.zoom.us/j/98980822569

Meeting ID: 989 8082 2569
Passcode: S452

FRS Zoom meeting 

Meeting-ID: 973 4809 5114

password: FRS-BT2021

Kenncode: 456657

00:23 206Hg setting - No beam in AIDA yet

      AIDA not writing to file

      Data being forwarded to MBS

      Fragments expected tonight (?)

00:45 GSI DAQ restart

      FEE64 Temps ok - attachment 1

      All system wide checks ok

01:00 Good event statistics - attachment 2

01:48 detector biases/ leakage currents - ok - attachment 3

02:10 beam on, plug removed

      beam delivered to AIDA

      rates spectra - attachment 4

      peaks at ~ 10 counts/channel in DSSD1 c.2000Hz

02:20 beam stop

02:26 beam restart - rate spectra - attachment 5

02:43 high energy channel spectra - attachment 6

      saved the high energy spectra to layout 2 in spectrum browser

03:00 beam stop

03:08 temperatures ok - attachment 7

      statistics ok - attachment 8

      bias/leakage currents ok - attachment 9

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

      all system wide checks ok

04:24 beam is back

      rate spectra - attachment 10

04:51 206Hg implanting in DSSD2

      Rate spectra - attachment 11

      HEC spectra - attachment 12

05:08 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias/leakage currents ok - attachment 15

      All system wide checks ok

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok 

05:45 beam stopped

05:50 beam back

06:45 beam running, but no implants in AIDA

      possible issues with beam delivery to S4

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

07:10 temperatures ok - attachment 16

      statistics ok - attachment 17

      detector bias/leakage currents ok - attachment 18

      all system wide checks OK














      

      

      

There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.

The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.

Beam plug is in front of AIDA. This is causing a huge spike in the Ge.

During testing for the S452 experiment yesterday we were running through the steps of a restart.

The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.

Checking the folder manually it could be seen that the Options file was there.

At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.

This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.

During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.

At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally.

screen -S caenhv /dev/ttyACM0

screen -x caenhv

screen -S pulser /dev/ttyUSB0

display settings

help

tmux attach -r

./check_usbs.sh

cat /MIDAS/TclHttpd/Html/ETape/1.txt

ssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup

rate -nst -rst -nev -rev -ndata -rdata -st

/MIDAS/Linux/startup/datarelaymbs 

Updated figures as of 27 Feb 2021 - S4 is now closed for experiments but may open on Wednesday.
Snout has been grounded and all FEEs are seemingly grounded properly.

Figure  1: Good Events
Figure  2: Rate Histograms (Log-y)
Figure  3: Waveforms (Odd)
Figure  4: Waveforms (Even)
Figure  5: Waveforms @ 5MS/s (Odd)
Figure  6: Waveforms @ 5MS/s (Even)
Figure  7: Waveforms with Pulser (Odd)
Figure  8: Waveforms with Pulser (Even)*
Figure  9: Waveforms with Pulser @ 5MS/s (Odd)
Figure 10: Waveforms with Pulser @ 5MS/s (Even)*

* y-Scale enlarged from 0-10000 to 0-15000 to capture top of pulse

The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.

(All data below taken with thresholds 0xA for all)

Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"

Leakage currents high but S4 very warm at the moment as roof is still closed.

Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.

Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done

Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)

aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3

A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21

Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64

S4 access will be limited. 

I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal.

> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
> 
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
> 
> The system has been powered up and biased successfully. To be monitored over the next few days.
> 
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
> 
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
> 
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
> 
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.

The hose has arrived and the water line will be prepared in my office mid next week.

All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)

A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.

The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.

There will be a DESPEC Dry Run next week Mon-Tue.

01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.

For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)

The system has been powered up and biased successfully. To be monitored over the next few days.

A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.

A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...

The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.

Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

AIDA Frame is now grounded to platform (will add photo later)

Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now)

FYI:

The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were *huge* and filled with the same error many times a second

Oct  4 15:01:22 raspberrypi kernel: [1030048.603846]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753]                                                                                             

A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version.
now the message doesn't seem to appear so hopefully the pis won't break so rapidly.

MIDAS was not working properly on nnrpi2 - power cycled. Now system does not boot properly, it gets stuck trying to login claiming "PAM critical error" - probable file system corruption?
Will investigate if SD card can be checked and repaired

Last week there was a power failure in Messhuette (FRS control room) - AIDA-3 and MBS were affected.
AIDA itself and S4 equipment is on UPS and was not shut down.

AIDA-3 has been restarted and MIDAS restarted. All seems OK

S4 conditions: 24.2 C / 33.5% RH / Td = 7.1 C 

Loaded new frimware into all 12 FEE64s in the system. IWR_Dec19_4.bin 

Booted ok after power-cycle.  

RESET/SETUP/GO all fine.

system wide checks ( SWC ) clocks, white rabbit ( WR ) OK, ADC calibrated : aida10 failed. Couldn't get them to calibrate.

Checked aida01 waveforms for ASIC1. set threshold for LED to 5000 to stop triggering in the noise.

Set White Rabbit register SYNC Rollover Target ( 14 ) to 0x4000 instead of 0xe to reduce the data rate to better understand the waveform activity.

All FEE64s are set to enable all channels of ASIC1 with low thresholds so a lot of data moving. Checked activity using statistics window.

Checked waveforms working for ASIC1 of aida01. Noisy but capturing the pulser pulse.

 Enabled all the waveforms on aida01 to see if they all function ok. After a few seconds waveform readout stopped. Not sure of the reason.

STOP, change all four thresholds to 5000, Check ASIC controls : all ok, GO. No activity.

RESET/SETUP ok.

SWC : clock failed, ADC failed , WR ok ????

Power-cycle...... concludes ok, RESET/SETUP ok, SWC : clocks all failed, ADC all failed, WR ok.

Checked WR status browser page for aida01. WR timestamp 0 ! status : 0xA should be 0 .

Checked tcl for SWC WR check and it doesn't check the status. Changed at DL and at GSI and now it does.

so SWC WR check no all fails.

Contacted NH and he found Fatima VME crate with WR source in had been switched off. It was powered on again.

RESET/SETUP , SWC: clocks all ok, ADC all ok, WR all ok.

set aida01 and all odd number  LED thresholds to 5000 and aida02 all even number thresholds to 10000 and activity of statistic Wave Good Events is similar across the system. 

Checked each of the Waveform capture controls pages ( only shows the status registers for ASIC1 ) and all are fine, nothing sticking.

enabled all the ASIC waveforms on aida01 again and it sticks, clear out three ASICs and operate the waveform reset in the System Functions and it restarts and runs.

So a qualified success. Don't enable too many channels at once ?

left running at 18:00 UK time and checked at 21:30 UK time , still operating ok.

08:25 Beam Stopped, end of S480

Tape Server stopped at R12_2097

Started again in no storage mode

Will leave compressing runs