AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 14 of 36  ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
  ID Date Authordown Subject
  157   Wed Mar 18 18:02:37 2020 PJCSNew Firmware loaded in all FEE64s

Loaded new frimware into all 12 FEE64s in the system. IWR_Dec19_4.bin 

Booted ok after power-cycle.  

RESET/SETUP/GO all fine.

system wide checks ( SWC ) clocks, white rabbit ( WR ) OK, ADC calibrated : aida10 failed. Couldn't get them to calibrate.

Checked aida01 waveforms for ASIC1. set threshold for LED to 5000 to stop triggering in the noise.

Set White Rabbit register SYNC Rollover Target ( 14 ) to 0x4000 instead of 0xe to reduce the data rate to better understand the waveform activity.

All FEE64s are set to enable all channels of ASIC1 with low thresholds so a lot of data moving. Checked activity using statistics window.

Checked waveforms working for ASIC1 of aida01. Noisy but capturing the pulser pulse.

 Enabled all the waveforms on aida01 to see if they all function ok. After a few seconds waveform readout stopped. Not sure of the reason.

STOP, change all four thresholds to 5000, Check ASIC controls : all ok, GO. No activity.

RESET/SETUP ok.

SWC : clock failed, ADC failed , WR ok ????

Power-cycle...... concludes ok, RESET/SETUP ok, SWC : clocks all failed, ADC all failed, WR ok.

Checked WR status browser page for aida01. WR timestamp 0 ! status : 0xA should be 0 .

Checked tcl for SWC WR check and it doesn't check the status. Changed at DL and at GSI and now it does.

so SWC WR check no all fails.

Contacted NH and he found Fatima VME crate with WR source in had been switched off. It was powered on again.

RESET/SETUP , SWC: clocks all ok, ADC all ok, WR all ok.

set aida01 and all odd number  LED thresholds to 5000 and aida02 all even number thresholds to 10000 and activity of statistic Wave Good Events is similar across the system. 

Checked each of the Waveform capture controls pages ( only shows the status registers for ASIC1 ) and all are fine, nothing sticking.

enabled all the ASIC waveforms on aida01 again and it sticks, clear out three ASICs and operate the waveform reset in the System Functions and it restarts and runs.

So a qualified success. Don't enable too many channels at once ?

left running at 18:00 UK time and checked at 21:30 UK time , still operating ok.

 

 

  209   Wed Mar 31 12:46:10 2021 PJCSHowTo : Calibrate the LMK3200 clock devices

Here is a document showing how and where to calibrate the LMK3200 devices that lock to the system clock and generate the clocks for the ADCs and the FPGA internal logic.

Attachment 1: Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf
Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf
  210   Wed Mar 31 15:40:42 2021 PJCSCheck Options files function added

There is a new operation available in the System Functions menu in the System Wide Checks page. ( The page needs to be reset once when the FEEs are powered before this function will appear )

Check the Options files are all the same size

Using this function will provide a list of the file sizes of the Options/< fee name >/CONTENTS files along with the last time they were accessed as a list.

It's up to the user, at present, to interpret the results but more is possible if required ....

  211   Wed Mar 31 15:46:09 2021 PJCSHowTo : Synchronize the ASIC clocks.

To Synchronize the FEE64 ASIC clocks to rise at the same timestamp time use the System function Synchronise the ASIC clocks in the System Wide Checks browser page.

The Server will read the current timestamp value and calculate, based on the number of FEEs and the access delay, a timestamp value sometime in the future. F_stamp

F_stamp is written to each FEE and the synchronization is enabled. 

In a state machine in the FEE ASIC clock control, logic is enabled to synchronize its ASIC clock. It compares the current timestamp with F_stamp and if the current timestamp is >= F_stamp then it will start the ASIC clock.

The report in the browser log window gives the 3 LSBs of the timestamp at the instant the ASIC clock is started. They should all be the same. They are from the 10ns timestamp.

 

  220   Wed Apr 14 12:14:47 2021 PJCSCorrections and changes

14/4/21 @12:00 UK time

Corrected the failure of FADC re-calibration All modules. The problem was a missing > in the .tml file

Edited the sys.tcl file in /MIDAS/TclHttpd/Html/RunControl to comment out where the status of the waveform enable was able to stop the creation of the .W spectra. This will now function at SETUP.

Edited the sys.tcl file to remove the test for Master module at the enable of the Correlation Scalar readout. All modules will now be enabled.

Successfully re-calibrated the LMK3200 in aida09 to remove the clock fault reported in System Wide Checks.

Successfully "sync ASIC clocks" followed by calibrate all module FADCs ( except aida09 which required seperate attention. )

All the Edits first carried out and tested at Daresbury T9 system.

  221   Wed Apr 14 15:14:55 2021 PJCSNote Well

Please note that for the forseeable future the system function in the menu on the System Wide Checks browser page labelled as 

"Collect the Timestamp RAM values"  and "Start the Readout Timestamp error tracing" 

Are for the use of engineers only unless otherwise instructed. 

There is no harm in using them but the results may be confusing and it is not possible, at present, to detail a definitive set of instructions for their use.

  300   Fri May 7 19:47:54 2021 PJCSHowTo mitigate excessive temperature in an FEE64

After tests in the Daresbury T9 system.

Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees.

To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF

The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected.

Alternatively STOP acquisition, set the ADC Control  register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.

 

  396   Wed Feb 16 10:32:52 2022 PJCSAIDA software changes and some suggestions

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

  490   Wed Jun 22 10:09:12 2022 PJCSINFO: FEE64 supply Voltages

Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

  514   Wed Sep 14 19:07:07 2022 PJCSINFO : Three Merger Statistics explained

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

 

  225   Fri Apr 16 00:47:36 2021 OH-MLAIDA DAQ Reset

1:18 (CET) No rate in AIDA 07.

Oscar did a AIDA DAQ reset and a quick wide check.

(Power cycle

All back to normal.

  1   Thu Sep 6 15:11:17 2018 OH, TDTuesday 4 September - Thursday 6 September
Attachment 1 - FEE64 temperatures illustrating (we assume) a faulty PSU temperature sensor for aida07

Attachment 2 - FEE64 temperatures following replacement of aida07

Attachment 3-5 - 1.*.W waveforms for aida01 - aida03 - LED settings for aida02 incorrect, aida01 adaptor PCB 
incorrectly aligned

Attachment 6-7 - 1.*.W waveforms for aida02 and aida04

Attachment 8-9 - aida01 - aida04 1.8.L spectra full range and expanded scale
                          pulser peak width c. 15-20 channels FWHM

Attachment 10-12 - good events stats for all FEE64s and all statistics for aida01 and aida05
                   Note aida01 - aida04 show rates expected, all other FEE64s show very rates due to noise
                   aida01 - aida04 adaptor PCB are well grounded to FEE64 copper mezzanine/cooling plate, the other 
FEE64s are not

Attachment 13-14 - aida01 - aida04 rate spectra before and after correcting alignment of aida01 adaptor PCB

Attachment 15 - aida01 - aida04 1.8.L spectra, expanded scale

Attachment 16 - illustrates high frequency noise (period 5 samples = 100ns) for aida01 which is not observed for 
aida02 - aida03
                subsequently determined that this is not due to a possible adaptor PCB fault by changing the adaptor 
PCB
                Note that with bias turned on this noise did appear when Bias was on but all the other groudning
connectors for aida05-aida12
were not in effect.

Attachment 17-19 - PuTTY connection to CAEN 1419ET 4ch high voltage supply illustrating PuTTY, N1419ET configuration 
and leakage current with 100V applied to adaptor PCBs (no input cabling or DSSSD connected)

Note the CAEN N1419ET appears as a 'USB ACM' device and we connect using /dev/ttyACM<n>. 
Attachment 1: 180904_1615_FeeTemp.png
180904_1615_FeeTemp.png
Attachment 2: 180905_1444_FEETemp.png
180905_1444_FEETemp.png
Attachment 3: 180905_1636_aida01W.png
180905_1636_aida01W.png
Attachment 4: 180905_1643_aida02W.png
180905_1643_aida02W.png
Attachment 5: 180905_1644_aida03W.png
180905_1644_aida03W.png
Attachment 6: 180905_1646_aida02W.png
180905_1646_aida02W.png
Attachment 7: 180905_1647_aida04W.png
180905_1647_aida04W.png
Attachment 8: 8_L.png
8_L.png
Attachment 9: 8_L_Zoom.png
8_L_Zoom.png
Attachment 10: 180905_1653_Stats.png
180905_1653_Stats.png
Attachment 11: 180905_1654_aida01Stats.png
180905_1654_aida01Stats.png
Attachment 12: 180905_1654_aida05Stats.png
180905_1654_aida05Stats.png
Attachment 13: 180905_1657_HitRateAida1-4.png
180905_1657_HitRateAida1-4.png
Attachment 14: 180905_1703_HitRateZoomPostAlign.png
180905_1703_HitRateZoomPostAlign.png
Attachment 15: 180905_1707_aida01_1_8_L.png
180905_1707_aida01_1_8_L.png
Attachment 16: 180905_1709_aida01W.png
180905_1709_aida01W.png
Attachment 17: 180906_1424_biasWindow.png
180906_1424_biasWindow.png
Attachment 18: 180906_1425_Putty.png
180906_1425_Putty.png
Attachment 19: 180906_1441_BiasOnNoDSSD.png
180906_1441_BiasOnNoDSSD.png
  2   Thu Sep 6 15:25:19 2018 OH, TDAIDA@DESPEC Setup Photos
  3   Thu Sep 6 15:42:53 2018 OH, TDFEE64 system console logs
Kernel panics during boot cab be observed in files ttyUSB1 and ttyUSB5

In each case the FEE64 successfully completed reboot c. 5 minutes later
Attachment 1: logs.tar
  5   Wed Sep 19 12:40:41 2018 OH, TDSingle DSSD Installation and Tests
On the 18th of September we installed DSSD 3208-14

Was found to have problems including exponential leakage behavior

  V      Ic
 -5   15.4uA
 -10  53uA
 -15  Tripped at over 100uA

Cables were double checked, bond wires were checked and no problems could be observed.

DSSD replaced with 3208-15

Bias at -100V Ic 10.2uA GP/grnd Jumpers removed,
                         rising upwards, possible short circuit as well.

Jumpers replaced

Bias -100V Ic 1.6uA
Bias -160V Ic 1.62uA

Possibility that mechanical sample 3208-17 and 3208-14 have been swapped. (To check at later date)

FEEs powered up and Adapter boards and all grounding installed with DSSD going to 9,10,11 and 12

Rates shown - attachment 1
Waveforms - attachment 2-4
1.8.l Spectra - attachment 5
FEEs 10 and 11 around 60FWHM and 9 and 12 around 80-90FWHM
Stats page - attachment 6

Aluminium foil used to make sure all ports are light tight shows a stats drop of around half - attachment 8

New peak widths

aida09 - 79.65
aida10 - 67.37
aida11 - 59.99
aida12 - 97.83

Going to begin a systematic study of shaping time

                              FWHM
Shaping Time   aida09   aida10   aida11   aida12
   
     8          79.65   67.37    59.99     97.83
     7          97.55   70.84    62.67     115.34
     6         100.72   70.04    62.77     120.05
     4         147.47   80.64    71.34     167.23
     2         217.62   114.98   133.35    232.47

Shaping time 8us 1.8.L Shots 
Shaping time 4us Stats appear obviously worse - attachment 7
                 1.8.L Widths - attachment 8
Shaping time 2us Stats appear worse again - attachment 9
                 1.8.L Widths - attachment 10
Shaping time 6us Stats are better than 2 and 4 - attachment 11
                 1.8.L widths - attachment 12
Shaping time 7us Stats are better than 6 - attachment 13
                 1.8.L widths - attachment 14


8us is the optimum shaping time.
Taken back to 8us and similar shaping times are observed.

Connected the ground of all 4 adapter boards with direct cables. Observed improvements in both FEEs 9 and 12
however the noise in 11 went
through the roof >500kHz - attachment 15

Tested just having 9,10 and 12 connected the rates are comparable to when disconnected - attachment 16

All 4 grounded together with a ground running to the snout as well stats No difference to all 4 together -
attachment 17
              1.8.L Widths - attachment 17
aida09 104.76
aida10 84.58
aida11 281.77
aida12 82.58

Tested just having 10 and 12 directly connected. Same side of the DSSD: Noise is the same as when disconnected.
Stats - attachment 19
1.8.L spectra - attachment 20

aida09 107.13
aida10 67.02
aida11 66.78
aida12 131.51

Tested having 11 and 12 connected directly. This is the two adapter boards responsible for the bias with 11
carrying the core and 12 the braid.
Massively reduced noise in 9 and 12, comparable noise in 10 but much more noise in 11. Stats - attachment 21
aida09 63.96
aida10 76.64
aida11 125.96
aida12 68.05
1.8.L spectra - attachment 23



18:39 Alpha run started in /TapeData/Sep18/R2
      Writing at 96kb/s
      11 Was fast at start bus asic control brought it down




                 
Attachment 1: 180919_1447_7us_1_8_L.png
180919_1447_7us_1_8_L.png
Attachment 2: 180919_1607_FEEsConnected.png
180919_1607_FEEsConnected.png
Attachment 3: 180919_1635_StatsSnoutGrnd.png
180919_1635_StatsSnoutGrnd.png
Attachment 4: 8_L_SnoutGround.png
8_L_SnoutGround.png
Attachment 5: 180919_1648_12and10Connect_Stats.png
180919_1648_12and10Connect_Stats.png
Attachment 6: 180919_1656_10and12_1_8_L.png
180919_1656_10and12_1_8_L.png
Attachment 7: 180919_1710_11and12_Stats.png
180919_1710_11and12_Stats.png
Attachment 8: 180919_1717_11and12_1_8_L.png
180919_1717_11and12_1_8_L.png
Attachment 9: 180919_1310_Rate.png
180919_1310_Rate.png
Attachment 10: 180919_1312_aida10W.png
180919_1312_aida10W.png
Attachment 11: 180919_1312_aida11W.png
180919_1312_aida11W.png
Attachment 12: 180919_1314_aida12W.png
180919_1314_aida12W.png
Attachment 13: 8_L.png
8_L.png
Attachment 14: 180919_1324_8usStats.png
180919_1324_8usStats.png
Attachment 15: 180919_1346_4usStats.png
180919_1346_4usStats.png
Attachment 16: 180919_1404_8us_1_8_L.png
180919_1404_8us_1_8_L.png
Attachment 17: 180919_1416_4us_1_8_L.png
180919_1416_4us_1_8_L.png
Attachment 18: 180919_1419_2usStats.png
180919_1419_2usStats.png
Attachment 19: 180919_1426_2us_1_8_L.png
180919_1426_2us_1_8_L.png
Attachment 20: 180919_1429_6usStats.png
180919_1429_6usStats.png
Attachment 21: 180919_1438_6us_1_8_L.png
180919_1438_6us_1_8_L.png
  214   Sat Apr 10 12:15:54 2021 OH, TDSaturday 10 April
12:37 DAQ found crashed. Had crashed at some time following 7:50am following previous statistics update
      Unable to recover the FEEs so forced to power cycle
      Following power cycle statistics much improved from yesterday.
      In the waveforms the lower frequency component that was prevalent yesterday is no longer visible.
      The waveforms are not perfect by any means though
      Also note that following an ASIC synchronisation all of the FEEs lose ADC calibration and must be manually recalibrated.

16.42 UTC+1

      per FEE64 1.8.W spectra

      20us FSR attachments 7 & 8
      200us FSR attachments 9 & 10
      2ms FSR attachments 11 & 12
      20ms FSR attachments 15 & 16


      rate spectra attachment 13
      good events statistics attachment 14
Attachment 1: 210410_1237_Aida_crash.png
210410_1237_Aida_crash.png
Attachment 2: 210410_1242_Crash.png
210410_1242_Crash.png
Attachment 3: 210410_1251_aida06_drop.png
210410_1251_aida06_drop.png
Attachment 4: 210410_1312_Waveform1.png
210410_1312_Waveform1.png
Attachment 5: 210304_1313_Waveform2.png
210304_1313_Waveform2.png
Attachment 6: 210410_1318_Stats.png
210410_1318_Stats.png
Attachment 7: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 10: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(4).png
Attachment 11: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(2).png
Attachment 12: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(6).png
Attachment 14: Screenshot_2021-04-10_Statistics_aidas-gsi.png
Screenshot_2021-04-10_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(7).png
Attachment 16: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(8).png
  291   Tue May 4 13:53:22 2021 OH, TDTuesday 4 May
NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290

14.40 Rate spectra - attachment 1
      adc data items - attachment 2
       slow comparator 0xa
       stats for p+n much improved, n+n remain poor
      1.8.W spectra - attachments 3 & 4
      1.8.L spectra - attachments 5 & 6
       pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM

17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7
Attachment 1: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-05-04_Statistics_aidas-gsi.png
Screenshot_2021-05-04_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png
Attachment 7: 210504_Leakage.png
210504_Leakage.png
  341   Fri May 21 07:03:50 2021 OH, TDFriday 21 May 08:00-16:00
08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png
210521_0829_Stats.png
Attachment 2: 210521_0829_Temp.png
210521_0829_Temp.png
Attachment 3: 210521_0830_Bias.png
210521_0830_Bias.png
Attachment 4: 210521_0957.png
210521_0957.png
Attachment 5: 210521_0957_Temp.png
210521_0957_Temp.png
Attachment 6: 210521_0958_Bias.png
210521_0958_Bias.png
Attachment 7: 210521_1232_Stats.png
210521_1232_Stats.png
Attachment 8: 210521_1233_Bias.png
210521_1233_Bias.png
Attachment 9: 210521_1233_Temp.png
210521_1233_Temp.png
  353   Wed Jun 2 14:53:05 2021 OH, TDMerger/Tapeserver issues after reboot of aida-3
On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen.
We were also unable to connected to MIDAS via ssh port forwarding.

Since the restart we have been unable to write to file.

The tapeserver will allocate/mount/open a file but nothing will be written to it.
The merger will be receiving data but never shows as having any "Current links with data".

/MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23  2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119

The current disk structure is:
[npg@aidas-gsi npg]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
sdb      8:16   0   7.3T  0 disk 
&#9500;&#9472;sdb1   8:17   0   200M  0 part 
&#9500;&#9472;sdb2   8:18   0   500M  0 part 
&#9492;&#9472;sdb3   8:19   0   7.3T  0 part 
  &#9500;&#9472;vg_aidas2-lv_root (dm-2)
       253:2    0    50G  0 lvm  
  &#9500;&#9472;vg_aidas2-lv_home (dm-3)
       253:3    0   7.2T  0 lvm  /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
  &#9492;&#9472;vg_aidas2-lv_swap (dm-4)
       253:4    0   7.8G  0 lvm  
sdc      8:32   0   7.3T  0 disk 
&#9492;&#9472;sdc1   8:33   0   7.3T  0 part /media/ThirdDrive
sdd      8:48   0   7.3T  0 disk 
&#9492;&#9472;sdd1   8:49   0   7.3T  0 part /media/SecondDrive
sda      8:0    0 465.8G  0 disk 
&#9500;&#9472;sda1   8:1    0   200M  0 part /boot/efi
&#9500;&#9472;sda2   8:2    0   500M  0 part /boot
&#9492;&#9472;sda3   8:3    0 465.1G  0 part 
  &#9500;&#9472;vg_aidasgsi-lv_root (dm-0)
       253:0    0    50G  0 lvm  /
  &#9500;&#9472;vg_aidasgsi-lv_swap (dm-1)
       253:1    0   7.8G  0 lvm  [SWAP]
  &#9492;&#9472;vg_aidasgsi-lv_home (dm-5)
       253:5    0 407.3G  0 lvm  /home


Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3

TapeServer config file - attachment 4
Merger options - attachment 5

Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png
Attachment 1: 210602_MergerTape_HTTPd_Startup
tidy up
tclsh8.5_copy2: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl25624
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "z5y30nlwy2e4"
httpd started on port 8115

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
/TapeServer
loading tcl/NewMergerControl.tcl for namespace ::
/DataAcquisitionControlServer
DefineMessage unknown
Run Control Server Implementation for MERGE
RunControlServer loaded
loading Html/RunControl/implementation.tcl
/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
RunControl loaded
loading Html/NewMerger/RunControl/implementation.tcl for namespace ::
ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
NewMerge Control loaded
Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl
Shared memory area located at 0x7f34d3028000
Tape Server comms table located at 0x7f34d3028000
MERGE command  30
setup MERGER
Stop Merge
Halt (1)
Action has completed: MergeCommand 2 Halt
Setup Merge
Setup (1)
Action has completed: MergeCommand 1 Setup
MERGE command  32
go MERGER
Go Merge
Go (1)
Action has completed: MergeCommand 3 Go
link 0 GO (1)
Action has completed: MergeCommand 22 {link 0 GO} 0 2
link 1 GO (1)
Action has completed: MergeCommand 22 {link 1 GO} 1 2
link 2 GO (1)
Action has completed: MergeCommand 22 {link 2 GO} 2 2
link 3 GO (1)
Action has completed: MergeCommand 22 {link 3 GO} 3 2
link 4 GO (1)
Action has completed: MergeCommand 22 {link 4 GO} 4 2
link 5 GO (1)
Action has completed: MergeCommand 22 {link 5 GO} 5 2
link 6 GO (1)
Action has completed: MergeCommand 22 {link 6 GO} 6 2
link 7 GO (1)
Action has completed: MergeCommand 22 {link 7 GO} 7 2
link 8 GO (1)
Action has completed: MergeCommand 22 {link 8 GO} 8 2
link 9 GO (1)
Action has completed: MergeCommand 22 {link 9 GO} 9 2
link 10 GO (1)
Action has completed: MergeCommand 22 {link 10 GO} 10 2
link 11 GO (1)
Action has completed: MergeCommand 22 {link 11 GO} 11 2
link 12 GO (1)
Action has completed: MergeCommand 22 {link 12 GO} 12 2
link 13 GO (1)
Action has completed: MergeCommand 22 {link 13 GO} 13 2
link 14 GO (1)
Action has completed: MergeCommand 22 {link 14 GO} 14 2
link 15 GO (1)
Action has completed: MergeCommand 22 {link 15 GO} 15 2
Resume MERGER

Attachment 2: 210602_NewMerger_Startup
Tidy up
MERGE: no process killed
merge64.AD: no process killed
Starting New Merger

[1] 25659
MERGE Master (25659): Message logger not contacted.
MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
Creating NetVars
Creating NetVars complete
MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
MERGE Master (25659): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0
MERGE Master (25659): File mapped object /SHM_11000 of size 536628 created
MERGE Master (25659): Memory mapping 536628 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0a99b8000.
MERGE Master (25659): File mapped object /SHM_11001 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0a39b6000.
MERGE Master (25659): File mapped object /SHM_11002 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc09d9b4000.
MERGE Master (25659): File mapped object /SHM_11003 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0979b2000.
MERGE Master (25659): File mapped object /SHM_11004 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0919b0000.
MERGE Master (25659): File mapped object /SHM_11005 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc08b9ae000.
MERGE Master (25659): File mapped object /SHM_11006 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0859ac000.
MERGE Master (25659): File mapped object /SHM_11007 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc07f9aa000.
MERGE Master (25659): File mapped object /SHM_11008 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0799a8000.
MERGE Master (25659): File mapped object /SHM_11009 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0739a6000.
MERGE Master (25659): File mapped object /SHM_11010 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc06d9a4000.
MERGE Master (25659): File mapped object /SHM_11011 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0679a2000.
MERGE Master (25659): File mapped object /SHM_11012 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0619a0000.
MERGE Master (25659): File mapped object /SHM_11013 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc05b99e000.
MERGE Master (25659): File mapped object /SHM_11014 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc05599c000.
MERGE Master (25659): File mapped object /SHM_11015 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc04f99a000.
MERGE Master (25659): File mapped object /SHM_11016 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc049998000.
MERGE Master (25659): Shared memory segment located at address 0x7fc049998000.
MERGE Master (25659): Starting stats/rates task
MERGE Master (25659): Stats/Rates task has pid 25661
MERGE Master (25659): Starting 16 link tasks
MERGE Master (25659): Link task has pid 25662
MERGE Master (25659): Link task has pid 25663
MERGE Statistics/Rates (25661): Message logger not contacted.
MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
Creating NetVars
Creating NetVars complete
MERGE Master (25659): Link task has pid 25664
MERGE Data Link (25662): Message logger not contacted.
MERGE Data Link (25662): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25662): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0
MERGE Data Link (25662): File mapped object /SHM_11000 accessed
MERGE Data Link (25662): Memory mapping 536628 bytes
MERGE Data Link (25662): Shared memory segment located at address 0x7f198ee7a000.
MERGE Data Link (25662): Allocated 71720 bytes for local input buffer
MERGE Data Link (25662): File mapped object /SHM_11001 accessed
MERGE Data Link (25662): Memory mapping 100669480 bytes
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Master (25659): Link task has pid 25665
MERGE Data Link (25663): Message logger not contacted.
MERGE Data Link (25663): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25663): Configuration: index = 1, SHM key=11000, TCP port base = 11002, module = 0, hardware = 0
MERGE Data Link (25663): File mapped object /SHM_11000 accessed
MERGE Data Link (25663): Memory mapping 536628 bytes
MERGE Data Link (25663): Shared memory segment located at address 0x7f3bee603000.
MERGE Data Link (25663): Allocated 71720 bytes for local input buffer
MERGE Data Link (25663): File mapped object /SHM_11002 accessed
MERGE Data Link (25663): Memory mapping 100669480 bytes
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Master (25659): Link task has pid 25666
MERGE Data Link (25664): Message logger not contacted.
MERGE Data Link (25665): Message logger not contacted.
MERGE Data Link (25664): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (25665): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
Creating NetVars complete
MERGE Data Link (25665): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0
MERGE Data Link (25665): File mapped object /SHM_11000 accessed
MERGE Data Link (25665): Memory mapping 536628 bytes
MERGE Data Link (25665): Shared memory segment located at address 0x7f7418ae6000.
MERGE Data Link (25665): Allocated 71720 bytes for local input buffer
MERGE Data Link (25665): File mapped object /SHM_11004 accessed
MERGE Data Link (25665): Memory mapping 100669480 bytes
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
Creating NetVars complete
MERGE Data Link (25664): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0
MERGE Data Link (25664): File mapped object /SHM_11000 accessed
MERGE Data Link (25664): Memory mapping 536628 bytes
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da72c8000.
MERGE Data Link (25664): Allocated 71720 bytes for local input buffer
MERGE Data Link (25664): File mapped object /SHM_11003 accessed
MERGE Data Link (25664): Memory mapping 100669480 bytes
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Master (25659): Link task has pid 25667
MERGE Master (25659): Link task has pid 25668
MERGE Data Link (25666): Message logger not contacted.
MERGE Data Link (25666): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25666): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0
MERGE Data Link (25666): File mapped object /SHM_11000 accessed
MERGE Data Link (25666): Memory mapping 536628 bytes
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7f4ec2000.
MERGE Data Link (25666): Allocated 71720 bytes for local input buffer
MERGE Data Link (25666): File mapped object /SHM_11005 accessed
MERGE Data Link (25666): Memory mapping 100669480 bytes
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Master (25659): Link task has pid 25669
MERGE Data Link (25667): Message logger not contacted.
MERGE Data Link (25667): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25667): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0
MERGE Data Link (25667): File mapped object /SHM_11000 accessed
MERGE Data Link (25667): Memory mapping 536628 bytes
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b878c9000.
MERGE Data Link (25667): Allocated 71720 bytes for local input buffer
MERGE Data Link (25667): File mapped object /SHM_11006 accessed
MERGE Data Link (25667): Memory mapping 100669480 bytes
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Master (25659): Link task has pid 25670
MERGE Data Link (25668): Message logger not contacted.
MERGE Data Link (25668): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25671
Creating NetVars complete
MERGE Data Link (25668): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0
MERGE Data Link (25668): File mapped object /SHM_11000 accessed
MERGE Data Link (25668): Memory mapping 536628 bytes
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c7c9a000.
MERGE Data Link (25668): Allocated 71720 bytes for local input buffer
MERGE Data Link (25668): File mapped object /SHM_11007 accessed
MERGE Data Link (25668): Memory mapping 100669480 bytes
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25669): Message logger not contacted.
MERGE Data Link (25669): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25672
Creating NetVars complete
MERGE Data Link (25669): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0
MERGE Data Link (25669): File mapped object /SHM_11000 accessed
MERGE Data Link (25669): Memory mapping 536628 bytes
MERGE Data Link (25669): Shared memory segment located at address 0x7f87e7704000.
MERGE Data Link (25669): Allocated 71720 bytes for local input buffer
MERGE Data Link (25669): File mapped object /SHM_11008 accessed
MERGE Data Link (25669): Memory mapping 100669480 bytes
MERGE Data Link (25669): Shared memory segment located at address 0x7f87e1702000.
MERGE Data Link (25671): Message logger not contacted.
MERGE Data Link (25671): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25671): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0
MERGE Data Link (25671): File mapped object /SHM_11000 accessed
MERGE Data Link (25671): Memory mapping 536628 bytes
MERGE Data Link (25671): Shared memory segment located at address 0x7f38b4f59000.
MERGE Data Link (25671): Allocated 71720 bytes for local input buffer
MERGE Data Link (25671): File mapped object /SHM_11010 accessed
MERGE Data Link (25671): Memory mapping 100669480 bytes
MERGE Data Link (25671): Shared memory segment located at address 0x7f38aef57000.
MERGE Data Link (25670): Message logger not contacted.
MERGE Data Link (25670): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25673
Creating NetVars complete
MERGE Data Link (25670): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0
MERGE Data Link (25670): File mapped object /SHM_11000 accessed
MERGE Data Link (25670): Memory mapping 536628 bytes
MERGE Data Link (25672): Message logger not contacted.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff7744ce000.
MERGE Data Link (25672): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (25670): Allocated 71720 bytes for local input buffer
MERGE Data Link (25670): File mapped object /SHM_11009 accessed
MERGE Data Link (25670): Memory mapping 100669480 bytes
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
Creating NetVars complete
MERGE Data Link (25672): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0
MERGE Data Link (25672): File mapped object /SHM_11000 accessed
MERGE Data Link (25672): Memory mapping 536628 bytes
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bb287e000.
MERGE Data Link (25672): Allocated 71720 bytes for local input buffer
MERGE Data Link (25672): File mapped object /SHM_11011 accessed
MERGE Data Link (25672): Memory mapping 100669480 bytes
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000.
MERGE Master (25659): Link task has pid 25674
MERGE Data Link (25673): Message logger not contacted.
MERGE Data Link (25673): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25673): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0
MERGE Data Link (25673): File mapped object /SHM_11000 accessed
MERGE Data Link (25673): Memory mapping 536628 bytes
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3cd54a000.
MERGE Data Link (25673): Allocated 71720 bytes for local input buffer
MERGE Data Link (25673): File mapped object /SHM_11012 accessed
MERGE Data Link (25673): Memory mapping 100669480 bytes
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Master (25659): Link task has pid 25675
MERGE Data Link (25674): Message logger not contacted.
MERGE Data Link (25674): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25674): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0
MERGE Data Link (25674): File mapped object /SHM_11000 accessed
MERGE Data Link (25674): Memory mapping 536628 bytes
MERGE Data Link (25674): Shared memory segment located at address 0x7faec5dbd000.
MERGE Data Link (25674): Allocated 71720 bytes for local input buffer
MERGE Data Link (25674): File mapped object /SHM_11013 accessed
MERGE Data Link (25674): Memory mapping 100669480 bytes
MERGE Data Link (25674): Shared memory segment located at address 0x7faebfdbb000.
MERGE Master (25659): Link task has pid 25676
MERGE Master (25659): Link task has pid 25677
MERGE Data Link (25675): Message logger not contacted.
MERGE Data Link (25675): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25675): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0
MERGE Data Link (25675): File mapped object /SHM_11000 accessed
MERGE Data Link (25675): Memory mapping 536628 bytes
MERGE Data Link (25675): Shared memory segment located at address 0x7fef43120000.
MERGE Data Link (25675): Allocated 71720 bytes for local input buffer
MERGE Data Link (25675): File mapped object /SHM_11014 accessed
MERGE Data Link (25675): Memory mapping 100669480 bytes
MERGE Data Link (25675): Shared memory segment located at address 0x7fef3d11e000.
MERGE Data Link (25676): Message logger not contacted.
MERGE Data Link (25676): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25676): Configuration: index = 14, SHM key=11000, TCP port base = 11015, module = 13, hardware = 0
MERGE Data Link (25676): File mapped object /SHM_11000 accessed
MERGE Data Link (25676): Memory mapping 536628 bytes
MERGE Data Link (25676): Shared memory segment located at address 0x7f6aea271000.
MERGE Data Link (25676): Allocated 71720 bytes for local input buffer
MERGE Data Link (25676): File mapped object /SHM_11015 accessed
MERGE Data Link (25676): Memory mapping 100669480 bytes
MERGE Data Link (25676): Shared memory segment located at address 0x7f6ae426f000.
MERGE Data Link (25677): Message logger not contacted.
MERGE Data Link (25677): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25677): Configuration: index = 15, SHM key=11000, TCP port base = 11016, module = 14, hardware = 0
MERGE Data Link (25677): File mapped object /SHM_11000 accessed
MERGE Data Link (25677): Memory mapping 536628 bytes
MERGE Data Link (25677): Shared memory segment located at address 0x7fc4281e9000.
MERGE Data Link (25677): Allocated 71720 bytes for local input buffer
MERGE Data Link (25677): File mapped object /SHM_11016 accessed
MERGE Data Link (25677): Memory mapping 100669480 bytes
MERGE Data Link (25677): Shared memory segment located at address 0x7fc4221e7000.
MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
MERGE Statistics thread starting
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000.
... 201 more lines ...
Attachment 3: 210602_TapeServerStartup
Tidy up
master(4596): Operation not permitted
Starting Tape Server

[1] 25639
MIDAS Tape Server: Message logger not contacted.
MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021  12:22:14
MIDAS Tape Server: Unable to change scheduling priority - Permission denied
MIDAS Tape Server: Using default startup
MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205.
MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19721000.
MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration
MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats
MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305                 
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002                
MIDAS Tape Server: Message reporting level = 0x180fff8
MIDAS Tape Server: Message logging level = 0xfff8
MIDAS Tape Server: Tape Server Options = 0x0
MIDAS Tape Server: File device path base = /TapeData
MIDAS Tape Server: Data buffer size = 65536
MIDAS Tape Server: Tape block size = 65536
MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19320000.
MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19881000.
MIDAS Tape Server: Capabilities restored.
MIDAS Tape Server: Master global area initialised.
MIDAS Tape Server: Stats task has pid 25641
MIDAS Tape Server: Driver process for /dev/file/0 has pid 25642
MIDAS Tape Server: Driver process for /dev/file/1 has pid 25643
MIDAS Tape Server: Driver process for /dev/null/0 has pid 25644
MIDAS Tape Server: Link task 0 has pid 25645
MIDAS Tape Server: Link task 1 has pid 25646
MIDAS Tape Server: Starting the RPC interface
MIDAS Tape Statistics: Message logger not contacted.
MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021  12:22:20
MIDAS Tape Statistics: Started with args 10205
MIDAS Tape Statistics: Configuration: SHM key=10205
MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Statistics: Shared memory ID is 3
MIDAS Tape Statistics: Shared memory segment located at address 7f9c551f7000.
MIDAS Data Link (25645): Message logger not contacted.
MIDAS Data Link (25645): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (25645): Started with args 10205 10305                 
MIDAS Data Link (25645): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1
MIDAS Tape Driver (25644): Message logger not contacted.
MIDAS Tape Driver (25644): Message logger not contacted.
MIDAS Data Link (25646): Message logger not contacted.
MIDAS Tape Driver (25644): Started with args 2 10205
MIDAS Data Link (25645): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25645): Shared memory ID is 3
MIDAS Data Link (25645): Shared memory segment located at address 7f6ba30ef000.
MIDAS Data Link (25646): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Tape Driver (25644): Configuration: driver=2, key=10205.
MIDAS Data Link (25646): Started with args 10205 21001 21002                
MIDAS Data Link (25645): Starting the network interface
MIDAS Tape Driver (25644): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25646): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2
MIDAS Tape Driver (25644): Shared memory ID is 3
MIDAS Tape Driver (25644): Shared memory segment located at address 7f09d718d000.
MIDAS Tape Driver (25644): Using device /dev/null/0 of type sink.
MIDAS Data Link (25646): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25646): Shared memory ID is 3
MIDAS Data Link (25646): Shared memory segment located at address 7fa75ae3c000.
MIDAS Data Link (25646): Starting the network interface
MIDAS Tape Driver (25642): Message logger not contacted.
MIDAS Data Link (25645): TCP socket receive buffer was 87380 - now 249856
MIDAS Tape Driver (25642): Message logger not contacted.
MIDAS Tape Driver (25642): Started with args 0 10205
MIDAS Tape Driver (25642): Configuration: driver=0, key=10205.
MIDAS Tape Driver (25643): Message logger not contacted.
MIDAS Tape Driver (25642): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (25643): Message logger not contacted.
MIDAS Tape Driver (25642): Shared memory ID is 3
MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (25645): TCP socket send buffer was 16384 - now 249856
MIDAS Tape Driver (25643): Started with args 1 10205
MIDAS Tape Driver (25642): Shared memory segment located at address 7f48b1ea4000.
MIDAS Tape Driver (25642): Using device /dev/file/0 of type file.
MIDAS Data Link (25645): MIDAS Data Link thread 0 using TCP port 10305.
MIDAS Data Link (25645): Entering server loop
MIDAS Tape Driver (25643): Configuration: driver=1, key=10205.
MIDAS Data Link (25645): thread 0 listening on port 10305
MIDAS Tape Driver (25643): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (25643): Shared memory ID is 3
MIDAS Tape Driver (25643): Shared memory segment located at address 7f6044052000.
MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856
MIDAS Tape Driver (25643): Using device /dev/file/1 of type file.
MIDAS Data Link (25646): MIDAS Data Link thread 0 using TCP port 21001.
MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (25646): MIDAS Data Link thread 1 using TCP port 21002.
MIDAS Data Link (25646): Entering server loop
MIDAS Data Link (25646): thread 0 listening on port 21001
MIDAS Data Link (25646): thread 1 listening on port 21002
MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205.
MIDAS Tape Server: Entering server loop
MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205.
MIDAS Data Link (25645): thread 0 accepted connection from 127.0.0.1, port 37330
MIDAS Data Link (25645): buffer size changed to 65536
MIDAS Tape Server: device /dev/file/0 allocated.
MIDAS Tape Server: Mounting volume S496 on device /dev/file/0.
MIDAS Tape Server: Opening file R31 on device /dev/file/0.
2222 64 0 31056336
3333 31056336 2048000 1 15 

Attachment 4: TS_configuration
#device configuration information
#    any line starting with a # is a comment and is ignored
stats /MIDAS/TapeServer/Linux64/stats
#    devices available   -   list ends with a null line
#    format       class_name     device_name    driver_task_path_name
#       class_name MUST be one of dlt, exabyte, scsitape, sink
file          /dev/file/0    /MIDAS/TapeServer/Linux64/driver
file          /dev/file/1    /MIDAS/TapeServer/Linux64/driver
sink          /dev/null/0     /MIDAS/TapeServer/Linux64/driver

#data link configuration information  - list end with a null line
#    format   link_task_path_name  
/MIDAS/TapeServer/Linux64/linkTCP 10305
/MIDAS/TapeServer/Linux64/linkTCP 21001 21002

#program options - list end with a null line
#tapeserver_options 0x2
tapeserver_options 0x0
msg_reporting_level 0x0180fff8
#   use 0x0080fff8 to enable msg logging
msg_logging_level 0xfff8
#    default for following is 16Kbytes - both MUST be the same at present
data_buffer_size 64
tape_block_size 64
disc_file_size 2000
#
file_path_base  /TapeData

#end of information


Attachment 5: CONTENTS
Index string MERGE.LinksAvailable&&RunNumber&&MERGE.LinksInUse&&MERGE.RunOptions
Stat.shift string 0
0x0006dead string 0x0000
MERGE.LinksAvailable string 16
Stat.offset string 0
RunNumber string 0
Rate.channels string 1024
Stat.channels string 1024
MERGE.RunOptions string 3
MERGE.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Merge.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%
Attachment 6: 210602_merger.png
210602_merger.png
  442   Fri May 13 07:36:18 2022 OH, TDFriday 13 May
08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)
Attachment 1: 220513_0839_R1_327_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259232212 ( 1719602.4 Hz)
        Other data format:    2687788 (   17829.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        986 (       6.5 Hz)
                               RESUME:        986 (       6.5 Hz)
                              SYNC100:      32562 (     216.0 Hz)
                              WR48-63:      32562 (     216.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2620692 (   17384.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     130683 (     866.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      150.751 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1                9.978                0.000
  2                0.059                0.000
  3               21.640                0.000
  4                0.158                0.000
  5               66.897                0.000
  6                1.778                0.000
  7               36.972                0.000
  8                0.172                0.000
  9                0.018                0.000
 10                0.000                7.315
 11               28.135                0.000
 12                0.007                0.000
 13                0.000                3.034
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7933827       2887          0          0          5          5        987        987          0        903          0       1141
  1   37526918       9508          0          0         50         50       4704       4704          0          0          0     122502
  2    8368319    1400718          0          0          2          2       1222       1222          0    1398270          0        647
  3   39185673    1207805          0          0        156        156       5116       5116          0    1197261          0        703
  4    4609566       1178          0          0          5          5        584        584          0          0          0       1170
  5   40049357      10540          0          0        309        309       4961       4961          0          0          0        297
  6   13162553      17502          0          0         13         13       1685       1685          0      14106          0        302
  7   41199857      20850          0          0        248        248       5101       5101          0      10152          0        241
  8   12021437       2864          0          0          5          5       1427       1427          0          0          0        467
  9    4748197       1188          0          0          2          2        592        592          0          0          0       1165
 10    3089949        744          0          0          0          0        372        372          0          0          0        247
 11   38258323       9856          0          0        190        190       4738       4738          0          0          0       1052
 12    6307578       1518          0          0          1          1        758        758          0          0          0        311
 13    2770658        630          0          0          0          0        315        315          0          0          0        438
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.600s (  952.392 blocks/s,  59.525 Mb/s)
Attachment 2: Screenshot_from_2022-05-13_14-28-00.png
Screenshot_from_2022-05-13_14-28-00.png
Attachment 3: Screenshot_from_2022-05-13_14-27-51.png
Screenshot_from_2022-05-13_14-27-51.png
Attachment 4: Screenshot_from_2022-05-13_14-27-31.png
Screenshot_from_2022-05-13_14-27-31.png
Attachment 5: Screenshot_from_2022-05-13_14-27-17.png
Screenshot_from_2022-05-13_14-27-17.png
Attachment 6: Screenshot_from_2022-05-13_14-26-32.png
Screenshot_from_2022-05-13_14-26-32.png
Attachment 7: Screenshot_from_2022-05-13_14-26-26.png
Screenshot_from_2022-05-13_14-26-26.png
Attachment 8: Screenshot_from_2022-05-13_14-26-21.png
Screenshot_from_2022-05-13_14-26-21.png
Attachment 9: Screenshot_from_2022-05-13_14-26-08.png
Screenshot_from_2022-05-13_14-26-08.png
Attachment 10: Screenshot_from_2022-05-13_14-25-46.png
Screenshot_from_2022-05-13_14-25-46.png
Attachment 11: Screenshot_from_2022-05-13_14-25-38.png
Screenshot_from_2022-05-13_14-25-38.png
Attachment 12: Screenshot_from_2022-05-13_14-25-22.png
Screenshot_from_2022-05-13_14-25-22.png
Attachment 13: Screenshot_from_2022-05-13_14-24-07.png
Screenshot_from_2022-05-13_14-24-07.png
Attachment 14: Screenshot_from_2022-05-13_14-22-44.png
Screenshot_from_2022-05-13_14-22-44.png
Attachment 15: Screenshot_from_2022-05-13_14-21-16.png
Screenshot_from_2022-05-13_14-21-16.png
Attachment 16: Screenshot_from_2022-05-13_14-21-04.png
Screenshot_from_2022-05-13_14-21-04.png
Attachment 17: Screenshot_from_2022-05-13_14-20-34.png
Screenshot_from_2022-05-13_14-20-34.png
Attachment 18: Screenshot_from_2022-05-13_14-20-14.png
Screenshot_from_2022-05-13_14-20-14.png
Attachment 19: Screenshot_from_2022-05-13_14-19-10.png
Screenshot_from_2022-05-13_14-19-10.png
Attachment 20: Screenshot_from_2022-05-13_14-18-40.png
Screenshot_from_2022-05-13_14-18-40.png
Attachment 21: Screenshot_from_2022-05-13_14-18-31.png
Screenshot_from_2022-05-13_14-18-31.png
Attachment 22: Screenshot_from_2022-05-13_14-18-14.png
Screenshot_from_2022-05-13_14-18-14.png
Attachment 23: Screenshot_from_2022-05-13_14-17-38.png
Screenshot_from_2022-05-13_14-17-38.png
Attachment 24: Screenshot_from_2022-05-13_14-17-23.png
Screenshot_from_2022-05-13_14-17-23.png
Attachment 25: Screenshot_from_2022-05-13_14-17-09.png
Screenshot_from_2022-05-13_14-17-09.png
Attachment 26: Screenshot_from_2022-05-13_14-16-56.png
Screenshot_from_2022-05-13_14-16-56.png
Attachment 27: Screenshot_from_2022-05-13_14-16-30.png
Screenshot_from_2022-05-13_14-16-30.png
Attachment 28: Screenshot_from_2022-05-13_14-16-11.png
Screenshot_from_2022-05-13_14-16-11.png
Attachment 29: R4_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259102731 ( 1572653.5 Hz)
        Other data format:    2817271 (   17099.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2412 (      14.6 Hz)
                               RESUME:       2414 (      14.7 Hz)
                              SYNC100:      32614 (     198.0 Hz)
                              WR48-63:      32614 (     198.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2747217 (   16674.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      30594 (     185.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      164.755 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                5.297                0.000
  2                0.000                0.000
  3               18.287                0.000
  4                0.000                0.000
  5               77.894                0.000
  6                0.000                0.000
  7               25.402                0.000
  8                0.001                0.000
  9                0.000                0.000
 10                0.000                2.705
 11               15.617                0.000
 12                0.000                0.000
 13                0.000               29.550
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6844719       2775          0          0          0          0        894        894          0        987          0        375
  1   39920437      10378          0          0        277        277       4912       4912          0          0          0      27546
  2    7063505    1453891          0          0          0          0       1085       1085          0    1451721          0        328
  3   42633132    1301928          0          0        410        410       5590       5590          0    1289928          0        251
  4    3625899        900          0          0          0          0        450        450          0          0          0        125
  5   38867835      10853          0          0        768        769       4658       4658          0          0          0        129
  6    7590830       4334          0          0          0          0        934        934          0       2466          0        227
  7   44498336      14120          0          0        564        565       5438       5438          0       2115          0        113
  8   10949373       2716          0          0          1          1       1357       1357          0          0          0        345
  9    3887105        996          0          0          0          0        498        498          0          0          0        273
 10    1999345        532          0          0          0          0        266        266          0          0          0        160
 11   43113077      11754          0          0        392        392       5485       5485          0          0          0        288
 12    6030564       1596          0          0          0          0        798        798          0          0          0        252
 13    2078574        498          0          0          0          0        249        249          0          0          0        182
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
Attachment 30: Screenshot_from_2022-05-13_15-11-36.png
Screenshot_from_2022-05-13_15-11-36.png
Attachment 31: Screenshot_from_2022-05-13_15-11-29.png
Screenshot_from_2022-05-13_15-11-29.png
Attachment 32: Screenshot_from_2022-05-13_15-11-17.png
Screenshot_from_2022-05-13_15-11-17.png
Attachment 33: Screenshot_from_2022-05-13_15-10-55.png
Screenshot_from_2022-05-13_15-10-55.png
Attachment 34: Screenshot_from_2022-05-13_15-10-41.png
Screenshot_from_2022-05-13_15-10-41.png
Attachment 35: Screenshot_from_2022-05-13_15-10-30.png
Screenshot_from_2022-05-13_15-10-30.png
ELOG V3.1.4-unknown