ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC, Page 26 of 35

Full | Summary | Threaded | Hide attachments

686 Entries

Goto page Previous 1, 2, 3 ... 25, 26, 27 ... 33, 34, 35 Next

Selected entries:

	ID	Date	Author	Subject
	191	Fri Mar 12 08:32:46 2021	TD	Friday 12 March 08.00-
08.31 DAQ continues OK - file R33_263 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 08.32 No merger server error/warning messages since most recent restart c. 00.00 today 08.44 System wide checks FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Base Current Difference aida05 fault 0x7a56 : 0x7a58 : 2 aida06 fault 0x65bb : 0x65bd : 2 aida07 fault 0xcdd1 : 0xcdd4 : 3 aida08 fault 0x2ab3 : 0x2ab5 : 2 White Rabbit error counter test result: Passed 8, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA Timestamp error counter test result: Passed 12, Failed 0 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last Returned 0 0 0 0 0 0 0 0 0 0 0 0 Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k aida01 : 14 6 3 2 1 2 2 3 3 4 11 : 58904 aida02 : 6 4 1 1 1 2 2 4 2 3 7 : 39848 aida03 : 5 6 4 2 1 3 1 3 2 3 7 : 39300 aida04 : 4 1 2 2 3 3 2 4 1 4 15 : 73912 aida05 : 1 5 2 2 2 4 2 3 2 3 7 : 39692 aida06 : 7 2 2 3 1 3 3 3 2 4 7 : 41836 aida07 : 6 3 1 2 3 2 2 3 3 3 7 : 40512 aida08 : 8 5 2 2 1 3 1 3 2 3 7 : 39272 aida09 : 1 2 3 1 1 3 2 3 1 4 7 : 40484 aida10 : 2 6 3 2 1 2 2 2 3 3 7 : 39912 aida11 : 2 3 1 0 2 4 1 3 2 3 7 : 39344 aida12 : 2 3 4 2 0 3 3 3 2 3 7 : 39712 FEE64 Temperatures OK - attachment 1 Good event statistics OK - attachment 2 Detector bias & leakage currents OK - attachment 3 Merger OK - 5.1M data items/s TapeServer OK - 16Mb/s 08.54 p+n junction strip HEC spectra - attachment 1 Rate spectra - attachment 2 09.15 Burst of Merger server warning messages of type MERGE Data Link (3547): bad timestamp 1 3 0x8128d6c8 0x05056328 0x00008d6c85056328 0x166b8d6c85056328 0x166b8d6d86da01ee also multiple Warning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: A The latter may reflect downstream problems writing MBS data
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi.png

Attachment 3: 30.png

Attachment 4: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png

Attachment 5: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png

	190	Thu Mar 11 23:07:23 2021	CA	March 12th 00:00 - 08:00 shift
00:10 DAQ continues OK - file R30_372 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 00:17 System wide checks all OK FEE64 Temperatures ok - attachment 1 Good event statistics ok - attachment 2 detector bias and leakage currents ok - attachment 3 00:20 DESPEC on run 145 00:24 Merger ok ~45M items/s Tapeserver ok ~15MB/s 01:00 aida07 crashed recovered, but all FEE64 crash shortly after called OH - stop DAQ, TapeService, Merger 01:20 all FEE64 powercycled, AIDA restart 01:30 AIDA recovered, DAQ now running - writing to R33 system wide checks ok except FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Understand status as follows Status bit 3 : firmware PLL that creates clocks from external clock not locked Status bit 2 : always logic '1' Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock If all these bits are not set then the operation of the firmware is unreliable FEE64 module aida07 failed FEE64 module aida09 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module 01:52 FEE64 Temperatures ok - attachment 4 -had to reload a few times to work, but otherwise ok Good event statistics - attachment 5 -aida05 and aida08 running faster than before detector bias and leakage currents ok - attachment 6 02:04 writing to file R33_34 Data forwarding to MBS ok AIDA ASIC settings ok 02:11 beam off 02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails 02:32 beam back - writing to file R33_48 04:30 DESPEC having issues with Go4 crashing ucesb reports AIDA/FATIMA/bplast timewarp events System wide checks: Clock error: FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Calibration: FEE64 module aida07 failed FEE64 module aida09 failed Calibration test result: Passed 10, Failed 2 White Rabbit: Base Current Difference aida07 fault 0xcdd1 : 0xcdd2 : 1 White Rabbit error counter test result: Passed 11, Failed 1 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FEE64 Temperatures ok - attachment 7 Good event statistics ok - attachment 8 detector bias and leakage currents ok - attachment 9 04:52 analyzer output for R33_113 - attachment 10 no timewarps aida06 dead time? (ignore idle time and rates) 05:02 Merger 5M data items/s TapeServer 17 MB/s 05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now 05:25 rates spectra - attachment 11 05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12 06:29 system wide checks: Clock error: FEE64 module aida09 global clocks failed, 6 Clock status test result: Passed 11, Failed 1 Calibration: FEE64 module aida07 failed FEE64 module aida09 failed Calibration test result: Passed 10, Failed 2 White Rabbit: Base Current Difference aida05 fault 0x7a56 : 0x7a58 : 2 aida06 fault 0x65bb : 0x65bd : 2 aida07 fault 0xcdd1 : 0xcdd4 : 3 aida08 fault 0x2ab3 : 0x2ab5 : 2 White Rabbit error counter test result: Passed 8, Failed 4 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR rest ok 06:32 FEE64 Temperatures ok - attachment 13 Good event statistics ok - attachment 14 detector bias and leakage currents ok - attachment 15 06:41 Merger 5.1M data items/s TapeServer 17 MB/s Data forwarding to MBS ok 08:54 restart MBS relay, requested by NH
Attachment 1: Screenshot_from_2021-03-11_23-14-57_-_1.png

Attachment 2: Screenshot_from_2021-03-11_23-15-31.png

Attachment 3: Screenshot_from_2021-03-11_23-16-08.png

Attachment 4: Screenshot_from_2021-03-12_00-55-09.png

Attachment 5: Screenshot_from_2021-03-12_00-55-35.png

Attachment 6: Screenshot_from_2021-03-12_00-58-59.png

Attachment 7: Screenshot_from_2021-03-12_03-35-47.png

Attachment 8: Screenshot_from_2021-03-12_03-36-35.png

Attachment 9: Screenshot_from_2021-03-12_03-37-20.png

Attachment 10: R33_113
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 260049813 ( 2219319.2 Hz) Other data format: 1870189 ( 15960.6 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 20863 ( 178.0 Hz) RESUME: 20864 ( 178.1 Hz) SYNC100: 437 ( 3.7 Hz) WR48-63: 437 ( 3.7 Hz) FEE64 disc: 442392 ( 3775.5 Hz) MBS info: 1385196 ( 11821.6 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 32687 ( 279.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 117.175 s FEE elapsed dead time(s) elapsed idle time(s) 1 49.525 113.280 2 56.916 110.595 3 47.492 115.964 4 57.911 115.427 5 7.968 116.501 6 1615384952.728 108.448 7 42.814 114.622 8 80.186 113.011 9 88.924 96.905 10 21.378 111.132 11 73.040 114.354 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 22118772 49791 0 0 18934 18933 32 32 11506 354 0 4012 1 22984048 20152 0 0 166 167 43 43 19337 396 0 5880 2 20394651 783427 0 0 163 164 37 37 255035 527991 0 3363 3 22242991 614086 0 0 167 167 40 40 2080 611592 0 620 4 21439332 55343 0 0 169 170 49 49 8463 46443 0 2902 5 16079204 115814 0 0 279 279 29 29 22786 92412 0 7566 6 22040349 143536 0 0 183 183 31 31 102212 40896 0 2700 7 22240015 67179 0 0 150 150 58 58 1651 65112 0 545 8 21453279 3177 0 0 179 179 23 23 2773 0 0 850 9 22650021 4350 0 0 185 184 20 20 3941 0 0 1026 10 24195088 12236 0 0 107 107 38 38 11946 0 0 3047 11 22212063 1098 0 0 181 181 37 37 662 0 0 176 12 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:17230.426s ( 1.857 blocks/s, 0.116 Mb/s)
Attachment 11: Screenshot_from_2021-03-12_04-23-51.png

Attachment 12: Screenshot_from_2021-03-12_04-29-06.png

Attachment 13: Screenshot_from_2021-03-12_05-34-02.png

Attachment 14: Screenshot_from_2021-03-12_05-34-28_-_1.png

Attachment 15: Screenshot_from_2021-03-12_05-34-51.png

	189	Thu Mar 11 07:09:25 2021	OH, LS	Thursday 11th of March
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration N.B. wrong date given to the screenshots by accident (It's early) Statistics - attachment 1 FEE temperatures (AIDA10 unavailable) - attachment 2 FEE temperature - attachment 3 Bias and leakage currents ok - attachment 4 08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it. Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself. Restarted but no stats in aida04 It did not recover from a reset Telnet in and did a reboot command 09:10 Finally recovered issue was it was not linking to merger There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms Manually set the ASIC settings for aida12. Rates look as expected after Will try to reset once beam is back 10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version Analysis of R30_21 looks fine 10:38 System wide checks ok Statistics - attachment 5 Temperature - attachment 6 Bias and leakage - attachment 7 12.10 (LS) System wide checks all okay, no fails Statistics (attachment8) Rate spectra (attachment9) FEE temps (attament10) Leakage currents written to sheet (attachment11) Merger ~45M items/s Tapeserver ~14MB/s 13.45 no beam (around file R30_98) 14.00 System wide checks all okay, no fails Statistics (attachment12) FEE temps (attament13) Leakage currents written to sheet (attachment14) Merger ~44M items/s Tapeserver ~14MB/s Beam still down 14.11 beam back aida file R30_112 14.13 beam back down, back at 14.16 14.20 no beam 14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal, double by analysing R30_101,102,103 no timewarps beam is back but low intensity at the moment 14.46 beam looks to be back at a higher intensity now R30_127 (attachment15) 16.00 System wide checks all okay, no fails Statistics (attachment16) Rate spectra (attachment17) FEE temps (attament18) Leakage currents written to sheet (attachment19) Merger ~45M items/s Tapeserver ~14MB/s 18.00 System wide checks all okay, no fails Statistics (attachment20) Rate spectra (attachment21) FEE temps (attament22) Leakage currents written to sheet (attachment23) Merger ~45M items/s Tapeserver ~15MB/s bunch of timewarp errors in ucesb, no errors in MBS relay terminal though now see error in MBS relay! "Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored. It triggered 63 times. I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser 19:10 MBS DAQ stopped to allow FRS tests 19:38 MBS DAQ is back 20:15 Aida07 fails FPGA check Base Current Difference aida07 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last All other system wide checks ok Statistics - attachment 24 (Again ignore the year in the filename) Temperatures - attachment 25 Bias and leakage currents - 26 21:36 They are resetting the time sorter due to FATIMA issues 22:38 System wide checks error on WR in addition to the previous FPGA error Base Current Difference aida05 fault 0x5cc9 : 0x5cca : 1 aida06 fault 0xde76 : 0xde77 : 1 aida07 fault 0xa566 : 0xa567 : 1 aida08 fault 0x6ab0 : 0x6ab1 : 1 White Rabbit error counter test result: Passed 8, Failed 4 Statistics - attachment 27 Temperature - attachment 28 Bias and leakage current - attachment 29 23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain.
Attachment 1: 210310_0805_Stats.png

Attachment 2: 210310_0806_Temp.png

Attachment 3: 210310_0806_Temp2.png

Attachment 4: 210310_0808_Bias.png

Attachment 5: 210311_1039_Stats.png

Attachment 6: 210311_1039_Temp.png

Attachment 7: 210311_1041_Bias.png

Attachment 8: 1200_Statistics.PNG

Attachment 9: 1200_SpectraRate.PNG

Attachment 10: 1200_FEETemps.PNG

Attachment 11: 1200_LeakageCurrents.PNG

Attachment 12: 1400_Statistics.PNG

Attachment 13: 1400_FEETemps.PNG

Attachment 14: 1400_LeakageCurrents.PNG

Attachment 15: 1450_SpectraRate.PNG

Attachment 16: 1600_Statistics.PNG

Attachment 17: 1600_SpectraRate.PNG

Attachment 18: 1600_FEETemps.PNG

Attachment 19: 1600_LeakageCurrents.PNG

Attachment 20: 1800_Statistics.PNG

Attachment 21: 1800_SpectraRate.PNG

Attachment 22: 1800_FEETemps.PNG

Attachment 23: 1800_LeakageCurrents.PNG

Attachment 24: 200311_2014_Stats.png

Attachment 25: 200311_2014_Temp.png

Attachment 26: 210311_2017_Bias.png

Attachment 27: 210311_2225_sTATS.png

Attachment 28: 210311_2226_Temp.png

Attachment 29: 210311_2227_Bias.png

	188	Thu Mar 11 00:38:33 2021	TD	Thusrday 11 March 00.00-08.00
01.38 DAQ continues OK - file R22_388 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse System wide checks all OK except FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module FEE64 Temperatures OK - attachment 1 Good event statistics OK - attachment 2 Detector bias & leakage currents OK - attachment 3 Merger OK - 4.4M data items/s TapeServer OK - 14Mb/s 01.47 no merger server error/warning messages since last check by TD at start of shift 00.00 03.44 merger server error/warning messages since last check e.g. MERGE Data Link (30260): bad timestamp 6 3 0xc1bd7e9a 0x0ce9c9c6 0x0000292e0ce9c9c6 0x166b292e0ce9c9c6 0x166b292e0cea5486 03.46 DAQ continues OK - file R22_441 System wide checks all OK except FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module FPGA timestamp errors Base Current Difference aida07 fault 0x1 : 0x2 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last FEE64 Temperatures OK - attachment 4 Good event statistics OK - attachment 5 Detector bias & leakage currents OK - attachment 6 Merger OK - 4.6M data items/s TapeServer OK - 15Mb/s 02.51 Rate spectra - attachment 7 04.14 p+n junction HEC spectra - attachment 8 all spectra zero'd 06.22 merger server error/warning messages since last check e.g. MERGE Data Link (30265): bad timestamp 11 3 0xc2fc8067 0x02df391e 0x0000324e62df391e 0x166b324e62df391e 0x166b324e643c775e 06.23 DAQ continues OK - file R22_508 System wide checks all OK except FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module White rabbit decoder status Base Current Difference aida01 fault 0xe67 : 0xe68 : 1 aida02 fault 0x6b22 : 0x6b23 : 1 aida03 fault 0x4c76 : 0x4c77 : 1 aida04 fault 0x7542 : 0x7543 : 1 aida05 fault 0x1597 : 0x1599 : 2 aida06 fault 0xe241 : 0xe243 : 2 aida07 fault 0x749 : 0x74c : 3 aida08 fault 0xf854 : 0xf856 : 2 White Rabbit error counter test result: Passed 4, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR FPGA timestamp errors Base Current Difference aida07 fault 0x1 : 0x2 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last FEE64 Temperatures OK - attachment 9 Good event statistics OK - attachment 10 Detector bias & leakage currents OK - attachment 11 Merger OK - 4.8M data items/s TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-03-11_Statistics_aidas-gsi.png

Attachment 3: 20.png

Attachment 4: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 5: Screenshot_2021-03-11_Statistics_aidas-gsi(1).png

Attachment 6: 21.png

Attachment 7: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi.png

Attachment 8: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi(1).png

Attachment 9: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 10: Screenshot_2021-03-11_Statistics_aidas-gsi(2).png

Attachment 11: 22.png

	187	Wed Mar 10 07:16:02 2021	OH, LS	Wednesday 10th March 08:00-24:00
08:16 System wide checks all ok Statistics - attachment 1 Temperatures - attachment 2 bias - attachment 3 Merger running 4.5e6 events per second Tape data writing at 42MB per second Currently no beam 09:45 While there is no beam we will perform a longer test of the new merger. Run R20 stopped and merger changed to neew version Started R21. The now usual behaviour of additional files at start of merge observed. N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was set to going -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_0 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_1 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_2 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_3 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_4 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_5 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_6 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_7 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_8 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_9 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_10 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_11 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_12 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_13 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_14 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_15 -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_16 -rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17 -rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18 TapeData rate of 13280 kB/sec No errors observed in ucesb Correlations seen between time machine in AIDA and Ge in online Pulser rate in the online makes sense With the current data rate remaining HDD space on current drive will last around 89 hours. 09.20 analysis of file R21_27 ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data 10:24 Message while performing system wide checks: Get returned with an error error: SOAP http transport timed out after 10000 ms NONE error: SOAP http transport timed out after 10000 ms while executing "$transport $procVarName $url $req" (procedure "::SOAP::invoke" line 18) invoked from within "::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10" ("eval" body line 1) invoked from within "eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args" (procedure "XAIDAAccessClient__Get" line 1) invoked from within "XAIDAAccessClient__Get $Addr" aida02 restarted itself during the reset process Looking at the log messages on aida02 cannot see any reason for the cause of the restart. System wide checks following the restart all ok. When restarting the MBS relay errors observed until a timestamp was observed: Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000 10:55 Statistics - attachment 5 Temperature - attachment 6 Bias - attachment 7 11:03 Time machine correlation spectra with new merger AIDA - FATIMA - Attachment 8 AIDA - Ge - Attachment 9 11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the layout to determine which FEE/ASIC caused the events before checking ASIC control 12:12 System wide checks all ok except adc calibration which is same as before Statistics - attachment 10 Temperature - attachment 11 Bias and leakage currents ok - attachment 12 A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat. After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time. They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon cable from the VME scalers. Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of DSSD3. 12:50 We have waveforms for some FEES Layout 7 attachment -14 Layout 8 - attachment 15 14.00 (LS) Still no beam System wide checks okay except same as before: FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module* Statistics (attachment16) Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16): aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on FEE Temps (attachment17) Leakage currents written to sheets(attachment18) Merger ~ 45M items/s TapeServer~ 14MB/s 14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file R22_98, rate spectra attached (attachment19) 14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw no timewarps so we are not losing anything. Restarted MBS relay, errors have not reappeared so far 16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks) Statistics (attachment20) Rate spectra (attachment21) FEE Temps (attachment22) Leakage currents written to sheets(attachment23) Merger ~ 4.3M items/s TapeServer~ 14MB/s Current file R22_148 16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from S452f115 Around R22_150 18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm) Rates of R22_122 (attachment25): First DSSSD (fee0-fee3) ~66/s Second DSSSD (fee4-fee7) ~49/s Third DSSSD (fee8-fee11) ~22/s Rates of R22_178 (attachment26): First DSSSD (fee0-fee3) ~122/s Second DSSSD (fee4-fee7) ~94/s Third DSSSD (fee8-fee11) ~49/s 18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks) Statistics (attachment27) Rate spectra (attachment28) FEE Temps (attachment29) Leakage currents written to sheets(attachment30), look to be on the way down Merger ~ 4.5M items/s TapeServer~ 15MB/s 18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier 19:34 MBS DAQ Crashed 20:00 DAQ is still down. They have Sultan working on it. 20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre reboot) We are borrowing a HDD from the SHIP group 20:30 System wide checks all ok Statistics - attachment 31 Temperature - attachment 32 Bias - attachment 33 22:10 MBS DAQ has crashed Merger terminal is now showing a large number of bad merge events It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS. 22:42 System wide checks all ok Statistics - attachment 34 Temperature - Attachment 35 Bias and leakage currents ok - attachment 36 23:33 System wide checks all ok - attachment 37 Temperatures - attachment 38 Bias and leakage currents ok - attachment 39
Attachment 1: 210310_0813_Stats.png

Attachment 2: 210310_0813_Temp.png

Attachment 3: 210310_0714_Bias.png

Attachment 4: R21_27
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 259319480 ( 1687258.2 Hz) Other data format: 2600520 ( 16920.2 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 1272 ( 8.3 Hz) RESUME: 1272 ( 8.3 Hz) SYNC100: 573 ( 3.7 Hz) WR48-63: 573 ( 3.7 Hz) FEE64 disc: 302889 ( 1970.7 Hz) MBS info: 2293941 ( 14925.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 153.693 s FEE elapsed dead time(s) elapsed idle time(s) 1 0.000 150.055 2 0.129 153.277 3 1.695 152.203 4 0.000 97.174 5 15.630 152.471 6 0.533 122.407 7 0.048 144.955 8 17.221 151.934 9 26.571 143.076 10 0.063 148.982 11 11.592 141.197 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 20482535 1736 0 0 381 381 52 52 0 870 0 0 1 26566394 1030 0 0 0 0 56 56 0 918 0 0 2 26980761 1455064 0 0 2 2 87 87 302889 1151997 0 0 3 26884821 1140284 0 0 5 5 65 65 0 1140144 0 0 4 7516935 23 0 0 0 0 7 7 0 9 0 0 5 16098094 1219 0 0 577 577 31 31 0 3 0 0 6 9441595 40 0 0 2 2 18 18 0 0 0 0 7 8769482 38 0 0 2 2 17 17 0 0 0 0 8 32618578 336 0 0 98 98 70 70 0 0 0 0 9 32790396 426 0 0 136 136 77 77 0 0 0 0 10 16087963 82 0 0 1 1 40 40 0 0 0 0 11 35081926 242 0 0 68 68 53 53 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:37059.496s ( 0.863 blocks/s, 0.054 Mb/s)
Attachment 5: 210310_1054_Stats.png

Attachment 6: 210310_1055_Temp.png

Attachment 7: 210310_1055_Bias.png

Attachment 8: AIDA_Fatima_Correlation.PNG

Attachment 9: AIDA_Ge_Correlation.PNG

Attachment 10: 210310_1210_Stats.png

Attachment 11: 210310_1211_Temp.png

Attachment 12: 210310_1212_Bias.png

Attachment 13: AIDA_leakage.PNG

Attachment 14: 210310_1348_Layout7.png

Attachment 15: 210310_1252_Layout8.png

Attachment 16: 1400_Statistics.PNG

Attachment 17: 1400_FEETemps.PNG

Attachment 18: 1400_LeakageCurrents.PNG

Attachment 19: 1400_SpectraRate.PNG

Attachment 20: 1600_Statistics.PNG

Attachment 21: 1600_SpectraRate.PNG

Attachment 22: 1600_FEETemps.PNG

Attachment 23: 1600_LeakageCurrents.PNG

Attachment 24: 1640_SpectraRate.PNG

Attachment 25: 1700_R22_122_Rates.PNG

Attachment 26: 1700_R22_178_Rates.PNG

Attachment 27: 1800_Statistics.PNG

Attachment 28: 1800_SpectraRate.PNG

Attachment 29: 1800_FEETemps.PNG

Attachment 30: 1800_LeakageCurrents.PNG

Attachment 31: 210310_2039_Stats.png

Attachment 32: 210310_2040_Temp.png

Attachment 33: 210310_2041_Bias.png

Attachment 34: 210310_2241_Stats.png

Attachment 35: 210310_2241_Temp.png

Attachment 36: 210310_2242_Bias.png

Attachment 37: 210310_2331_Stats.png

Attachment 38: 210310_2332_Temp.png

Attachment 39: 210310_2333_Bias.png

	186	Tue Mar 9 23:12:17 2021	CA	March 10th 00:00 - 08:00
00:13 beam back, but some spills 'missing' DESPEC keeping beam as is, but may stop later if it worsens 00:15 DAQ continues OK - file R20_622 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 00:20 System wide checks all OK except ADC Calibration FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Check FPGA Timestamp Errors Base Current Difference aida12 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last 00:24 still no bad timestamp errors in NewMerger since 16:30 UTC 00:30 FEE64 Temperatures OK - attachment 1 Good event statistics OK - attachment 2 Detector bias & leakage currents OK - attachment 3 Merger OK - 4.2M data items/s TapeServer OK - 45 Mb/s 01:30 rate spectra - attachment 4 HEC spectra - attachment 5 & 6 note aida04 spectrum still not showing 02:07 beam off - file R20_760 02:11 beam back - file R20_764 02:44 System wide checks all OK except ADC Calibration FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 Check FPGA Timestamp Errors Base Current Difference aida12 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 still no bad timestamp errors in NewMerger since 16:30 UTC FEE64 Temperatures OK - attachment 7 Good event statistics OK - attachment 8 Detector bias & leakage currents OK - attachment 9 Merger OK - 4.2M data items/s TapeServer OK - 44 Mb/s 04:16 bad timestamp errors in NewMerger terminal, first since 16:30 UTC - attachment 10 System wide checks all OK except ADC Calibration FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 Check FPGA Timestamp Errors Base Current Difference aida12 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 FEE64 Temperatures OK - attachment 11 Good event statistics OK - attachment 12 Detector bias & leakage currents OK - attachment 13 05:03 Merger OK - 4.5M data items/s TapeServer OK - 43 Mb/s no bad timestamp errors for ~40 mins, data forwarding to MBS ok at usual rate 06:11 rates spectra - attachment 14 06:14 System wide checks ADC Calibration FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 Check WR decoder status Base Current Difference aida05 fault 0x1591 : 0x1593 : 2 aida06 fault 0xe23c : 0xe23e : 2 aida07 fault 0x743 : 0x745 : 2 aida08 fault 0xf84e : 0xf850 : 2 aida09 fault 0x5449 : 0x544a : 1 White Rabbit error counter test result: Passed 7, Failed 5 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Check FPGA Timestamp Errors Base Current Difference aida12 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 still no further bad timestamp errors in NewMerger terminal collected all WR and FPGA errors from baseline, system wide checks ok FEE64 Temperatures OK - attachment 15 Good event statistics OK - attachment 16 Detector bias & leakage currents OK - attachment 17 06:24 AIDA writing to file R20_1080 Merger OK - 4.5M data items/s TapeServer OK - 46 Mb/s 06:26 another burst of bad timestamp errors - attachment 18 performed system wide checks again - all ok aside from aida07/aida10 calibration 07:32 HEC spectra - attachments 19/20 07:35 no beam
Attachment 1: Screenshot_from_2021-03-09_23-27-20.png

Attachment 2: Screenshot_from_2021-03-09_23-29-22.png

Attachment 3: Screenshot_from_2021-03-09_23-29-39.png

Attachment 4: Screenshot_from_2021-03-10_00-27-45.png

Attachment 5: Screenshot_from_2021-03-10_00-28-39.png

Attachment 6: Screenshot_from_2021-03-10_00-29-25.png

Attachment 7: Screenshot_from_2021-03-10_01-40-24.png

Attachment 8: Screenshot_from_2021-03-10_01-42-57.png

Attachment 9: Screenshot_from_2021-03-10_01-43-39_-_1.png

Attachment 10: Screenshot_from_2021-03-10_03-17-56.png

Attachment 11: Screenshot_from_2021-03-10_03-23-03.png

Attachment 12: Screenshot_from_2021-03-10_03-23-26.png

Attachment 13: Screenshot_from_2021-03-10_03-23-43.png

Attachment 14: Screenshot_from_2021-03-10_05-10-19.png

Attachment 15: Screenshot_from_2021-03-10_05-17-39.png

Attachment 16: Screenshot_from_2021-03-10_05-18-04.png

Attachment 17: Screenshot_from_2021-03-10_05-21-47.png

Attachment 18: Screenshot_from_2021-03-10_05-27-03.png

Attachment 19: Screenshot_from_2021-03-10_06-31-51.png

Attachment 20: Screenshot_from_2021-03-10_06-32-18.png

	185	Tue Mar 9 20:04:20 2021	TD	Tuesday 9 March
21.04 DAQ continues OK - file R20_379 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse System wide checks all OK except ADC Calibration FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Check FPGA Timestamp Errors Base Current Difference aida12 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last FEE64 Temperatures OK - attachment 1 Good event statistics OK - attachment 2 Detector bias & leakage currents OK - attachment 3 Merger OK - 4.2M data items/s TapeServer OK - 45 Mb/s 21.20 no merger server error/warning messages since 16.30UTC 21.35 p+n junction strip HEC & LEC spectra, Rate spectra common y-scale 0-30000 & 0-10 all spectra zero'd 22.05 no beam for 2-3h whilst accelerator generator is fixed
Attachment 1: Screenshot_2021-03-09_Temperature_and_status_scan_aidas-gsi.png

Attachment 2: Screenshot_2021-03-09_Statistics_aidas-gsi.png

Attachment 3: 10.png

Attachment 4: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi.png

Attachment 5: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(1).png

Attachment 6: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(2).png

Attachment 7: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(3).png

	184	Tue Mar 9 10:23:45 2021	TD	Analysis of R14 (w/ NewMerger with min info code 4 & 5 data items)
Per https://elog.ph.ed.ac.uk/DESPEC/183 R14 data files have been analysed I think we still need paired info code 4 & 5 data items. I also don't understand the pattern of file sizes ... 'old' TapeServer buffers being flushed? [npg@aidas-gsi analyser]$ ls -lrth /TapeData/S452/R14* -rw-rw-r--. 1 npg npg 64K Mar 9 10:07 /TapeData/S452/R14_0 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_1 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_2 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_3 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_4 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_5 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_6 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_7 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_8 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_9 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_10 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_11 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_12 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_13 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_14 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_15 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_16 -rw-rw-r--. 1 npg npg 64K Mar 9 10:23 /TapeData/S452/R14_17 -rw-rw-r--. 1 npg npg 363M Mar 9 10:23 /TapeData/S452/R14_18 -rw-rw-r--. 1 npg npg 2.0G Mar 9 10:26 /TapeData/S452/R14_19 -rw-rw-r--. 1 npg npg 1.6G Mar 9 10:28 /TapeData/S452/R14_20 R14_0 (attachment 1) has multiple PAUSE/RESUME data items at start of file (R14_0) followed by 1x info code 4 & 5 data items (as expected) Also true for other short (64k) files - multiple PAUSE/RESUME data items, 1x info code 4 & 5 data items, and ADC data from one FEE64 module only [npg@aidas-gsi analyser]$ ./analyser v /TapeData/S452/R14_0 \| more * TDR format 3.3.0 analyser - TD - January 2019 verbose * RESUME timestamp: block: 1 ptr: 7 data: 0x843A160B module: 4 information type: 3 information field: 0x000A160B ts: 0x0000A160B1733D66 * PAUSE timestamp: block: 1 ptr: 9 data: 0x842A160B module: 4 information type: 2 information field: 0x000A160B ts: 0x0000A160B6508028 * RESUME timestamp: block: 1 ptr: 11 data: 0x873A160B module: 7 information type: 3 information field: 0x000A160B ts: 0x0000A160B7727448 * RESUME timestamp: block: 1 ptr: 13 data: 0x823A160B module: 2 information type: 3 information field: 0x000A160B ts: 0x0000A160B775CA9E * PAUSE timestamp: block: 1 ptr: 15 data: 0x872A160B module: 7 information type: 2 information field: 0x000A160B ts: 0x0000A160BC4F8028 * PAUSE timestamp: block: 1 ptr: 17 data: 0x822A160B module: 2 information type: 2 information field: 0x000A160B ts: 0x0000A160BC548028 * RESUME timestamp: block: 1 ptr: 19 data: 0x863A160B module: 6 information type: 3 information field: 0x000A160B ts: 0x0000A160BFE804B2 * PAUSE timestamp: block: 1 ptr: 21 data: 0x862A160C module: 6 information type: 2 information field: 0x000A160C ts: 0x0000A160C4C70028 * RESUME timestamp: block: 1 ptr: 23 data: 0x803A160C module: 0 information type: 3 information field: 0x000A160C ts: 0x0000A160C863646A * RESUME timestamp: block: 1 ptr: 25 data: 0x833A160C module: 3 information type: 3 information field: 0x000A160C ts: 0x0000A160CC641992 * PAUSE timestamp: block: 1 ptr: 27 data: 0x802A160C module: 0 information type: 2 information field: 0x000A160C ts: 0x0000A160CD410028 * PAUSE timestamp: block: 1 ptr: 29 data: 0x832A160D module: 3 information type: 2 information field: 0x000A160D ts: 0x0000A160D1420028 * RESUME timestamp: block: 1 ptr: 31 data: 0x853A160D module: 5 information type: 3 information field: 0x000A160D ts: 0x0000A160D3D39E8C * RESUME timestamp: block: 1 ptr: 33 data: 0x883A160D module: 8 information type: 3 information field: 0x000A160D ts: 0x0000A160D3D4D5EA * PAUSE timestamp: block: 1 ptr: 35 data: 0x852A160D module: 5 information type: 2 information field: 0x000A160D ts: 0x0000A160D8B30028 * PAUSE timestamp: block: 1 ptr: 37 data: 0x882A160D module: 8 information type: 2 information field: 0x000A160D ts: 0x0000A160D8B30028 * RESUME timestamp: block: 1 ptr: 39 data: 0x813A160D module: 1 information type: 3 information field: 0x000A160D ts: 0x0000A160D9F76CF8 * PAUSE timestamp: block: 1 ptr: 41 data: 0x812A160D module: 1 information type: 2 information field: 0x000A160D ts: 0x0000A160DED50028 * RESUME timestamp: block: 1 ptr: 43 data: 0x843A160F module: 4 information type: 3 information field: 0x000A160F ts: 0x0000A160F7AD970E * RESUME timestamp: block: 1 ptr: 45 data: 0x8A3A160F module: 10 information type: 3 information field: 0x000A160F ts: 0x0000A160F938391C * RESUME timestamp: block: 1 ptr: 47 data: 0x893A160F module: 9 information type: 3 information field: 0x000A160F ts: 0x0000A160FDC0600E * PAUSE timestamp: block: 1 ptr: 49 data: 0x8A2A160F module: 10 information type: 2 information field: 0x000A160F ts: 0x0000A160FE178028 * RESUME timestamp: block: 1 ptr: 51 data: 0x823A1610 module: 2 information type: 3 information field: 0x000A1610 ts: 0x0000A161009DF9B2 * RESUME timestamp: block: 1 ptr: 53 data: 0x873A1610 module: 7 information type: 3 information field: 0x000A1610 ts: 0x0000A16100B7FD08 * PAUSE timestamp: block: 1 ptr: 55 data: 0x892A1610 module: 9 information type: 2 information field: 0x000A1610 ts: 0x0000A161029F8028 * RESUME timestamp: block: 1 ptr: 57 data: 0x863A1610 module: 6 information type: 3 information field: 0x000A1610 ts: 0x0000A16108134396 * RESUME timestamp: block: 1 ptr: 59 data: 0x803A1610 module: 0 information type: 3 information field: 0x000A1610 ts: 0x0000A1610C484524 * RESUME timestamp: block: 1 ptr: 61 data: 0x833A1610 module: 3 information type: 3 information field: 0x000A1610 ts: 0x0000A1610F8B0910 * RESUME timestamp: block: 1 ptr: 63 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16112BA0000 * RESUME timestamp: block: 1 ptr: 65 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16112E48000 * RESUME timestamp: block: 1 ptr: 67 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161130F0000 * RESUME timestamp: block: 1 ptr: 69 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16113398000 * RESUME timestamp: block: 1 ptr: 71 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16113640000 * RESUME timestamp: block: 1 ptr: 73 data: 0x883A1611 module: 8 information type: 3 information field: 0x000A1611 ts: 0x0000A161137B1560 * RESUME timestamp: block: 1 ptr: 75 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161138E8000 * RESUME timestamp: block: 1 ptr: 77 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16113B90000 * RESUME timestamp: block: 1 ptr: 79 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16113E38000 * RESUME timestamp: block: 1 ptr: 81 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161140E0000 * RESUME timestamp: block: 1 ptr: 83 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16114388000 * RESUME timestamp: block: 1 ptr: 85 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16114630000 * RESUME timestamp: block: 1 ptr: 87 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161148D8000 * RESUME timestamp: block: 1 ptr: 89 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16114B80000 * RESUME timestamp: block: 1 ptr: 91 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16114E28000 * RESUME timestamp: block: 1 ptr: 93 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161150D0000 * RESUME timestamp: block: 1 ptr: 95 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16115378000 * RESUME timestamp: block: 1 ptr: 97 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16115620000 * RESUME timestamp: block: 1 ptr: 99 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161158C8000 * RESUME timestamp: block: 1 ptr: 101 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16115B70000 * RESUME timestamp: block: 1 ptr: 103 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16115E18000 * RESUME timestamp: block: 1 ptr: 105 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161160C0000 * RESUME timestamp: block: 1 ptr: 107 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16116368000 * RESUME timestamp: block: 1 ptr: 109 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16116610000 * RESUME timestamp: block: 1 ptr: 111 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161168B8000 * RESUME timestamp: block: 1 ptr: 113 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16116B60000 * RESUME timestamp: block: 1 ptr: 115 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16116E08000 * RESUME timestamp: block: 1 ptr: 117 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161170B0000 * RESUME timestamp: block: 1 ptr: 119 data: 0x853A1611 module: 5 information type: 3 information field: 0x000A1611 ts: 0x0000A161171BADC4 * RESUME timestamp: block: 1 ptr: 121 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16117358000 * RESUME timestamp: block: 1 ptr: 123 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A16117600000 * RESUME timestamp: block: 1 ptr: 125 data: 0x8B3A1611 module: 11 information type: 3 information field: 0x000A1611 ts: 0x0000A161178A8000 * RESUME timestamp: block: 1 ptr: 127 data: 0x813A1611 module: 1 information type: 3 information field: 0x000A1611 ts: 0x0000A1611A161F3C * RESUME timestamp: block: 1 ptr: 129 data: 0x8A3A1613 module: 10 information type: 3 information field: 0x000A1613 ts: 0x0000A1613C6AAE72 * RESUME timestamp: block: 1 ptr: 131 data: 0x893A1613 module: 9 information type: 3 information field: 0x000A1613 ts: 0x0000A1613F230BBE * WR48-64 timestamp: block: 1 ptr: 133 data: 0x8450166A module: 4 information type: 5 information field: 0x0000166A ts: 0x166AA1613483FE84 * SYNC100 timestamp: block: 1 ptr: 135 data: 0x844A1622 module: 4 information type: 4 information field: 0x000A1622 ts: 0x166AA1622483FE84 * ADC data: block: 1 ptr: 137 data: 0xC13C7FEB module: 4 fail: 0 range: 0 id: 316 channel: 60 adc: 32747 ts: 0x166AA1622483FE84 * ADC data: block: 1 ptr: 139 data: 0xC1067E53 module: 4 fail: 0 range: 0 id: 262 channel: 6 adc: 32339 ts: 0x166AA16224840654 * ADC data: block: 1 ptr: 141 data: 0xC1167E4C module: 4 fail: 0 range: 0 id: 278 channel: 22 adc: 32332 ts: 0x166AA16224840654 * ADC data: block: 1 ptr: 143 data: 0xC1247EAF module: 4 fail: 0 range: 0 id: 292 channel: 36 adc: 32431 ts: 0x166AA16224840654 * ADC data: block: 1 ptr: 145 data: 0xC11F7D3F module: 4 fail: 0 range: 0 id: 287 channel: 31 adc: 32063 ts: 0x166AA16224840E24 * ADC data: block: 1 ptr: 147 data: 0xC12D7EF0 module: 4 fail: 0 range: 0 id: 301 channel: 45 adc: 32496 ts: 0x166AA16224840E24 * ADC data: block: 1 ptr: 149 data: 0xC1117E24 module: 4 fail: 0 range: 0 id: 273 channel: 17 adc: 32292 ts: 0x166AA16224842594 * ADC data: block: 1 ptr: 151 data: 0xC11E7D06 module: 4 fail: 0 range: 0 id: 286 channel: 30 adc: 32006 ts: 0x166AA16224844CA4 * ADC data: block: 1 ptr: 153 data: 0xC1217EB3 module: 4 fail: 0 range: 0 id: 289 channel: 33 adc: 32435 ts: 0x166AA16224845474 * ADC data: block: 1 ptr: 155 data: 0xC1347E05 module: 4 fail: 0 range: 0 id: 308 channel: 52 adc: 32261 ts: 0x166AA16224845474 * ADC data: block: 1 ptr: 157 data: 0xC1037EBC module: 4 fail: 0 range: 0 id: 259 channel: 3 adc: 32444 ts: 0x166AA16224845C44 * ADC data: block: 1 ptr: 159 data: 0xC13E7E99 module: 4 fail: 0 range: 0 id: 318 channel: 62 adc: 32409 ts: 0x166AA16224845C44 * ADC data: block: 1 ptr: 161 data: 0xC1047D6F module: 4 fail: 0 range: 0 id: 260 channel: 4 adc: 32111 ts: 0x166AA16224846414 * ADC data: block: 1 ptr: 163 data: 0xC1137EE5 module: 4 fail: 0 range: 0 id: 275 channel: 19 adc: 32485 ts: 0x166AA16224846414 * ADC data: block: 1 ptr: 165 data: 0xC10E7F30 module: 4 fail: 0 range: 0 id: 270 channel: 14 adc: 32560 ts: 0x166AA16224846BE4 * ADC data: block: 1 ptr: 167 data: 0xC1157D12 module: 4 fail: 0 range: 0 id: 277 channel: 21 adc: 32018 ts: 0x166AA16224846BE4 * ADC data: block: 1 ptr: 169 data: 0xC1287DD3 module: 4 fail: 0 range: 0 id: 296 channel: 40 adc: 32211 ts: 0x166AA16224846BE4 * ADC data: block: 1 ptr: 171 data: 0xC1197E69 module: 4 fail: 0 range: 0 id: 281 channel: 25 adc: 32361 ts: 0x166AA162248473B4 * ADC data: block: 1 ptr: 173 data: 0xC11B7EC0 module: 4 fail: 0 range: 0 id: 283 channel: 27 adc: 32448 ts: 0x166AA16224847B84 * ADC data: block: 1 ptr: 175 data: 0xC1017D82 module: 4 fail: 0 range: 0 id: 257 channel: 1 adc: 32130 ts: 0x166AA16224848354 * ADC data: block: 1 ptr: 177 data: 0xC1247E94 module: 4 fail: 0 range: 0 id: 292 channel: 36 adc: 32404 ts: 0x166AA162248492F4 * ADC data: block: 1 ptr: 179 data: 0xC1047D33 module: 4 fail: 0 range: 0 id: 260 channel: 4 adc: 32051 ts: 0x166AA1622484F884 * ADC data: block: 1 ptr: 181 data: 0xC13C7FFC module: 4 fail: 0 range: 0 id: 316 channel: 60 adc: 32764 ts: 0x166AA16224853ED4 * ADC data: block: 1 ptr: 183 data: 0xC1067E6A module: 4 fail: 0 range: 0 id: 262 channel: 6 adc: 32362 ts: 0x166AA16224855E14 * ADC data: block: 1 ptr: 185 data: 0xC1117E44 module: 4 fail: 0 range: 0 id: 273 channel: 17 adc: 32324 ts: 0x166AA16224858CF4 * ADC data: block: 1 ptr: 187 data: 0xC1167EA2 module: 4 fail: 0 range: 0 id: 278 channel: 22 adc: 32418 ts: 0x166AA162248594C4 * ADC data: block: 1 ptr: 189 data: 0xC1017EFB module: 4 fail: 0 range: 0 id: 257 channel: 1 adc: 32507 ts: 0x166AA1622485B404 * ADC data: block: 1 ptr: 191 data: 0xC1017F83 module: 4 fail: 0 range: 0 id: 257 channel: 1 adc: 32643 ts: 0x166AA16224864874 * ADC data: block: 1 ptr: 193 data: 0xC1197E3C module: 4 fail: 0 range: 0 id: 281 channel: 25 adc: 32316 ts: 0x166AA162248667B4 * ADC data: block: 1 ptr: 195 data: 0xC1017DF9 module: 4 fail: 0 range: 0 id: 257 channel: 1 adc: 32249 ts: 0x166AA1622486DCE4 * ADC data: block: 1 ptr: 197 data: 0xC1117E4E module: 4 fail: 0 range: 0 id: 273 channel: 17 adc: 32334 ts: 0x166AA162248759E4 * ADC data: block: 1 ptr: 199 data: 0xC1167E55 module: 4 fail: 0 range: 0 id: 278 channel: 22 adc: 32341 ts: 0x166AA1622487AFD4 * ADC data: block: 1 ptr: 201 data: 0xC1247E58 module: 4 fail: 0 range: 0 id: 292 channel: 36 adc: 32344 ts: 0x166AA1622487B7A4 * ADC data: block: 1 ptr: 203 data: 0xC10B7DFE module: 4 fail: 0 range: 0 id: 267 channel: 11 adc: 32254 ts: 0x166AA1622487CF14 * ADC data: block: 1 ptr: 205 data: 0xC12B7E6F module: 4 fail: 0 range: 0 id: 299 channel: 43 adc: 32367 ts: 0x166AA1622487CF14 * ADC data: block: 1 ptr: 207 data: 0xC1157D3B module: 4 fail: 0 range: 0 id: 277 channel: 21 adc: 32059 ts: 0x166AA1622487D6E4 * ADC data: block: 1 ptr: 209 data: 0xC1347E96 module: 4 fail: 0 range: 0 id: 308 channel: 52 adc: 32406 ts: 0x166AA1622487D6E4 * ADC data: block: 1 ptr: 211 data: 0xC11A7D38 module: 4 fail: 0 range: 0 id: 282 channel: 26 adc: 32056 ts: 0x166AA1622487DEB4 * ADC data: block: 1 ptr: 213 data: 0xC13C7F8E module: 4 fail: 0 range: 0 id: 316 channel: 60 adc: 32654 ts: 0x166AA1622487DEB4 * ADC data: block: 1 ptr: 215 data: 0xC11B7E79 module: 4 fail: 0 range: 0 id: 283 channel: 27 adc: 32377 ts: 0x166AA1622487E684 * ADC data: block: 1 ptr: 217 data: 0xC13D7EDB module: 4 fail: 0 range: 0 id: 317 channel: 61 adc: 32475 ts: 0x166AA1622487E684 * ADC data: block: 1 ptr: 219 data: 0xC1347EB8 module: 4 fail: 0 range: 0 id: 308 channel: 52 adc: 32440 ts: 0x166AA16224886B54 * ADC data: block: 1 ptr: 221 data: 0xC13C7F67 module: 4 fail: 0 range: 0 id: 316 channel: 60 adc: 32615 ts: 0x166AA16224887324 * ADC data: block: 1 ptr: 223 data: 0xC13D7F18 module: 4 fail: 0 range: 0 id: 317 channel: 61 adc: 32536 ts: 0x166AA1622488F7F4 * ADC data: block: 1 ptr: 225 data: 0xC1157D5C module: 4 fail: 0 range: 0 id: 277 channel: 21 adc: 32092 ts: 0x166AA16224890794 * ADC data: block: 1 ptr: 227 data: 0xC1047DED module: 4 fail: 0 range: 0 id: 260 channel: 4 adc: 32237 ts: 0x166AA16224891F04 *** ADC data: block: 1 ptr: 229 data: 0xC1127E66 module: 4 fail: 0 range: 0 id: 274 channel: 18 adc: 32358 ts: 0x166AA16224891F04 --More-- R14_19 (attachment 2) looks OK - ignore rate information as this is based upon incomplete timestamp data
Attachment 1: R14_0
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 1 ADC data format: 8120 ( 96053.8 Hz) Other data format: 65 ( 768.9 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 11 ( 130.1 Hz) RESUME: 52 ( 615.1 Hz) SYNC100: 1 ( 11.8 Hz) WR48-63: 1 ( 11.8 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 0.085 s FEE elapsed dead time(s) elapsed idle time(s) 1 0.994 0.000 2 1.146 0.000 3 1.045 0.000 4 1.097 0.000 5 1.047 0.000 6 1.129 0.000 7 1.148 0.000 8 0.986 0.000 9 1.015 0.000 10 1.046 0.000 11 0.000 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 0 3 0 0 1 2 0 0 0 0 0 0 1 0 3 0 0 1 2 0 0 0 0 0 0 2 0 3 0 0 1 2 0 0 0 0 0 0 3 0 3 0 0 1 2 0 0 0 0 0 0 4 8120 5 0 0 1 2 1 1 0 0 0 0 5 0 3 0 0 1 2 0 0 0 0 0 0 6 0 3 0 0 1 2 0 0 0 0 0 0 7 0 3 0 0 1 2 0 0 0 0 0 0 8 0 3 0 0 1 2 0 0 0 0 0 0 9 0 3 0 0 1 2 0 0 0 0 0 0 10 0 3 0 0 1 2 0 0 0 0 0 0 11 0 30 0 0 0 30 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 0.004s ( 256.000 blocks/s, 16.000 Mb/s)
Attachment 2: R14_19
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 261598852 ( 1789659.8 Hz) Other data format: 321148 ( 2197.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 1009 ( 6.9 Hz) RESUME: 1010 ( 6.9 Hz) SYNC100: 545 ( 3.7 Hz) WR48-63: 545 ( 3.7 Hz) FEE64 disc: 317229 ( 2170.2 Hz) MBS info: 810 ( 5.5 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 146.172 s FEE elapsed dead time(s) elapsed idle time(s) 1 0.992 139.586 2 0.853 142.002 3 1.182 142.271 4 0.575 139.050 5 6.293 131.265 6 0.208 139.586 7 0.037 142.002 8 39.955 138.244 9 9.898 143.613 10 0.448 141.465 11 32.263 145.760 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 21196377 1535 0 0 326 327 36 36 0 810 0 0 1 25527261 100 0 0 4 4 46 46 0 0 0 0 2 26242581 317335 0 0 6 6 47 47 317229 0 0 0 3 27512430 132 0 0 5 5 61 61 0 0 0 0 4 8193786 46 0 0 4 4 19 19 0 0 0 0 5 17476404 516 0 0 220 220 38 38 0 0 0 0 6 12155075 60 0 0 5 5 25 25 0 0 0 0 7 9702993 52 0 0 1 1 25 25 0 0 0 0 8 33325064 534 0 0 206 206 61 61 0 0 0 0 9 34533675 282 0 0 56 56 85 85 0 0 0 0 10 12187954 94 0 0 2 2 45 45 0 0 0 0 11 33545252 462 0 0 174 174 57 57 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time:40890.051s ( 0.783 blocks/s, 0.049 Mb/s)
	183	Tue Mar 9 06:58:14 2021	CA, LS	March 9th 08:00-11:00 shift
08:00 DAQ continues ok - writing to file R13_120 Merger ok - 4.6M data items/s TapeServer ok - 43 MB/s 08:03 - all system wide checks ok - Temperatures ok - attachment 1 - good event statistics ok - attachment 2 - detector bias/leakage currents ok - attachment 3 08:08 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 08:32 CA briefly lost network connection - back now 08:42 DESPEC on file 102 09:24 Rate spectra - attachment 4 09:32 HEC spectra - attachment 5/6 beam off - writing to file R13_243 09:42 merger bad timestamp errors - attachment 7 all system wide checks ok 10:30 NH - Beam was down, I try the new merger from Vic again (R14) I made a mistake and had to powercycle fees as they got stuck! aida10 seemed to reboot itslef After fixing my mistakes, all running with new merger and rate 15 MB/s (R14) It all looked OK even going to MBS/Go4 I reverted back to original hot merger, R15 and look at the data. Beam is now back and I return AIDA to shifters :) 11.30 System wide checks okay except ADC calibration: FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Reran calibration for both FEEs but still fail Statistics (attachment8) Rate spectra (attachment9) FEE temps (attachment10) Leakage currents written to sheet (attachment11) Tapeserver ~33MB/s Merger ~ 3.4M items/s AIDA file R15_92 11.55 Beam off for a couple minutes but back now 12.00 New Merge terminal disappeared so stopped daq and restarted, and also restarted MBS back working Now writing to R16 Merger ~ 4.5M items/s Tapeserver ~44MB/s 12.41 no beam 13.00 DAQ crashed (aida03 no responding), AIDA recovered from power cycle while beam is down Beam is back now, forwarding to MBS Merger ~ 45M items/s Tapeserver ~44MB/s Writing to R17 13.20 System wide checks okay except ADC calibration: FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Statistics (attachment12) Rate spectra (attachment13) FEE temps (attachment14) Leakage currents written to sheet (attachment15) Tapeserver ~45MB/s Merger ~ 4.4M items/s 14.27 beam off file R17_115 back a couple minutes later 14.40 no beam again back at 14.46 15.40 System wide checks okay except ADC calibration: FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Statistics (attachment16) Rate spectra (attachment17) FEE temps (attachment18) Leakage currents written to sheet (attachment19) Tapeserver ~44MB/s Merger ~ 4.4M items/s 15.52 beam down R17_221, back at 15.58 R17_229 16.07 Beam down 16:20 Removed S480 files from /media/1e... and /media/SecondDrive Set merger to no output and stopped tapedata. Changed symbolic link of /TapeData to /media/1e..... Restarted tapedata and the merger with outputting to file R20 Local DESPEC was made aware of this before carrying out and stopped the run. Beam back 17.12 beam down for a couple of minutes R20_90 17.40 System wide checks okay except ADC calibration: FEE64 module aida07 failed FEE64 module aida10 failed Calibration test result: Passed 10, Failed 2 If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module Statistics (attachment20) Rate spectra (attachment21) FEE temps (attachment22) Leakage currents written to sheet (attachment23) Tapeserver ~45MB/s Merger ~ 4.5M items/s
Attachment 1: Screenshot_from_2021-03-09_07-05-35.png

Attachment 2: Screenshot_from_2021-03-09_07-06-17.png

Attachment 3: Screenshot_from_2021-03-09_07-06-53.png

Attachment 4: Screenshot_from_2021-03-09_08-08-45_-_1.png

Attachment 5: Screenshot_from_2021-03-09_08-24-06.png

Attachment 6: Screenshot_from_2021-03-09_08-26-47.png

Attachment 7: Screenshot_from_2021-03-09_08-44-17.png

Attachment 8: 1130_Statistics.PNG

Attachment 9: 1130_SpectraRates.PNG

Attachment 10: 1130_FEETemps.PNG

Attachment 11: 1130_LeakageCurrents.PNG

Attachment 12: 1330_Statistics.PNG

Attachment 13: 1330_SpectraRates.PNG

Attachment 14: 1330_FEETemps.PNG

Attachment 15: 1330_LeakageCurrents.PNG

Attachment 16: 1540_Statistics.PNG

Attachment 17: 1540_SpectraRates.PNG

Attachment 18: 1540_FEETemps.PNG

Attachment 19: 1540_LeakageCurrents.PNG

Attachment 20: 1730_Statistics.PNG

Attachment 21: 1730_SpectraRates.PNG

Attachment 22: 1730_FEETemps.PNG

Attachment 23: 1730_LeakageCurrents.PNG

	182	Mon Mar 8 22:51:47 2021	OH	Tuesday 9th March
18:00 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 1 FEE Temperatures - attachment 2 Bias and leakage currents ok - attachment 3 00:12 Noticed this log in UCESB. Closed connection [140.181.60.97]... 1 clients... AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848) AIDA timewarp is over, skipped 2728 AIDA event(s) => Not emitting timewarped event (before 166a7a92fdac5c02) Recovered from timewarp but skipped 5 event(s) at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961 Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message 01:09 Another timewarp message AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974) => Not emitting timewarped event (before 166a83bc6226e9ea) Recovered from timewarp but skipped 6 event(s) AIDA timewarp is over, skipped 323841 AIDA event Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175 Again no timewarps in analysis of file. 02:00 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 4 FEE Temperatures - attachment 5 Bias and leakage currents ok - attachment 6 03:10 Problem with accelerator will be no beam for ~30 minutes 03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place. There are now also errors in all FEEs for WR events Base Current Difference aida01 fault 0xc45f : 0xc46a : 11 aida02 fault 0xd250 : 0xd25b : 11 aida03 fault 0x27b2 : 0x27bd : 11 aida04 fault 0x714b : 0x7156 : 11 aida05 fault 0x74f5 : 0x7500 : 11 aida06 fault 0x558a : 0x5595 : 11 aida07 fault 0x8d20 : 0x8d2b : 11 aida08 fault 0x8131 : 0x813c : 11 aida09 fault 0xf412 : 0xf41d : 11 aida10 fault 0x82c2 : 0x82cd : 11 aida11 fault 0x2845 : 0x2850 : 11 aida12 fault 0x1bf6 : 0x1c01 : 11 White Rabbit error counter test result: Passed 0, Failed 12 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Also errors in FPGA Base Current Difference aida03 fault 0x0 : 0x1 : 1 aida04 fault 0x0 : 0x1 : 1 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x1 : 1 aida12 fault 0x1 : 0x2 : 1 FPGA Timestamp error counter test result: Passed 7, Failed 5 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last The following are timestamp values from each of the FEEs taken in sequence If time does not increase in a reasonable manner run the system wide checks aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000 aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000 aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000 aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000 aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000 aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000 aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000 aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000 aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000 aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000 aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000 aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000 System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results AIDA DAQ Stopped (Still no beam) The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA). 03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully. All system wide checks in AIDA now pass. (AIDA06 no longer shows the error) Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again. 05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7 The problem was with the accelerator operators system. There was no beam. 06:20 Error in WR system wide checks. Rest all ok This time there is no bad merge items recorded in the merger terminal. There are also no timewarp events seen in ucesb which were observed with the issues at 3am. Base Current Difference aida01 fault 0xdbe9 : 0xdbea : 1 aida02 fault 0xa93d : 0xa93e : 1 aida03 fault 0x8fc4 : 0x8fc5 : 1 aida04 fault 0x3e56 : 0x3e57 : 1 aida05 fault 0xb04c : 0xb04d : 1 aida06 fault 0x5576 : 0x5577 : 1 aida07 fault 0x99ff : 0x9a00 : 1 aida08 fault 0x8412 : 0x8413 : 1 aida09 fault 0x2457 : 0x2458 : 1 White Rabbit error counter test result: Passed 3, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR The following are timestamp values from each of the FEEs taken in sequence If time does not increase in a reasonable manner run the system wide checks aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000 aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000 aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000 aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000 aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000 aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000 aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000 aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000 aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000 aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000 aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000 aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000 Possibly occured at 05:28 as a ucesb reports a timewarp at the time. AIDA Timewarp (166a922b75368928 before 166a922b799e6c92) AIDA timewarp is over, skipped 2728 AIDA event(s) Cannot check file as was not writing to file. Statistics - attachment 8 FEE Temperatures - attachment 9 Bias and leakage currents ok - attachment 10 06:40 There are occasional seconds of beam. According to operators it comes and then it goes again 06:45 Beam fairly stable so we are running again. AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19 07:00 Beam stopped again 07:02 Beam returned There are apparently issues with MUSIC chamber one and its resolution 07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing.
Attachment 1: 210308_2354_Stats.png

Attachment 2: 210308_2355_Temp.png

Attachment 3: 210308_2357_Bias.png

Attachment 4: 210309_0104_Stats.png

Attachment 5: 210309_0205_Temp.png

Attachment 6: 210309_0205_Bias.png

Attachment 7: 210309_0529_layout1.png

Attachment 8: 210309_0619_Stats.png

Attachment 9: 210309_0620_Temp.png

Attachment 10: 210309_0623_Bias.png

	181	Mon Mar 8 17:02:38 2021	CA	Monday 8th March 18:00 - 00:00
18:00 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 1 FEE Temperatures - attachment 2 Bias and leakage currents ok - attachment 3 18:24 Rate spectra - attachment 4 18:30 Merger ok - 4.4M data items/s TapeServer ok - 43MB/s Data forwarding to MBS ok 19:28 beam back after a brief outage - AIDA on file R9_736 FRS having DAQ issues 19:57 FRS DAQ/Scalars back 20:12 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 5 FEE Temperatures - attachment 6 Bias and leakage currents ok - attachment 7 Merger ok - 4.5M data items/s TapeServer ok - 46MB/s Data forwarding to MBS ok 20:33 No beam - AIDA on file R9_821 21:08 rates spectra - attachment 8 22:35 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 9 FEE Temperatures - attachment 10 Bias and leakage currents ok - attachment 11 Merger ok - 4.4M data items/s TapeServer ok - 43MB/s Data forwarding to MBS ok temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload. 23:07 media/ThirdDrive at 65% capacity writing to file R9_1021 file compression currently up to R1_686 23:36 beam stable ~1e9 pps rate spectra - attachment 12
Attachment 1: Screenshot_from_2021-03-08_17-05-44.png

Attachment 2: Screenshot_from_2021-03-08_17-07-04.png

Attachment 3: Screenshot_from_2021-03-08_17-08-51.png

Attachment 4: Screenshot_from_2021-03-08_17-23-58.png

Attachment 5: Screenshot_from_2021-03-08_19-10-56.png

Attachment 6: Screenshot_from_2021-03-08_19-11-33.png

Attachment 7: Screenshot_from_2021-03-08_19-11-53.png

Attachment 8: Screenshot_from_2021-03-08_20-08-12.png

Attachment 9: Screenshot_from_2021-03-08_21-28-29.png

Attachment 10: Screenshot_from_2021-03-08_21-29-08.png

Attachment 11: Screenshot_from_2021-03-08_21-32-35.png

Attachment 12: Screenshot_from_2021-03-08_22-38-41.png

	180	Mon Mar 8 09:45:20 2021	CA, TD	Analysis R7_20 (new version of NewMerger with min info code 4 & 5 data items)
An analysis of file R6_70 can be found at https://elog.ph.ed.ac.uk/DESPEC/177 attachment 19 We observe c. 80M ADC data items per 2Gb file and c. 160M info code 4 & 5 data items. This am NH switched NewMerger for a new version which minimises the number of info code 4 & 5 data items. The analysis of one of these data files (R7_20) is appended - attachment 1. We observe c. 2M ADC data items per 2Gb file. The number of info code 4 data items is significantly reduced as expected but info code 5 data items are not observed. Info code 4 & 5 data items will increment every (2^28 * 1e-9) 0.268s and (2^48 * 1e-9) 3.26d. If we examine the verbose output (attachment 2) we observe that each data block only contains c. 60 ADC data items - a small fraction of the capacity of each data block.
Attachment 1: R7_20
* TDR format 3.3.0 analyser - TD - January 2019 * ERROR: READ I/O error: 5002 blocks: 32000 ADC data format: 1822115 ( 2111176.3 Hz) Other data format: 1885 ( 2184.0 Hz) Sample trace data format: 0 ( 0.0 Hz) Undefined format: 0 ( 0.0 Hz) Other data format type: PAUSE: 8 ( 9.3 Hz) RESUME: 10 ( 11.6 Hz) SYNC100: 4 ( 4.6 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 1857 ( 2151.6 Hz) MBS info: 6 ( 7.0 Hz) Other info: 0 ( 0.0 Hz) ADC data range bit set: 0 ( 0.0 Hz) Timewarps: ADC: 0 ( 0.0 Hz) PAUSE: 0 ( 0.0 Hz) RESUME: 0 ( 0.0 Hz) SYNC100: 0 ( 0.0 Hz) WR48-63: 0 ( 0.0 Hz) FEE64 disc: 0 ( 0.0 Hz) MBS info: 0 ( 0.0 Hz) Undefined: 0 ( 0.0 Hz) Sample trace: 0 ( 0.0 Hz) * Timestamp elapsed time: 0.863 s FEE elapsed dead time(s) elapsed idle time(s) 1 0.000 0.000 2 0.206 0.000 3 0.000 0.000 4 0.000 0.000 5 0.060 0.000 6 0.000 0.000 7 0.000 0.000 8 0.201 0.000 9 0.166 0.805 10 0.000 0.000 11 0.592 0.000 12 0.000 0.000 13 0.000 0.000 14 0.000 0.000 15 0.000 0.000 16 0.000 0.000 17 0.000 0.000 18 0.000 0.000 19 0.000 0.000 20 0.000 0.000 21 0.000 0.000 22 0.000 0.000 23 0.000 0.000 24 0.000 0.000 25 0.000 0.000 26 0.000 0.000 27 0.000 0.000 28 0.000 0.000 29 0.000 0.000 30 0.000 0.000 31 0.000 0.000 32 0.000 0.000 * Statistics FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data 0 150071 10 0 0 2 2 0 0 0 6 0 0 1 194013 1 0 0 0 0 1 0 0 0 0 0 2 150373 1859 0 0 1 1 0 0 1857 0 0 0 3 230996 0 0 0 0 0 0 0 0 0 0 0 4 53452 0 0 0 0 0 0 0 0 0 0 0 5 142131 5 0 0 2 3 0 0 0 0 0 0 6 178856 0 0 0 0 0 0 0 0 0 0 0 7 74144 0 0 0 0 0 0 0 0 0 0 0 8 183174 3 0 0 1 2 0 0 0 0 0 0 9 218137 5 0 0 1 1 3 0 0 0 0 0 10 83482 0 0 0 0 0 0 0 0 0 0 0 11 163286 2 0 0 1 1 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 0 0 * Timewarps FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 31 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 * Program elapsed time: 23.008s ( 1390.832 blocks/s, 86.927 Mb/s)
Attachment 2: R7_20_verbose
[npg@aidas-gsi analyser]$ ./analyser v /TapeData/S452/R7_20 \| more * TDR format 3.3.0 analyser - TD - January 2019 verbose * ADC data: block: 1 ptr: 7 data: 0xC1177F65 module: 4 fail: 0 range: 0 id: 279 channel: 23 adc: 32613 ts: 0x00000000078DEB0E * ADC data: block: 1 ptr: 9 data: 0xC0478143 module: 1 fail: 0 range: 0 id: 71 channel: 7 adc: 33091 ts: 0x00000000078DEB4A * ADC data: block: 1 ptr: 11 data: 0xC0D182FC module: 3 fail: 0 range: 0 id: 209 channel: 17 adc: 33532 ts: 0x00000000078DEB68 * ADC data: block: 1 ptr: 13 data: 0xC2C081B9 module: 11 fail: 0 range: 0 id: 704 channel: 0 adc: 33209 ts: 0x00000000078DF068 * ADC data: block: 1 ptr: 15 data: 0xC2D7820D module: 11 fail: 0 range: 0 id: 727 channel: 23 adc: 33293 ts: 0x00000000078DF068 * ADC data: block: 1 ptr: 17 data: 0xC2E4835E module: 11 fail: 0 range: 0 id: 740 channel: 36 adc: 33630 ts: 0x00000000078DF068 * ADC data: block: 1 ptr: 19 data: 0xC2F681F4 module: 11 fail: 0 range: 0 id: 758 channel: 54 adc: 33268 ts: 0x00000000078DF068 * ADC data: block: 1 ptr: 21 data: 0xC00B7FDF module: 0 fail: 0 range: 0 id: 11 channel: 11 adc: 32735 ts: 0x00000000078DF1B2 * ADC data: block: 1 ptr: 23 data: 0xC0147DDE module: 0 fail: 0 range: 0 id: 20 channel: 20 adc: 32222 ts: 0x00000000078DF1B2 * ADC data: block: 1 ptr: 25 data: 0xC034803A module: 0 fail: 0 range: 0 id: 52 channel: 52 adc: 32826 ts: 0x00000000078DF1B2 * ADC data: block: 1 ptr: 27 data: 0xC272814E module: 9 fail: 0 range: 0 id: 626 channel: 50 adc: 33102 ts: 0x00000000078DF2A2 * ADC data: block: 1 ptr: 29 data: 0xC11B7E13 module: 4 fail: 0 range: 0 id: 283 channel: 27 adc: 32275 ts: 0x00000000078DF2DE * ADC data: block: 1 ptr: 31 data: 0xC1327D8E module: 4 fail: 0 range: 0 id: 306 channel: 50 adc: 32142 ts: 0x00000000078DF2DE * ADC data: block: 1 ptr: 33 data: 0xC05F7FB9 module: 1 fail: 0 range: 0 id: 95 channel: 31 adc: 32697 ts: 0x00000000078DF31A * ADC data: block: 1 ptr: 35 data: 0xC0C28087 module: 3 fail: 0 range: 0 id: 194 channel: 2 adc: 32903 ts: 0x00000000078DF338 * ADC data: block: 1 ptr: 37 data: 0xC0DB8075 module: 3 fail: 0 range: 0 id: 219 channel: 27 adc: 32885 ts: 0x00000000078DF338 * ADC data: block: 1 ptr: 39 data: 0xC1BA7ECE module: 6 fail: 0 range: 0 id: 442 channel: 58 adc: 32462 ts: 0x00000000078DF644 * ADC data: block: 1 ptr: 41 data: 0xC2C381FB module: 11 fail: 0 range: 0 id: 707 channel: 3 adc: 33275 ts: 0x00000000078DF838 * ADC data: block: 1 ptr: 43 data: 0xC2D981FD module: 11 fail: 0 range: 0 id: 729 channel: 25 adc: 33277 ts: 0x00000000078DF838 * ADC data: block: 1 ptr: 45 data: 0xC2E682A5 module: 11 fail: 0 range: 0 id: 742 channel: 38 adc: 33445 ts: 0x00000000078DF838 * ADC data: block: 1 ptr: 47 data: 0xC2F782E0 module: 11 fail: 0 range: 0 id: 759 channel: 55 adc: 33504 ts: 0x00000000078DF838 * ADC data: block: 1 ptr: 49 data: 0xC0177CE4 module: 0 fail: 0 range: 0 id: 23 channel: 23 adc: 31972 ts: 0x00000000078DF982 * ADC data: block: 1 ptr: 51 data: 0xC27A7F8E module: 9 fail: 0 range: 0 id: 634 channel: 58 adc: 32654 ts: 0x00000000078DFA72 * ADC data: block: 1 ptr: 53 data: 0xC0C3816A module: 3 fail: 0 range: 0 id: 195 channel: 3 adc: 33130 ts: 0x00000000078DFB08 * ADC data: block: 1 ptr: 55 data: 0xC0FD81FD module: 3 fail: 0 range: 0 id: 253 channel: 61 adc: 33277 ts: 0x00000000078DFB08 * ADC data: block: 1 ptr: 57 data: 0xC0847F9A module: 2 fail: 0 range: 0 id: 132 channel: 4 adc: 32666 ts: 0x00000000078DFCB6 * ADC data: block: 1 ptr: 59 data: 0xC0B77EED module: 2 fail: 0 range: 0 id: 183 channel: 55 adc: 32493 ts: 0x00000000078DFCB6 * ADC data: block: 1 ptr: 61 data: 0xC1AD8002 module: 6 fail: 0 range: 0 id: 429 channel: 45 adc: 32770 ts: 0x00000000078DFE14 * ADC data: block: 1 ptr: 63 data: 0xC2C4837A module: 11 fail: 0 range: 0 id: 708 channel: 4 adc: 33658 ts: 0x00000000078E0008 * ADC data: block: 1 ptr: 65 data: 0xC2DC8250 module: 11 fail: 0 range: 0 id: 732 channel: 28 adc: 33360 ts: 0x00000000078E0008 * ADC data: block: 1 ptr: 67 data: 0xC2EA8214 module: 11 fail: 0 range: 0 id: 746 channel: 42 adc: 33300 ts: 0x00000000078E0008 * ADC data: block: 1 ptr: 69 data: 0xC27B812C module: 9 fail: 0 range: 0 id: 635 channel: 59 adc: 33068 ts: 0x00000000078E0242 * ADC data: block: 1 ptr: 71 data: 0xC0C5811F module: 3 fail: 0 range: 0 id: 197 channel: 5 adc: 33055 ts: 0x00000000078E02D8 * ADC data: block: 1 ptr: 73 data: 0xC0877F4D module: 2 fail: 0 range: 0 id: 135 channel: 7 adc: 32589 ts: 0x00000000078E0486 * ADC data: block: 1 ptr: 75 data: 0xC2C782D6 module: 11 fail: 0 range: 0 id: 711 channel: 7 adc: 33494 ts: 0x00000000078E07D8 * ADC data: block: 1 ptr: 77 data: 0xC2DE8261 module: 11 fail: 0 range: 0 id: 734 channel: 30 adc: 33377 ts: 0x00000000078E07D8 * ADC data: block: 1 ptr: 79 data: 0xC2EB8115 module: 11 fail: 0 range: 0 id: 747 channel: 43 adc: 33045 ts: 0x00000000078E07D8 * ADC data: block: 1 ptr: 81 data: 0xC0027C33 module: 0 fail: 0 range: 0 id: 2 channel: 2 adc: 31795 ts: 0x00000000078E0922 * ADC data: block: 1 ptr: 83 data: 0xC0C68218 module: 3 fail: 0 range: 0 id: 198 channel: 6 adc: 33304 ts: 0x00000000078E0AA8 * ADC data: block: 1 ptr: 85 data: 0xC0D58287 module: 3 fail: 0 range: 0 id: 213 channel: 21 adc: 33415 ts: 0x00000000078E0AA8 * ADC data: block: 1 ptr: 87 data: 0xC08C4EFE module: 2 fail: 0 range: 0 id: 140 channel: 12 adc: 20222 ts: 0x00000000078E0C56 * ADC data: block: 1 ptr: 89 data: 0xC2F48203 module: 11 fail: 0 range: 0 id: 756 channel: 52 adc: 33283 ts: 0x00000000078E0FA8 * ADC data: block: 1 ptr: 91 data: 0xC267809D module: 9 fail: 0 range: 0 id: 615 channel: 39 adc: 32925 ts: 0x00000000078E11E2 * ADC data: block: 1 ptr: 93 data: 0xC044810C module: 1 fail: 0 range: 0 id: 68 channel: 4 adc: 33036 ts: 0x00000000078E125A * ADC data: block: 1 ptr: 95 data: 0xC0C9815B module: 3 fail: 0 range: 0 id: 201 channel: 9 adc: 33115 ts: 0x00000000078E1278 * ADC data: block: 1 ptr: 97 data: 0xC0DA8174 module: 3 fail: 0 range: 0 id: 218 channel: 26 adc: 33140 ts: 0x00000000078E1278 * ADC data: block: 1 ptr: 99 data: 0xC1827F8D module: 6 fail: 0 range: 0 id: 386 channel: 2 adc: 32653 ts: 0x00000000078E1584 * ADC data: block: 1 ptr: 101 data: 0xC1A77E56 module: 6 fail: 0 range: 0 id: 423 channel: 39 adc: 32342 ts: 0x00000000078E1584 * ADC data: block: 1 ptr: 103 data: 0xC2FD846E module: 11 fail: 0 range: 0 id: 765 channel: 61 adc: 33902 ts: 0x00000000078E1778 * ADC data: block: 1 ptr: 105 data: 0xC27081A7 module: 9 fail: 0 range: 0 id: 624 channel: 48 adc: 33191 ts: 0x00000000078E19B2 * ADC data: block: 1 ptr: 107 data: 0xC0CA8171 module: 3 fail: 0 range: 0 id: 202 channel: 10 adc: 33137 ts: 0x00000000078E1A48 * ADC data: block: 1 ptr: 109 data: 0xC0DF80F0 module: 3 fail: 0 range: 0 id: 223 channel: 31 adc: 33008 ts: 0x00000000078E1A48 * ADC data: block: 1 ptr: 111 data: 0xC1F28069 module: 7 fail: 0 range: 0 id: 498 channel: 50 adc: 32873 ts: 0x00000000078E1A7A * ADC data: block: 1 ptr: 113 data: 0xC1837E89 module: 6 fail: 0 range: 0 id: 387 channel: 3 adc: 32393 ts: 0x00000000078E1D54 * ADC data: block: 1 ptr: 115 data: 0xC1987EC6 module: 6 fail: 0 range: 0 id: 408 channel: 24 adc: 32454 ts: 0x00000000078E1D54 * ADC data: block: 1 ptr: 117 data: 0xC1B37F93 module: 6 fail: 0 range: 0 id: 435 channel: 51 adc: 32659 ts: 0x00000000078E1D54 * ADC data: block: 1 ptr: 119 data: 0xC2C28169 module: 11 fail: 0 range: 0 id: 706 channel: 2 adc: 33129 ts: 0x00000000078E1F48 * ADC data: block: 2 ptr: 7 data: 0xC2D5830E module: 11 fail: 0 range: 0 id: 725 channel: 21 adc: 33550 ts: 0x00000000078E1F48 * ADC data: block: 2 ptr: 9 data: 0xC2E18150 module: 11 fail: 0 range: 0 id: 737 channel: 33 adc: 33104 ts: 0x00000000078E1F48 * ADC data: block: 2 ptr: 11 data: 0xC0117F5D module: 0 fail: 0 range: 0 id: 17 channel: 17 adc: 32605 ts: 0x00000000078E2092 * ADC data: block: 2 ptr: 13 data: 0xC27E81BD module: 9 fail: 0 range: 0 id: 638 channel: 62 adc: 33213 ts: 0x00000000078E2182 * ADC data: block: 2 ptr: 15 data: 0xC07D817F module: 1 fail: 0 range: 0 id: 125 channel: 61 adc: 33151 ts: 0x00000000078E21FA * ADC data: block: 2 ptr: 17 data: 0xC0CB8125 module: 3 fail: 0 range: 0 id: 203 channel: 11 adc: 33061 ts: 0x00000000078E2218 * ADC data: block: 2 ptr: 19 data: 0xC1C28219 module: 7 fail: 0 range: 0 id: 450 channel: 2 adc: 33305 ts: 0x00000000078E224A * ADC data: block: 2 ptr: 21 data: 0xC1F77FD8 module: 7 fail: 0 range: 0 id: 503 channel: 55 adc: 32728 ts: 0x00000000078E224A * ADC data: block: 2 ptr: 23 data: 0xC08B7FBA module: 2 fail: 0 range: 0 id: 139 channel: 11 adc: 32698 ts: 0x00000000078E23C6 * ADC data: block: 2 ptr: 25 data: 0xC09F7E74 module: 2 fail: 0 range: 0 id: 159 channel: 31 adc: 32372 ts: 0x00000000078E23C6 * ADC data: block: 2 ptr: 27 data: 0xC1867FE8 module: 6 fail: 0 range: 0 id: 390 channel: 6 adc: 32744 ts: 0x00000000078E2524 * ADC data: block: 2 ptr: 29 data: 0xC19A7EC3 module: 6 fail: 0 range: 0 id: 410 channel: 26 adc: 32451 ts: 0x00000000078E2524 * ADC data: block: 2 ptr: 31 data: 0xC2C98098 module: 11 fail: 0 range: 0 id: 713 channel: 9 adc: 32920 ts: 0x00000000078E2718 * ADC data: block: 2 ptr: 33 data: 0xC2DA837D module: 11 fail: 0 range: 0 id: 730 channel: 26 adc: 33661 ts: 0x00000000078E2718 * ADC data: block: 2 ptr: 35 data: 0xC2E582C0 module: 11 fail: 0 range: 0 id: 741 channel: 37 adc: 33472 ts: 0x00000000078E2718 * ADC data: block: 2 ptr: 37 data: 0xC0CC81D9 module: 3 fail: 0 range: 0 id: 204 channel: 12 adc: 33241 ts: 0x00000000078E29E8 * ADC data: block: 2 ptr: 39 data: 0xC187804E module: 6 fail: 0 range: 0 id: 391 channel: 7 adc: 32846 ts: 0x00000000078E2CF4 * ADC data: block: 2 ptr: 41 data: 0xC2DB81B9 module: 11 fail: 0 range: 0 id: 731 channel: 27 adc: 33209 ts: 0x00000000078E2EE8 * ADC data: block: 2 ptr: 43 data: 0xC2E782E0 module: 11 fail: 0 range: 0 id: 743 channel: 39 adc: 33504 ts: 0x00000000078E2EE8 * ADC data: block: 2 ptr: 45 data: 0xC0CD80DF module: 3 fail: 0 range: 0 id: 205 channel: 13 adc: 32991 ts: 0x00000000078E31B8 * ADC data: block: 2 ptr: 47 data: 0xC0D380C6 module: 3 fail: 0 range: 0 id: 211 channel: 19 adc: 32966 ts: 0x00000000078E31B8 * ADC data: block: 2 ptr: 49 data: 0xC1897EEF module: 6 fail: 0 range: 0 id: 393 channel: 9 adc: 32495 ts: 0x00000000078E34C4 * ADC data: block: 2 ptr: 51 data: 0xC2E98401 module: 11 fail: 0 range: 0 id: 745 channel: 41 adc: 33793 ts: 0x00000000078E36B8 * ADC data: block: 2 ptr: 53 data: 0xC0E08147 module: 3 fail: 0 range: 0 id: 224 channel: 32 adc: 33095 ts: 0x00000000078E3988 * ADC data: block: 2 ptr: 55 data: 0xC2C881AC module: 11 fail: 0 range: 0 id: 712 channel: 8 adc: 33196 ts: 0x00000000078E3E88 * ADC data: block: 2 ptr: 57 data: 0xC2ED8349 module: 11 fail: 0 range: 0 id: 749 channel: 45 adc: 33609 ts: 0x00000000078E3E88 * ADC data: block: 2 ptr: 59 data: 0xC2F380C2 module: 11 fail: 0 range: 0 id: 755 channel: 51 adc: 32962 ts: 0x00000000078E3E88 * ADC data: block: 2 ptr: 61 data: 0xC1BD7C77 module: 6 fail: 0 range: 0 id: 445 channel: 61 adc: 31863 ts: 0x00000000078E4464 * ADC data: block: 2 ptr: 63 data: 0xC2D3837F module: 11 fail: 0 range: 0 id: 723 channel: 19 adc: 33663 ts: 0x00000000078E4658 * ADC data: block: 2 ptr: 65 data: 0xC24F80FC module: 9 fail: 0 range: 0 id: 591 channel: 15 adc: 33020 ts: 0x00000000078E4892 * ADC data: block: 2 ptr: 67 data: 0xC26C80FA module: 9 fail: 0 range: 0 id: 620 channel: 44 adc: 33018 ts: 0x00000000078E4892 * ADC data: block: 2 ptr: 69 data: 0xC27782CE module: 9 fail: 0 range: 0 id: 631 channel: 55 adc: 33486 ts: 0x00000000078E4892 * ADC data: block: 2 ptr: 71 data: 0xC0C08052 module: 3 fail: 0 range: 0 id: 192 channel: 0 adc: 32850 ts: 0x00000000078E4928 * ADC data: block: 2 ptr: 73 data: 0xC08E4C1D module: 2 fail: 0 range: 0 id: 142 channel: 14 adc: 19485 ts: 0x00000000078E4AD6 * ADC data: block: 2 ptr: 75 data: 0xC1817D36 module: 6 fail: 0 range: 0 id: 385 channel: 1 adc: 32054 ts: 0x00000000078E4C34 * ADC data: block: 2 ptr: 77 data: 0xC1BF7FAF module: 6 fail: 0 range: 0 id: 447 channel: 63 adc: 32687 ts: 0x00000000078E4C34 * ADC data: block: 2 ptr: 79 data: 0xC2E2826E module: 11 fail: 0 range: 0 id: 738 channel: 34 adc: 33390 ts: 0x00000000078E55F8 * ADC data: block: 2 ptr: 81 data: 0xC1C380E1 module: 7 fail: 0 range: 0 id: 451 channel: 3 adc: 32993 ts: 0x00000000078E58FA * ADC data: block: 2 ptr: 83 data: 0xC0F78257 module: 3 fail: 0 range: 0 id: 247 channel: 55 adc: 33367 ts: 0x00000000078E6098 * ADC data: block: 2 ptr: 85 data: 0xC2C681D1 module: 11 fail: 0 range: 0 id: 710 channel: 6 adc: 33233 ts: 0x00000000078E6598 * ADC data: block: 2 ptr: 87 data: 0xC04E8051 module: 1 fail: 0 range: 0 id: 78 channel: 14 adc: 32849 ts: 0x00000000078E684A * ADC data: block: 2 ptr: 89 data: 0xC0758189 module: 1 fail: 0 range: 0 id: 117 channel: 53 adc: 33161 ts: 0x00000000078E684A * ADC data: block: 2 ptr: 91 data: 0xC0948066 module: 2 fail: 0 range: 0 id: 148 channel: 20 adc: 32870 ts: 0x00000000078E6A16 * ADC data: block: 2 ptr: 93 data: 0xC0057DFD module: 0 fail: 0 range: 0 id: 5 channel: 5 adc: 32253 ts: 0x00000000078E6EB2 * ADC data: block: 2 ptr: 95 data: 0xC2418102 module: 9 fail: 0 range: 0 id: 577 channel: 1 adc: 33026 ts: 0x00000000078E6FA2 * ADC data: block: 2 ptr: 97 data: 0xC26B8030 module: 9 fail: 0 range: 0 id: 619 channel: 43 adc: 32816 ts: 0x00000000078E6FA2 * ADC data: block: 2 ptr: 99 data: 0xC0077C68 module: 0 fail: 0 range: 0 id: 7 channel: 7 adc: 31848 ts: 0x00000000078E7682 * ADC data: block: 2 ptr: 101 data: 0xC0107DBC module: 0 fail: 0 range: 0 id: 16 channel: 16 adc: 32188 ts: 0x00000000078E7682 * ADC data: block: 2 ptr: 103 data: 0xC26F8145 module: 9 fail: 0 range: 0 id: 623 channel: 47 adc: 33093 ts: 0x00000000078E7772 * ADC data: block: 2 ptr: 105 data: 0xC0D18120 module: 3 fail: 0 range: 0 id: 209 channel: 17 adc: 33056 ts: 0x00000000078E7808 * ADC data: block: 2 ptr: 107 data: 0xC2C081F4 module: 11 fail: 0 range: 0 id: 704 channel: 0 adc: 33268 ts: 0x00000000078E7D08 * ADC data: block: 2 ptr: 109 data: 0xC2D78240 module: 11 fail: 0 range: 0 id: 727 channel: 23 adc: 33344 ts: 0x00000000078E7D08 * ADC data: block: 2 ptr: 111 data: 0xC2E48241 module: 11 fail: 0 range: 0 id: 740 channel: 36 adc: 33345 ts: 0x00000000078E7D08 * ADC data: block: 2 ptr: 113 data: 0xC0137D35 module: 0 fail: 0 range: 0 id: 19 channel: 19 adc: 32053 ts: 0x00000000078E7E52 * ADC data: block: 2 ptr: 115 data: 0xC2728164 module: 9 fail: 0 range: 0 id: 626 channel: 50 adc: 33124 ts: 0x00000000078E7F42 * ADC data: block: 2 ptr: 117 data: 0xC0C18052 module: 3 fail: 0 range: 0 id: 193 channel: 1 adc: 32850 ts: 0x00000000078E7FD8 * ADC data: block: 2 ptr: 119 data: 0xC2C38244 module: 11 fail: 0 range: 0 id: 707 channel: 3 adc: 33348 ts: 0x00000000078E84D8 * ADC data: block: 3 ptr: 7 data: 0xC2D982DC module: 11 fail: 0 range: 0 id: 729 channel: 25 adc: 33500 ts: 0x00000000078E84D8 * ADC data: block: 3 ptr: 9 data: 0xC2E683C3 module: 11 fail: 0 range: 0 id: 742 channel: 38 adc: 33731 ts: 0x00000000078E84D8 * ADC data: block: 3 ptr: 11 data: 0xC2F68233 module: 11 fail: 0 range: 0 id: 758 channel: 54 adc: 33331 ts: 0x00000000078E84D8 * ADC data: block: 3 ptr: 13 data: 0xC0177D5C module: 0 fail: 0 range: 0 id: 23 channel: 23 adc: 32092 ts: 0x00000000078E8622 * ADC data: block: 3 ptr: 15 data: 0xC0C2803E module: 3 fail: 0 range: 0 id: 194 channel: 2 adc: 32830 ts: 0x00000000078E87A8 * ADC data: block: 3 ptr: 17 data: 0xC0FD81FC module: 3 fail: 0 range: 0 id: 253 channel: 61 adc: 33276 ts: 0x00000000078E87A8 * ADC data: block: 3 ptr: 19 data: 0xC1AD7F36 module: 6 fail: 0 range: 0 id: 429 channel: 45 adc: 32566 ts: 0x00000000078E8AB4 * ADC data: block: 3 ptr: 21 data: 0xC2F78642 module: 11 fail: 0 range: 0 id: 759 channel: 55 adc: 34370 ts: 0x00000000078E8CA8 * ADC data: block: 3 ptr: 23 data: 0xC0C3815B module: 3 fail: 0 range: 0 id: 195 channel: 3 adc: 33115 ts: 0x00000000078E8F78 * ADC data: block: 3 ptr: 25 data: 0xC0847EE9 module: 2 fail: 0 range: 0 id: 132 channel: 4 adc: 32489 ts: 0x00000000078E9126 * ADC data: block: 3 ptr: 27 data: 0xC0B77F35 module: 2 fail: 0 range: 0 id: 183 channel: 55 adc: 32565 ts: 0x00000000078E9126 * ADC data: block: 3 ptr: 29 data: 0xC27B8125 module: 9 fail: 0 range: 0 id: 635 channel: 59 adc: 33061 ts: 0x00000000078E96B2 * ADC data: block: 3 ptr: 31 data: 0xC08C3881 module: 2 fail: 0 range: 0 id: 140 channel: 12 adc: 14465 ts: 0x00000000078E98F6 * ADC data: block: 3 ptr: 33 data: 0xC2C181D1 module: 11 fail: 0 range: 0 id: 705 channel: 1 adc: 33233 ts: 0x00000000078E9C48 * ADC data: block: 3 ptr: 35 data: 0xC2D480BF module: 11 fail: 0 range: 0 id: 724 channel: 20 adc: 32959 ts: 0x00000000078E9C48 * ADC data: block: 3 ptr: 37 data: 0xC2648191 module: 9 fail: 0 range: 0 id: 612 channel: 36 adc: 33169 ts: 0x00000000078E9E82 * ADC data: block: 3 ptr: 39 data: 0xC04780C3 module: 1 fail: 0 range: 0 id: 71 channel: 7 adc: 32963 ts: 0x00000000078E9EFA * ADC data: block: 3 ptr: 41 data: 0xC0E6809B module: 3 fail: 0 range: 0 id: 230 channel: 38 adc: 32923 ts: 0x00000000078E9F18 * ADC data: block: 3 ptr: 43 data: 0xC2C4834F module: 11 fail: 0 range: 0 id: 708 channel: 4 adc: 33615 ts: 0x00000000078EA418 * ADC data: block: 3 ptr: 45 data: 0xC2DC8132 module: 11 fail: 0 range: 0 id: 732 channel: 28 adc: 33074 ts: 0x00000000078EA418 * ADC data: block: 3 ptr: 47 data: 0xC2FD8325 module: 11 fail: 0 range: 0 id: 765 channel: 61 adc: 33573 ts: 0x00000000078EA418 * ADC data: block: 3 ptr: 49 data: 0xC0C581D9 module: 3 fail: 0 range: 0 id: 197 channel: 5 adc: 33241 ts: 0x00000000078EA6E8 * ADC data: block: 3 ptr: 51 data: 0xC2E183D2 module: 11 fail: 0 range: 0 id: 737 channel: 33 adc: 33746 ts: 0x00000000078EABE8 * ADC data: block: 3 ptr: 53 data: 0xC0C9806C module: 3 fail: 0 range: 0 id: 201 channel: 9 adc: 32876 ts: 0x00000000078EAEB8 * ADC data: block: 3 ptr: 55 data: 0xC0DA814C module: 3 fail: 0 range: 0 id: 218 channel: 26 adc: 33100 ts: 0x00000000078EAEB8 * ADC data: block: 3 ptr: 57 data: 0xC1C28260 module: 7 fail: 0 range: 0 id: 450 channel: 2 adc: 33376 ts: 0x00000000078EAEEA * ADC data: block: 3 ptr: 59 data: 0xC08B7FEC module: 2 fail: 0 range: 0 id: 139 channel: 11 adc: 32748 ts: 0x00000000078EB066 * ADC data: block: 3 ptr: 61 data: 0xC09F7EF1 module: 2 fail: 0 range: 0 id: 159 channel: 31 adc: 32497 ts: 0x00000000078EB066 * ADC data: block: 3 ptr: 63 data: 0xC2668171 module: 9 fail: 0 range: 0 id: 614 channel: 38 adc: 33137 ts: 0x00000000078EB5F2 * ADC data: block: 3 ptr: 65 data: 0xC0CA8094 module: 3 fail: 0 range: 0 id: 202 channel: 10 adc: 32916 ts: 0x00000000078EB688 * ADC data: block: 3 ptr: 67 data: 0xC0DF806C module: 3 fail: 0 range: 0 id: 223 channel: 31 adc: 32876 ts: 0x00000000078EB688 * ADC data: block: 3 ptr: 69 data: 0xC1F782AB module: 7 fail: 0 range: 0 id: 503 channel: 55 adc: 33451 ts: 0x00000000078EB6BA * ADC data: block: 3 ptr: 71 data: 0xC2C282CB module: 11 fail: 0 range: 0 id: 706 channel: 2 adc: 33483 ts: 0x00000000078EBB88 * ADC data: block: 3 ptr: 73 data: 0xC07D8111 module: 1 fail: 0 range: 0 id: 125 channel: 61 adc: 33041 ts: 0x00000000078EBE3A * ADC data: block: 3 ptr: 75 data: 0xC0CB81B8 module: 3 fail: 0 range: 0 id: 203 channel: 11 adc: 33208 ts: 0x00000000078EBE58 * ADC data: block: 3 ptr: 77 data: 0xC2E98262 module: 11 fail: 0 range: 0 id: 745 channel: 41 adc: 33378 ts: 0x00000000078EC358 * ADC data: block: 3 ptr: 79 data: 0xC044807F module: 1 fail: 0 range: 0 id: 68 channel: 4 adc: 32895 ts: 0x00000000078EC60A * ADC data: block: 3 ptr: 81 data: 0xC19A7DA3 module: 6 fail: 0 range: 0 id: 410 channel: 26 adc: 32163 ts: 0x00000000078EC934 * ADC data: block: 3 ptr: 83 data: 0xC2ED8207 module: 11 fail: 0 range: 0 id: 749 channel: 45 adc: 33287 ts: 0x00000000078ECB28 * ADC data: block: 3 ptr: 85 data: 0xC2F380FC module: 11 fail: 0 range: 0 id: 755 channel: 51 adc: 33020 ts: 0x00000000078ECB28 * ADC data: block: 3 ptr: 87 data: 0xC2708152 module: 9 fail: 0 range: 0 id: 624 channel: 48 adc: 33106 ts: 0x00000000078ECD62 * ADC data: block: 3 ptr: 89 data: 0xC1B47F47 module: 6 fail: 0 range: 0 id: 436 channel: 52 adc: 32583 ts: 0x00000000078ED104 * ADC data: block: 3 ptr: 91 data: 0xC2C88150 module: 11 fail: 0 range: 0 id: 712 channel: 8 adc: 33104 ts: 0x00000000078ED2F8 * ADC data: block: 3 ptr: 93 data: 0xC27781B4 module: 9 fail: 0 range: 0 id: 631 channel: 55 adc: 33204 ts: 0x00000000078ED532 * ADC data: block: 3 ptr: 95 data: 0xC0C081FF module: 3 fail: 0 range: 0 id: 192 channel: 0 adc: 33279 ts: 0x00000000078ED5C8 * ADC data: block: 3 ptr: 97 data: 0xC1C780DD module: 7 fail: 0 range: 0 id: 455 channel: 7 adc: 32989 ts: 0x00000000078ED5FA * ADC data: block: 3 ptr: 99 data: 0xC1BD7D53 module: 6 fail: 0 range: 0 id: 445 channel: 61 adc: 32083 ts: 0x00000000078ED8D4 * ADC data: block: 3 ptr: 101 data: 0xC2D38300 module: 11 fail: 0 range: 0 id: 723 channel: 19 adc: 33536 ts: 0x00000000078EDAC8 * ADC data: block: 3 ptr: 103 data: 0xC0027F2E module: 0 fail: 0 range: 0 id: 2 channel: 2 adc: 32558 ts: 0x00000000078EDC12 * ADC data: block: 3 ptr: 105 data: 0xC0117FD8 module: 0 fail: 0 range: 0 id: 17 channel: 17 adc: 32728 ts: 0x00000000078EDC12 * ADC data: block: 3 ptr: 107 data: 0xC2588009 module: 9 fail: 0 range: 0 id: 600 channel: 24 adc: 32777 ts: 0x00000000078EDD02 * ADC data: block: 3 ptr: 109 data: 0xC1CD810E module: 7 fail: 0 range: 0 id: 461 channel: 13 adc: 33038 ts: 0x00000000078EDDCA * ADC data: block: 3 ptr: 111 data: 0xC1927F0C module: 6 fail: 0 range: 0 id: 402 channel: 18 adc: 32524 ts: 0x00000000078EE0A4 * ADC data: block: 3 ptr: 113 data: 0xC2DB8197 module: 11 fail: 0 range: 0 id: 731 channel: 27 adc: 33175 ts: 0x00000000078EE298 * ADC data: block: 3 ptr: 115 data: 0xC2E7821D module: 11 fail: 0 range: 0 id: 743 channel: 39 adc: 33309 ts: 0x00000000078EE298 * ADC data: block: 3 ptr: 117 data: 0xC0097F87 module: 0 fail: 0 range: 0 id: 9 channel: 9 adc: 32647 ts: 0x00000000078EE3E2 * ADC data: block: 3 ptr: 119 data: 0xC2598015 module: 9 fail: 0 range: 0 id: 601 channel: 25 adc: 32789 ts: 0x00000000078EE4D2 * ADC data: block: 4 ptr: 7 data: 0xC0E080A6 module: 3 fail: 0 range: 0 id: 224 channel: 32 adc: 32934 ts: 0x00000000078EE568 * ADC data: block: 4 ptr: 9 data: 0xC08E4C7E module: 2 fail: 0 range: 0 id: 142 channel: 14 adc: 19582 ts: 0x00000000078EE716 * ADC data: block: 4 ptr: 11 data: 0xC2E88140 module: 11 fail: 0 range: 0 id: 744 channel: 40 adc: 33088 ts: 0x00000000078EEA68 * ADC data: block: 4 ptr: 13 data: 0xC279812E module: 9 fail: 0 range: 0 id: 633 channel: 57 adc: 33070 ts: 0x00000000078EECA2 * ADC data: block: 4 ptr: 15 data: 0xC0438270 module: 1 fail: 0 range: 0 id: 67 channel: 3 adc: 33392 ts: 0x00000000078EED1A * ADC data: block: 4 ptr: 17 data: 0xC0D280BB module: 3 fail: 0 range: 0 id: 210 channel: 18 adc: 32955 ts: 0x00000000078EED38 * ADC data: block: 4 ptr: 19 data: 0xC0F78176 module: 3 fail: 0 range: 0 id: 247 channel: 55 adc: 33142 ts: 0x00000000078EED38 * ADC data: block: 4 ptr: 21 data: 0xC1817BAB module: 6 fail: 0 range: 0 id: 385 channel: 1 adc: 31659 ts: 0x00000000078EF044 * ADC data: block: 4 ptr: 23 data: 0xC2F481FD module: 11 fail: 0 range: 0 id: 756 channel: 52 adc: 33277 ts: 0x00000000078EF238 * ADC data: block: 4 ptr: 25 data: 0xC27A7E9E module: 9 fail: 0 range: 0 id: 634 channel: 58 adc: 32414 ts: 0x00000000078EF472 * ADC data: block: 4 ptr: 27 data: 0xC04E80E8 module: 1 fail: 0 range: 0 id: 78 channel: 14 adc: 33000 ts: 0x00000000078EF4EA * ADC data: block: 4 ptr: 29 data: 0xC0D3809C module: 3 fail: 0 range: 0 id: 211 channel: 19 adc: 32924 ts: 0x00000000078EF508 * ADC data: block: 4 ptr: 31 data: 0xC2C6810C module: 11 fail: 0 range: 0 id: 710 channel: 6 adc: 33036 ts: 0x00000000078EFA08 * ADC data: block: 4 ptr: 33 data: 0xC257808A module: 9 fail: 0 range: 0 id: 599 channel: 23 adc: 32906 ts: 0x00000000078EFC42 * ADC data: block: 4 ptr: 35 data: 0xC0107E1C module: 0 fail: 0 range: 0 id: 16 channel: 16 adc: 32284 ts: 0x00000000078F0322 * ADC data: block: 4 ptr: 37 data: 0xC25A7FA5 module: 9 fail: 0 range: 0 id: 602 channel: 26 adc: 32677 ts: 0x00000000078F0412 [npg@aidas-gsi analyser]$
	179	Mon Mar 8 07:02:17 2021	CA, LS	Monday 8th March 08:00 - 17:00
08:00 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude 1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 1 FEE Temperatures - attachment 2 Bias and leakage currents ok - attachment 3 08:20 Merger ~4.5M data items/s TapeServer ~ 45MB/s Writing to file R6_1148 09:00 Beam off for optimisation - writing to file R6_1212 Expected back at 09:30 09:30 Beam will be off for a bit longer - an hour or two 09:34 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 4 FEE Temperatures - attachment 5 Bias and leakage currents ok - attachment 6 Merger ~4.5M data items/s TapeServer ~ 45MB/s 10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds to finish Something confusing, will look at data and report to VP Reverted to original merger and back... 45 MB/s R9 12.00 (LS) System wide checks okay except 'FEE64 module aida06 global clocks failed, 6' Statistics (attachment7) FEE temps (attatchment8) Rate spectra (attachment9), beam still down Leakage Currents written to sheets (attachment10) Merger ~45M items/s Tapeserver ~43MB/s 14.00 Still no beam but is expected to return soon System wide checks okay except 'FEE64 module aida06 global clocks failed, 6' Statistics (attachment11) FEE temps (attatchment12) Leakage Currents written to sheets (attachment13) Merger ~45M items/s Tapeserver ~44MB/s 14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14) 15.00 Beam is back (attachment15) Still writing to R9 (roughly around R9_386) 15.28 No beam (attachment16) 15.41 Beam back (attachment17) Roughly R9 (roughly around R9_439) 16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18) 16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6' Statistics (attachment19) Spectra Rate (attachment20) FEE temps (attatchment21) Leakage Currents written to sheets (attachment22), still rising but looking on Grafana 7 day period should start to lower around 17.00 Merger ~45M items/s Tapeserver ~43MB/s 16.55 Beam down (attachment23), back a couple minutes later 15.51 beam down (attachment24)
Attachment 1: Screenshot_from_2021-03-08_07-04-21.png

Attachment 2: Screenshot_from_2021-03-08_07-07-24.png

Attachment 3: Screenshot_from_2021-03-08_07-07-49.png

Attachment 4: Screenshot_from_2021-03-08_08-31-37.png

Attachment 5: Screenshot_from_2021-03-08_08-32-58.png

Attachment 6: Screenshot_from_2021-03-08_08-34-25.png

Attachment 7: 1200_Statistics.PNG

Attachment 8: 1200_FEETemps.PNG

Attachment 9: 1200_SpectraRate.PNG

Attachment 10: 1200_LeakageCurrents.PNG

Attachment 11: 1400_Statistics.PNG

Attachment 12: 1400_FEETemps.PNG

Attachment 13: 1400_LeakageCurrents.PNG

Attachment 14: 1500_Grafana_LeakageCurrents.PNG

Attachment 15: 1500_SpectraRate.PNG

Attachment 16: 1528_SpectraRate.PNG

Attachment 17: 1541_SpectraRate.PNG

Attachment 18: 1600_R9_469_hit_rates.PNG

Attachment 19: 1620_Statistics.PNG

Attachment 20: 1620_SpectraRate.PNG

Attachment 21: 1620_FEETemps.PNG

Attachment 22: 1620_LeakageCurrents.PNG

Attachment 23: 1655_SpectraRate.PNG

Attachment 24: 1749_SpectraRate.PNG

	178	Sun Mar 7 23:13:50 2021	OH	Monday 8th March
00:00 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 00:14 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 1 FEE Temperatures - attachment 2 Bias and leakage currents ok - attachment 3 01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4 02:15 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 5 FEE Temperatures - attachment 6 Bias and leakage currents ok - attachment 7 03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped 04:00 Noticed more bad merge events System wide checks has also noted an FPGA error. All other system wide checks same results as previous Base Current Difference aida12 fault 0x0 : 0x1 : 1 FPGA Timestamp error counter test result: Passed 11, Failed 1 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last The following are timestamp values from each of the FEEs taken in sequence If time does not increase in a reasonable manner run the system wide checks aida01 : White Rabbit=> 166A3F29 BA0DF4D2 , WR/10=> 23DD31DC5CE3215, Readout Time => 23DD31DC6F34000 aida02 : White Rabbit=> 166A3F29 CC49ECD7 , WR/10=> 23DD31DC7A0FE15, Readout Time => 23DD31DC8428000 aida03 : White Rabbit=> 166A3F29 DF55B7E8 , WR/10=> 23DD31DC9889264, Readout Time => 23DD31DCA85C000 aida04 : White Rabbit=> 166A3F29 EECF82B7 , WR/10=> 23DD31DCB14C045, Readout Time => 23DD31DCC2B8000 aida05 : White Rabbit=> 166A3F29 FED2EFD2 , WR/10=> 23DD31DCCAEB195, Readout Time => 23DD31DCDC14000 aida06 : White Rabbit=> 166A3F2A 0EDC023D , WR/10=> 23DD31DCE49336C, Readout Time => 23DD31DCF204000 aida07 : White Rabbit=> 166A3F2A 1E8170D3 , WR/10=> 23DD31DCFD9BE7B, Readout Time => 23DD31DD10A0000 aida08 : White Rabbit=> 166A3F2A 30448F76 , WR/10=> 23DD31DD1A074BF, Readout Time => 23DD31DD2A34000 aida09 : White Rabbit=> 166A3F2A 3FE36E61 , WR/10=> 23DD31DD33057D6, Readout Time => 23DD31DD3B94000 aida10 : White Rabbit=> 166A3F2A 4E83A1F7 , WR/10=> 23DD31DD4A6C365, Readout Time => 23DD31DD46A8000 aida11 : White Rabbit=> 166A3F2A 5FD19086 , WR/10=> 23DD31DD661C1A7, Readout Time => 23DD31DD7664000 aida12 : White Rabbit=> 166A3F2A 6FE06F3D , WR/10=> 23DD31DD7FCD7EC, Readout Time => 23DD31DD9684000 04:07 Beam is back AIDA on file R6_815 Statistics - attachment 8 FEE Temperatures - attachment 9 Bias and leakage currents ok - attachment 10 ASIC check ok Multiple merge data error events observed 06:04 All system wide checks ok except aida06 fails clock status 6 Statistics - attachment 11 FEE Temperatures - attachment 12 Bias and leakage currents ok - attachment 13 07:22 Beam noticed dropped but back quickly
Attachment 1: 210308_0014_Stats.png

Attachment 2: 2303108_0015_Temp.png

Attachment 3: 210308_0016_Bias.png

Attachment 4: 210308_0120_System.png

Attachment 5: 210308_0213_Stats.png

Attachment 6: 210308_0213_Temp.png

Attachment 7: 210308_0215_Bias.png

Attachment 8: 210308_0403_Stats.png

Attachment 9: 210308_0404_Temp.png

Attachment 10: 210308_04011_Bias.png

Attachment 11: 210308_0602_Stats.png

Attachment 12: 210308_0603_Temp.png

Attachment 13: 210308_0604_Bias.png

	177	Sun Mar 7 07:23:26 2021	CA, LS	March 7th
08:00 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xa BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 08:30 - all system wide checks ok - Temperatures ok - attachment 1 - good event statistics ok - attachment 2 - detector bias/leakage currents ok - attachment 3 DAQ continues to run - R1_1148 Merger ok - 4.6M data items/s TapeServer ok - 43MB/s 08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots 09:19 zeroed all histograms writing to file R1_1204 FRS on file 72 DESPEC to add AIDA XY hit patterns to online analysis histograms 09:35 beam stop 09:56 beam back writing to file R1_1249 rate spectra - attachment 5 09:30 AIDA DAQ crashed - lost connection to aida05 power cycle and reset 09:40 AIDA ok now, writing to file R2 09:50 LS takes over 11.10 (German time) System wide checks produces some fails: FEE64 module aida06 global clocks failed, 6 (screenshot6) This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there FEE64 module aida06 failed FEE64 module aida07 failed Calibration test result: Passed 10, Failed 2 (screenshot7) Tried a recalibration for the two FEEs in FADC align & control tab but error still there Rest of the checks pass Statistics (screenshot8) FEE temps okay (screenshot9) Leakage currents written to sheets (screenshot10) Merger ~4.5M data items/s TapeServer ~ 45MB/s 13.00 System wide checks produces same fails as last time: FEE64 module aida06 global clocks failed, 6 FEE64 module aida06 failed FEE64 module aida07 failed Calibration test result: Passed 10, Failed 2 Rest of the checks are okay Statistics (screenshot11) Spectra rate (screenshot12) FEE temps okay (screenshot13) Leakage currents written to sheets (screenshot14) Merger ~4.5M data items/s TapeServer ~ 45MB/s There is 1.5TB left on current drive 14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold Slow comparators in FEE9-12 raised to 0xd
Attachment 1: Screenshot_from_2021-03-07_07-30-28.png

Attachment 2: Screenshot_from_2021-03-07_07-31-27.png

Attachment 3: Screenshot_from_2021-03-07_07-31-55.png

Attachment 4: Screenshot_from_2021-03-07_07-53-05.png

Attachment 5: Screenshot_from_2021-03-07_08-59-23.png

Attachment 6: 1110_Systemchecks.PNG

Attachment 7: 1110_Systemchecks2.PNG

Attachment 8: 1110_Statistics.PNG

Attachment 9: 1110_FEEtemps.PNG

Attachment 10: 1110_Leakagecurrents.PNG

Attachment 11: 1300_Statistics.PNG

Attachment 12: 1300_Spectrumrate.PNG

Attachment 13: 1300_FEEtemps.PNG

Attachment 14: 1300_Leakagecurrents.PNG

	176	Sat Mar 6 23:37:22 2021	TD	Sunday 7 March 00.00-08:00
00.30 UNILAC problem - no beam 00:39 DAQ continues R1_518 FRS 190W setting ASIC settinfs 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xa BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse 01:07 all system wide checks OK detector biases & leakage currents OK - see attachment 1 FEE64 temperatures OK - see attachment 2 good event statistics - OK - see attachment 3 Merger OK - see attachment 4 TapeServer OK - see attachment 5 01:20 zero'd all histograms ASIC check load rate spectra - see attachment 6 01:48 beam return run74 (R1_608) 03.05 all system wide checks OK detector biases & leakage currents OK - see attachment 7 FEE64 temperatures OK - see attachment 8 good event statistics - OK - see attachment 9 Merger OK - see attachment 10 TapeServer OK - see attachment 11 rate spectra - see attachment 12 04.46 run74 ends (R1_841) 05.00 run75 (R1_859) 05:40 detector biases & leakage currents OK - see attachment 13 FEE64 temperatures OK - see attachment 14 good event statistics - OK - see attachment 15 Merger OK - see attachment 16 TapeServer OK - see attachment 17 rate spectra - see attachment 18 07:29 run76 (R1_1055)
Attachment 1: 1.png

Attachment 2: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png

Attachment 3: Screenshot_2021-03-07_Statistics_aidas-gsi.png

Attachment 4: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png

Attachment 5: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png

Attachment 6: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png

Attachment 7: 2.png

Attachment 8: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png

Attachment 9: Screenshot_2021-03-07_Statistics_aidas-gsi(1).png

Attachment 10: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png

Attachment 11: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png

Attachment 12: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png

Attachment 13: 3.png

Attachment 14: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png

Attachment 15: Screenshot_2021-03-07_Statistics_aidas-gsi(2).png

Attachment 16: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png

Attachment 17: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png

Attachment 18: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png

	175	Sat Mar 6 06:52:06 2021	OH, LS	Saturday 6th March
07:50 Pulser settings - attachment 1 07:52 System wide checks all ok Statistics ok - attachment 2 Bias ok - attachment 3 09:30 The intensity of fragments is being increased They are trying to do a factor of 10 increase first which should take us to around 100Hz 10:00 System wide checks all ok Statistics - attachment 4 Layout 1 showing implants - attachment 5 Fee temperatures ok - attachment 6 12.00 System wide checks all okay, no failures Statistics (screenshot7) Spectrum rate (screenshot8) FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal (screenshot10) Leakage currents okay, increase from previous values written to sheets (screenshot12) Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer than usual, another reload shows all FEE temps as normal. Merger and tape server okay 14.00 System wide checks all okay, no failures Statistics (screenshot13) Spectrum rate (screenshot14) FEE temps normal, no issue with reload this time (screenshot15) Leakage currents okay still increasing, written to sheets (screenshot16) Merger okay ~4.5M items/sec Tape server ~4.5Mb/sec 14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011) 16.00 System wide checks all okay, no failures Statistics (screenshot17) Spectrum rate (screenshot18) FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20) Leakage currents added to sheet (screenshot21) Merger okay ~4.5M items/sec Tape server ~4.5Mb/sec 17.17 no beam, preparing for the 190Ta setting 17:44 OH Takes over System wide checks all ok Statistics - attachment 22 Temp ok - Attachment 23 Bias and leakage currents ok - Attatchment 24 While MBS was not currently writing data the merger -> tapeserver link was briefly toggle The Directory for the experiment was then setup S452 The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued. ASIC Check all ok 18:10 Started writing data to file File directory S452 Run number R1_0 Seems to have skipped a few files First full file R1_10 Currently writing to /media/SecondDrive/TapeData Current free space 4211084180 kB Current write speed 46163 kB/s Time remaining until full 91222 seconds 25 hours remaining space 18:34 DESPEC starting new MBS file AIDA Currently on file R1_41 19:00 DESPEC Closing file AIDA Currently on R1_75 19:05 Comparing last years statistics to this years FEE Last year This year Factor increase 1 60960 183111 3.00 2 183494 201256 1.09 3 91506 196647 2.15 4 73914 208290 2.81 5 31979 63571 1.99 6 54084 144638 2.67 7 35621 137605 3.86 8 38626 95858 2.48 9 264436 225790 0.85 10 209709 198896 0.94 11 88443 133899 1.51 12 183493 204996 1.12 Sum 1316265 1994557 1.51 19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file) AIDA on R1_104 Looking int to high write rate to see if we can do anything to reduce it It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24 19:46 Much of the rate is coming from a small number of channels - attachment 25 19:17 WR Timestamp error. Mentioned to Nic and he said that sometimes small glitches like this are observed Base Current Difference aida01 fault 0x9e8b : 0x9e8d : 2 aida02 fault 0x9a8f : 0x9a91 : 2 aida03 fault 0x29e9 : 0x29eb : 2 aida04 fault 0x7f66 : 0x7f68 : 2 aida05 fault 0x1f4f : 0x1f51 : 2 aida06 fault 0x8faa : 0x8fac : 2 aida07 fault 0x53f0 : 0x53f2 : 2 aida08 fault 0x7066 : 0x7068 : 2 White Rabbit error counter test result: Passed 4, Failed 8 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR All other checks passed 20:52 DESPEC Stopping file AIDA on R1_221 20:53 DESPEC Stopped file because UCESB crashed AIDA on file R1_237 21:20 All system wide checks ok Statistics DISC info- Attachment 27 FEE Temp ok - Attachment 28 Bias and leakage current ok - Attachment 29 Statistics good events - Attachment 30 20:51 Started compressing data with command nice -n 10 gzip -v R1_* Have discussed with Helena and Nic about the potential issues we could run into with the high data rate I am not sure if MBS can keep up with this rate We are still writing to file though. 23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis 23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us
Attachment 1: Pulser.png

Attachment 2: 210306_0752_Stats.png

Attachment 3: 210306_0754_Bias.png

Attachment 4: 210306_0958_Stats.png

Attachment 5: 210306_0959_Layout1.png

Attachment 6: 210306_1000_Temp.png

Attachment 7: 1102_Statistics.PNG

Attachment 8: 1102_SpectrumRate.PNG

Attachment 9: 1102_FEETemps.PNG

Attachment 10: 1102_FEETemps2.PNG

Attachment 11: 1102_FEETemps3.PNG

Attachment 12: 1102_LeakageCurrents.PNG

Attachment 13: 1300_Statistics.PNG

Attachment 14: 1300_SpectrumRate.PNG

Attachment 15: 1300_FEETemps.PNG

Attachment 16: 1300_LeakageCurrents.PNG

Attachment 17: 1500_Statistics.PNG

Attachment 18: 1500_SpectrumRate.PNG

Attachment 19: 1500_FEETemps.PNG

Attachment 20: 1500_FEETemps2.PNG

Attachment 21: 1500_LeakageCurrents.PNG

Attachment 22: 210306_1726_Stats.png

Attachment 23: 210306_1747_Temp.png

Attachment 24: 210327_1748_Bias.png

Attachment 25: 210326_1920_MergerStats.png

Attachment 26: 210306_1946_Layout1.png

Attachment 27: 210306_2120_Stats.png

Attachment 28: 210306_2121_Temp.png

Attachment 29: 210306_2123_Bias.png

Attachment 30: 210326_1920_MergerStats.png

	174	Fri Mar 5 23:45:41 2021	TD	S452 information
aida-gsi Anydesk 897170655 DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/ S452 shift checks - attachment 1 S452 shift plan - PDF attachment 2 - Google doc https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1745844766 Mattermost - https://mattermost.gsi.de/despec/channels/experiment-s452-nearline-sort DESPEC Zoom meeting Join Zoom Meeting https://gsi-fair.zoom.us/j/98980822569 Meeting ID: 989 8082 2569 Passcode: S452 FRS Zoom meeting Meeting-ID: 973 4809 5114 password: FRS-BT2021 Kenncode: 456657
Attachment 1: S452__Shifts_checkpoints_v1_(1).pdf

Attachment 2: S452_Shift_Plan.pdf

	173	Fri Mar 5 23:21:59 2021	CA, OH	March 6th 2021 00:00 - 08:00 shift
00:23 206Hg setting - No beam in AIDA yet AIDA not writing to file Data being forwarded to MBS Fragments expected tonight (?) 00:45 GSI DAQ restart FEE64 Temps ok - attachment 1 All system wide checks ok 01:00 Good event statistics - attachment 2 01:48 detector biases/ leakage currents - ok - attachment 3 02:10 beam on, plug removed beam delivered to AIDA rates spectra - attachment 4 peaks at ~ 10 counts/channel in DSSD1 c.2000Hz 02:20 beam stop 02:26 beam restart - rate spectra - attachment 5 02:43 high energy channel spectra - attachment 6 saved the high energy spectra to layout 2 in spectrum browser 03:00 beam stop 03:08 temperatures ok - attachment 7 statistics ok - attachment 8 bias/leakage currents ok - attachment 9 NewMerger ok - ~4.5M data items merge rate Tape service ok - ~46MB/s, NoStorage Data forwarding to MBS ok all system wide checks ok 04:24 beam is back rate spectra - attachment 10 04:51 206Hg implanting in DSSD2 Rate spectra - attachment 11 HEC spectra - attachment 12 05:08 temperatures ok - attachment 13 statistics ok - attachment 14 detector bias/leakage currents ok - attachment 15 All system wide checks ok NewMerger ok - ~4.5M data items merge rate Tape service ok - ~46MB/s, NoStorage Data forwarding to MBS ok 05:45 beam stopped 05:50 beam back 06:45 beam running, but no implants in AIDA possible issues with beam delivery to S4 NewMerger ok - ~4.5M data items merge rate Tape service ok - ~46MB/s, NoStorage Data forwarding to MBS ok 07:10 temperatures ok - attachment 16 statistics ok - attachment 17 detector bias/leakage currents ok - attachment 18 all system wide checks OK
Attachment 1: Screenshot_from_2021-03-06_00-00-27.png

Attachment 2: Screenshot_from_2021-03-06_00-05-18.png

Attachment 3: Screenshot_from_2021-03-06_00-47-41.png

Attachment 4: Screenshot_from_2021-03-06_01-17-12.png

Attachment 5: Screenshot_from_2021-03-06_01-25-51.png

Attachment 6: Screenshot_from_2021-03-06_01-42-58.png

Attachment 7: Screenshot_from_2021-03-06_02-05-46.png

Attachment 8: Screenshot_from_2021-03-06_02-06-32.png

Attachment 9: Screenshot_from_2021-03-06_02-07-47.png

Attachment 10: Screenshot_from_2021-03-06_03-24-28.png

Attachment 11: Screenshot_from_2021-03-06_03-53-35.png

Attachment 12: Screenshot_from_2021-03-06_03-54-17_-_1.png

Attachment 13: Screenshot_from_2021-03-06_04-09-25.png

Attachment 14: Screenshot_from_2021-03-06_04-10-07.png

Attachment 15: Screenshot_from_2021-03-06_04-10-32.png

Attachment 16: Screenshot_from_2021-03-06_06-07-19.png

Attachment 17: Screenshot_from_2021-03-06_06-08-16.png

Attachment 18: Screenshot_from_2021-03-06_06-08-44.png

	172	Fri Mar 5 22:53:05 2021	OH	Thursday 5th of March
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time. At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system. At first a reset and setup was performed to try and regain correlation but that did not prove successful. A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue. The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk. Beam plug is in front of AIDA. This is causing a huge spike in the Ge.

Goto page Previous 1, 2, 3 ... 25, 26, 27 ... 33, 34, 35 Next

ELOG V3.1.4-unknown