 Fri Apr 26 07:01:26 2024, Norah, 08:00 - 16:00 Friday 26 April 
|
10:00 Checks:
All seems smooth.
DSSD bias and leakage current ( Attachment 1)
FEE64 temperatures (Attachment 2)
Per FEE64 Rate spectra (Attachment 3)
ADC data item stats -(Attachment 4)
Ucesb (Attachment 5)
10:21
Aida05 was zero not reading had to call Nic to fix it.
10:34
Now, Aida05 is running well.
.......................................................
12:00 Checks:
Everything appears to be going smoothly, and I took screenshot of them.
12:28, Aida02 showed as not reading , had to email Nic to fix it.
12:30, it resumed reading and returned to normal operation.
.........................................................
14:00 Checks:
Everything is going well , and I have captured screenshots of each one.
......................................................
16:00 Checks:
All seems smooth, and I took screenshot of them.
|
|
Thu Apr 25 22:45:30 2024, Tom, Muneerah, Norah, Over night shift AIDA 26 Fri 9x
|
01:02
Aida04 was zero not reading had to call Tom to fix it.
02:00 checked all good and attached.
Its been running good after that till 7:30 again the same problem happened and Tom fixed it.
|
|
Thu Apr 25 15:02:50 2024, Marc, 16:00 - 0:00 Thursday 25 April 8x
|
16:20:
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
16.45 Checks:
DSSSD bias & leakage current - Grafana - attachments 2
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 6
ucesb - attachment 7
XX.XX Checks
DSSSD bias & leakage current - Grafana - attachments X
FEE64 temperatures OK - attachment X
ADC data item stats - attachment X
per FEE64 Rate spectra - attachment X
Merger etc - attachment X
ucesb - attachment X |
|
Thu Apr 25 08:26:50 2024, TD, 08:00-16:00 Thursday 25 April 42x
|
09.56 DAQ continues data file S100_alpha/R18_387
FRS switching settings
DSSSD bias & leakage current OK - attachments 1
FEE64 temperatures OK - attachment 2
aida02 ASIC temp - known fault
All system wide checks OK *except* as shown - attachments 3-5
WR timestamps OK - attachment 6
ADC, PAUSE, RESUME & Correlation Scaler data time stats - attachments 7-10
per FEE64 Rate spectra - attachments 11
per p+n FEE64 1.8.L spectra - attachments 12-13
aida09 pulser peak width 57 ch FWHM
common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths
per n+n FEE64 1.8.L spectra - attachment 14
per FEE64 1.8.H spectra - attachments 15-16
per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?
Merger, TapeServer etc - attachments 19-21
BNC PB-5 settings - attachment 22
10.07 data file S100_alpha/R20
new run for new FRS setting
10.08 all histograms, stats, merger stats and data link stats zero'd
10.10 ADC data item stats (beam OFF) - attachment 23
11/16 < 20k, max 129k (aida08)
10.31 analysis of data file R18_349 (closed 06.00 this morning)
max deadtime 12% (aida04)
no timewarps
HEC data item rate 1.7kHz
11.00 tuning 168dy FRS setting
ucesb - attachment 24
spill length 3.5s, extraction time 1.5s
11.30 degraders now set to implant in AIDA DSSSDs
high rates expected
per FEE64 Rate spectra - attachment 25
ADC data item stats - attachment 26
Merger etc - attachment 27
in spill rates to MBS up 30+Mb/s, 2.5M data item/s merged
DSSSD bias & leakage current - attachment 28
c. few hundred nA step in leakage current
analysis of data file R20_33 - attachment 29
max deadtime 18% (aida08)
no timewarps
HEC data item rate 4.1kHz (c. 2.4x previous FRS setting)
Correcting for spill cycle => 5.9kHz in spill (online analysis shows no HEC events observed outside spill)
assume typical HEC event multiplicity = 3 => implant rate ~ 2.0kHz in spill or 6800 implants per spill
SC42 L/R currently report c. 4.5kHz
Note that we currently see a higher fraction of events in DSSSD#2 cf. previous setting so multiplicity assumption may be low
If we assume multipicity = 6 => implant rate ~ 1.0kHz in spill or 3200 implants per spill
we are probably somewhere in between
12.50 DSSSD bias & leakage current - Grafana - attachment 30
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 31
ADC data item stats - attachment 32
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
14.56 analysis of data file R20_106 - attachment 35
max deadtime c. 18% (aida04 & aida08), c. 12% (aida03), c. 11% (aida06), c. 9% (aida02 & aida13), c. 6% (aida05), all others <2%
no timewarps
HEC data item rate 4.2kHz
15.40 DSSSD bias & leakage current - Grafana - attachments 36-37
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41
ucesb - attachment 42
15.57 all histograms zero'd
|
|
Wed Apr 24 21:39:51 2024, TD, 00:00-08:00 Thursday 25 April 12x
|
22.43 24.4.24
DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 rate spectra - attachments 4
Merger etc - attachment 5
analysis of data file R18_266 - attachment 6
max dead time 12% (aida04)
no timewarps
HEC data item rate 1.7kHz
03.37 DSSSD bias & leakage current OK - attachments 7-8
FEE64 temperatures OK - attachment 9
ADC data item stats - attachment 10
per FEE64 rate spectra - attachments 11
Merger etc - attachment 12 |
|
Wed Apr 24 18:44:05 2024, PP, 19:30 checks 8x
|
All seems smooth.
Screenshots attached. |
|
Wed Apr 24 15:37:59 2024, PP, 16:00-00:00 shift Wednesday 24 April 8x
|
All seems OK.
Screenshots attached. |
|
Wed Apr 24 09:04:55 2024, TD, 08:00-16:00 Wednesday 24 April 33x
|
09.56 DAQ continues data file S100_alpha/R18_122
FRS setting 162Eu
DSSSD bias & leakage current OK - attachments 1-2
FEE64 temperatures OK - attachment 3
aida02 ASIC temp - known fault
All system wide checks OK *except* as shown - attachments 4-6
WR timestamps OK - attachment 7
ASIC settings aida01 & aida02 - attachments 8-9
ADC, DISC, PAUSE, RESUME, WR & Correlation Scaler data time stats - attachments 10-15
per FEE64 Rate spectra - attachments 16-18
unusually can observe HEC Rates esp. in DSSSD #1
per p+n FEE64 1.8.L spectra - attachments 19-21
aida09 pulser peak width 61 ch FWHM
common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths
per n+n FEE64 1.8.L spectra - attachment 22
per FEE64 1.8.H spectra - attachments 23-24
per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?
Merger, TapeServer etc - attachments 27-28
ucesb - attachments 29-30
spill cycle reported as 3.5s, extraction 1.5s
10.36 All histograms & stats zero'd
BNC PB-5 settings - attachment 31
DSSSD bias & leakage current - Grafana Alerting - attachment 32
can observe bias & leakage current in greater detail
DSSSD #1 bias stable, c. 10s nA micro structure observed in leakage current
DSSSD #2 bias c. 10mV variations, micro structure observed in leakage current but smaller variations cf. DSSSD#1
10.46 analysis of data file R18_131
max dead time 12% aida04, 6% aida02, all others 0.1% or less
no timewarps
HEC data item rates 1.6kHz - correcting for spill cycle => 2.3kHz in spill (online analysis shows no HEC events observed outside spill)
assume typical HEC event multiplicity = 3 => implant rate ~ 760Hz in spill or 2700 implants per spill
SC42 L/R currently report c. 2kHz
|
|
Tue Apr 23 22:37:46 2024, TD, 00:00-08:00 Wednesday 24 April 18x
|
23.34 23.4.24
DSSSD bias & leakage current OK - attachment 2
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 rate spectra - attachments 4
Merger etc - attachment 5
03.41 fDSSSD bias & leakage current OK - attachment 6
FEE64 temperatures OK - attachment 7
ADC data item stats - attachment 8
per FEE64 rate spectra (off and on spill) - attachments 9-10
Merger etc - attachment 11
03.54 analysis data file R18_67 - attachment 12
max deadtime 11% (aida04)
HEC data item rate c. 1.5kHz
no timewarps
0.35 fDSSSD bias & leakage current OK - attachment 13
FEE64 temperatures OK - attachment 14
ADC data item stats - attachment 15
per FEE64 rate spectra - attachments 16
Merger etc - attachment 17
03.54 analysis data file R18_96 - attachment 18
max deadtime 12% (aida04)
HEC data item rate c. 1.6kHz
no timewarps |
|
Tue Apr 23 19:23:42 2024, PP, 20:30 checks 8x
|
All looks good.
Screenshots attached. |
|
Tue Apr 23 15:11:50 2024, TD, Tuesday 23 April 16:00-00:00 20x
|
16.12 Analysis of data file R17_101 - attachment 1
max deadtime c. 2% (aida02)
no timewarps observed
16.15 DSSSD bias & leakage current OK - attachment 2
FEE64 temperatures OK - attachment 7
ADC data item stats - attachment 3
per FEE64 rate spectra (on and off spill) - attachments 4-5
Merger etc - attachment 6
16.53 aida04 stops producing data
DAQ STOP fails for aida04
Restart Merger to enable aida04 to complete DAQ STOP
aida04 rebooted and DAQ restarted OK
data file S100_alpha/R18 |
|
Tue Apr 23 13:15:43 2024, PP, 14:00 checks 8x
|
All looks good.
Screenshots attached |
|
Tue Apr 23 13:08:28 2024, TD, Summary of AIDA FEE64 stops
|
We have observed a number of AIDA FEE64s stop producing data.
Summary of dates & (approximate) times
aida03 21.4.24 08.20
aida04 22.4.24 09.14 https://elog.ph.ed.ac.uk/DESPEC/586
aida01 22.4.24 16.00
aida02 22.4.24 22.00
aida02 23.4.24 10.21
aida02 23.4.24 11.25
aida04 23.4.24 16.53? attachment 2
aida04/aida05 25.4.24 22.28 no messages in aida04 system console
aida04 27.8.24 c.23:00
aida04 28.8.24 c. 12-13:00 power cycle
aida04 28.8.24 c. 16.00 reset per Elog 618
aida02 28.4.24 c. 22.20 reset per Elog 618, no error messages in system console
aida04 29.4.24 01.15
aida04 30.4.24 <03.30
aida02 30.4.24 <10.00
aida02 2.5.24 c. 19.30 reset
In future, it would be useful to copy system console reports to the Elog to check whether we can identify common error messages etc.
ssh pi@nnrpi1
<password>
cd logs
cat /ttyUSBx
The current mapping of USB port # to AIDA FEE64s - attachment 1
aida02 system console c. 19.30 2.5.24
02:05:24/14:40:51|executing generic doStop
02:05:24/19:27:28|Halt (1)
02:05:24/19:27:29|Halt (2)
02:05:24/19:27:29|Halt (3)
02:05:24/19:27:30|Halt (4)
02:05:24/19:27:30|Halt (5)
02:05:24/19:27:31|Halt (6)
02:05:24/19:27:31|Halt (7)
02:05:24/19:27:32|Halt (8)
02:05:24/19:27:32|Halt (9)
02:05:24/19:27:33|Halt (10)
02:05:24/19:27:33|Halt (11)
02:05:24/19:27:34|Halt (12)
02:05:24/19:27:34|Halt (13)
02:05:24/19:27:35|Halt (14)
02:05:24/19:27:35|Halt (15)
02:05:24/19:27:36|Halt (16)
02:05:24/19:27:36|Halt (17)
02:05:24/19:27:37|Halt (18)
02:05:24/19:27:38|Halt (19)
02:05:24/19:27:38|Action has not completed
02:05:24/19:27:39|Giving up waiting
02:05:24/19:27:39|disconnect xfer stream #0 (1)
02:05:24/19:27:39|disconnect xfer stream #0 (2)
02:05:24/19:27:39|disconnect xfer stream #0 (3)
02:05:24/19:27:40|disconnect xfer stream #0 (4)
02:05:24/19:27:40|disconnect xfer stream #0 (5)
02:05:24/19:27:41|disconnect xfer stream #0 (6)
02:05:24/19:27:41|disconnect xfer stream #0 (7)
02:05:24/19:27:42|disconnect xfer stream #0 (8)
02:05:24/19:27:42|disconnect xfer stream #0 (9)
02:05:24/19:27:43|disconnect xfer stream #0 (10)
02:05:24/19:27:43|disconnect xfer stream #0 (11)
02:05:24/19:27:44|disconnect xfer stream #0 (12)
02:05:24/19:27:44|disconnect xfer stream #0 (13)
02:05:24/19:27:45|disconnect xfer stream #0 (14)
02:05:24/19:27:46|disconnect xfer stream #0 (15)
02:05:24/19:27:46|disconnect xfer stream #0 (16)
02:05:24/19:27:47|disconnect xfer stream #0 (17)
02:05:24/19:27:47|disconnect xfer stream #0 (18)
02:05:24/19:27:48|disconnect xfer stream #0 (19)
02:05:24/19:27:48|Action has not completed
02:05:24/19:27:49|Giving up waiting
02:05:24/19:27:49|executing generic Stop_Merger
02:05:24/19:27:49|executing generic Stop_DataStorageServer
02:05:24/19:27:49|completed generic doStop
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:28:01|Transfer Error - : Connection reset by peer
02:05:24/19:28:08|send() failed:
02:05:24/19:28:08|Aida state now Stopped. AidaExecV10.0_Jul 6 2022: Build 15:10:57
02:05:24/19:28:08|do_GetState returned z=0 and 0
02:05:24/19:28:13|executing generic doGo
02:05:24/19:28:45|Initialising communication with storage server 192.168.11.99 on port 11002
02:05:24/19:28:45|Setting Transfer Block Size 65536
02:05:24/19:28:45|Setting Transfer Mode 3
02:05:24/19:28:45|Setting Overlap Mode 0
02:05:24/19:28:45|Setting Blocking Mode 0
02:05:24/19:28:45|Setting nice 0
02:05:24/19:28:45|TCP transfer library version 3.12
02:05:24/19:28:45|TCP socket send buffer was 16384 - now 221184
02:05:24/19:28:45|TCP socket receive buffer was 87380 - now 221184
02:05:24/19:28:45|TCP socket created OK - now connecting to 192.168.11.99 port 11002
02:05:24/19:28:45|Connected to 192.168.11.99 port 11002
02:05:24/19:28:45|connect xfer stream #0 (1)
02:05:24/19:28:45|Data Acquisition Statistics counters now cleared
02:05:24/19:28:46|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
02:05:24/19:28:46|Aida state now Going. AidaExecV10.0: Build Jul 6 2022_15:10:57
02:05:24/19:28:46|Go (1)
02:05:24/19:28:46|Running Go_Electronics for aida02 with TSMaster aida01
02:05:24/19:28:46|ADCs Calibrated.
02:05:24/19:28:46|Finished Go_Electronics for AIDA
02:05:24/19:28:46|executing generic Go_Merger
02:05:24/19:28:46|executing generic Go_DataStorageServer
02:05:24/19:28:46|completed generic doGo
02:05:24/19:28:46|do_GetState returned z=0 and 1
02:05:24/19:28:46|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
02:05:24/19:28:51|Also in RDOGo_Operate: Enabled waveform readout
02:05:24/19:28:51|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0201
02:05:24/19:28:51|WV:0:Buffer info. Length = 64 : Offset = 0
02:05:24/19:28:51|WV:1:Buffer info. Length = 62 : Offset = 2
02:05:24/19:28:51|WV:2:Buffer info. Length = 60 : Offset = 4
02:05:24/19:28:52|WV:3:Buffer info. Length = 58 : Offset = 6
02:05:24/19:28:52|WV:4:Buffer info. Length = 56 : Offset = 8
02:05:24/19:28:52|WV:5:Buffer info. Length = 54 : Offset = 10
02:05:24/19:28:52|WV:6:Buffer info. Length = 52 : Offset = 12
02:05:24/19:28:52|WV:7:Buffer info. Length = 50 : Offset = 14
02:05:24/19:28:52|WV:8:Buffer info. Length = 48 : Offset = 16
02:05:24/19:28:52|WV:9:Buffer info. Length = 46 : Offset = 18
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 1048, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52| |
|
Tue Apr 23 11:09:00 2024, PP, 12:00 checks 8x
|
Everything is OK now.
Screenshots attached. |
|
Tue Apr 23 10:52:39 2024, PP, Aida02 down and back
|
We lost aida02 at about 11:25 again. Nick fixed it about 11:50.
It is now back in the DAQ. |
|
Tue Apr 23 10:10:31 2024, PP, aida02 is back
|
Nick rebooted AIDA and aida02 came back at around 10:55.
AIDA was out of the DAQ from about 10:30 until 10:55 |
|
Tue Apr 23 09:21:01 2024, PP, 10:20 checks - aida02 is down 8x
|
Aida02 is down. Contacted Nick.
The rest looks stable, see screenshots.
10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14
Go > All 16 writing to merger, looks OK
aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again |
|
Tue Apr 23 07:27:18 2024, PP, 08:00-16:00 Tuesday 24 April shift 8x
|
Checks 08:00-08:30
Screenshots are attached and everything looks OK. |
|
Mon Apr 22 22:35:59 2024, TD, 00.00-08.00 Tuesday 22 April 26x
|
23.36 Analysis of data file R17_34 - attachment 1
max deadtime c. 1% (aida02)
no timewarps
23.38 DSSSD bias & leakage current OK - attachment 2
FEE64 temperatures OK - attachment 3
aida02 ASIC temp - known fault
ADC data item stats - attachment 4
per FEE64 Rate spectra - attachment 5
Merger etc - attachment 6
01.03 DSSSD bias & leakage current OK - attachment 7
FEE64 temperatures OK - attachment 8
aida02 ASIC temp - known fault
ADC data item stats - attachment 9
per FEE64 Rate spectra - attachment 10
Merger etc - attachment 11
03.06 DSSSD bias & leakage current OK - attachment 12
FEE64 temperatures OK - attachment 13
aida02 ASIC temp - known fault
ADC data item stats - attachment 14
per FEE64 Rate spectra - attachment 15
Merger etc - attachment 16
05.04 DSSSD bias & leakage current OK - attachment 17
FEE64 temperatures OK - attachment 18
aida02 ASIC temp - known fault
ADC data item stats - attachment 19
per FEE64 Rate spectra - attachment 20
Merger etc - attachment 21
07.29 DSSSD bias & leakage current OK - attachment 22
FEE64 temperatures OK - attachment 23
aida02 ASIC temp - known fault
ADC data item stats - attachment 24
per FEE64 Rate spectra - attachment 25
Merger etc - attachment 26 |
|
Mon Apr 22 15:17:07 2024, DSJ, 16.00-0.00 22/04/24 26x
|
First checks of shift. All looks ok.
Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.
Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)
checks at 16.00 - merger has crashed - aida01 dropped out - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09
Reset 01, restarted DAQ, and restarted as R16.
16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.
Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms
RUN17 STARTED 16.51|
HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.
20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift
21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps
22.48 - things apear ok. (Attachments 23-26) |