| ID |
Date |
Author |
Subject |
|
267
|
Fri Apr 23 15:13:28 2021 |
DSJ | 23 April 16.00-20.00 shift | 16:00 taking over from Muneerah
16:15 - temp, ucesb, database, stats etc. look ok.
16:41 - Nic restarting MBS
17:05 - temp, ucesb, database, stats etc. look ok.
17:43 - temp, ucesb, database, stats etc. look ok.
18.10 - system wide checks. All same as previous checks - eg elog 264 + 266
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Check FPGA Timestamp Error - HTML errors
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 7 4 7 4 3 2 4 2 3 6 : 36388
aida02 : 1 2 5 2 0 4 3 4 3 4 3 : 27044
aida03 : 28 4 4 4 4 4 2 2 3 3 6 : 36432
aida04 : 5 5 2 6 2 3 3 3 3 3 6 : 36892
aida05 : 24 4 7 3 3 3 2 2 3 3 6 : 36240
aida06 : 21 6 6 3 3 4 2 2 3 3 6 : 36356
aida07 : 15 7 4 3 3 4 2 2 3 3 6 : 36308
aida08 : 23 6 7 1 3 4 1 3 3 3 6 : 36572
aida09 : 1 8 2 1 1 2 4 2 3 3 6 : 36292
aida10 : 3 4 6 2 2 2 3 3 2 3 6 : 35660
aida11 : 15 7 4 1 2 3 2 4 2 3 6 : 36052
aida12 : 25 13 5 2 2 3 2 3 3 3 6 : 36700
18:45 - temp, ucesb, database, stats etc. look ok.
19:03 - temp, ucesb, database, stats etc. look ok.
19:26 - temp, ucesb, database, stats etc. look ok.
TapeData (df -h) - 7.2T 3.9T 2.9T 58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
19:30 - beam being shared with R3B, implantation rates fluctuation by~ 50% back to normal 19:40
19:50 system checks - all as previous
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Check FPGA Timestamp Error - HTML errors
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 25 6 4 7 4 3 2 4 2 3 6 : 36404
aida02 : 20 7 5 1 0 4 2 3 2 3 4 : 27384
aida03 : 28 6 3 5 3 5 2 2 3 3 6 : 36528
aida04 : 19 8 4 6 2 4 4 2 3 3 6 : 36876
aida05 : 7 3 8 3 4 3 1 3 3 3 6 : 36500
aida06 : 25 8 4 3 3 4 2 3 3 3 6 : 36868
aida07 : 18 7 3 4 3 4 2 2 3 3 6 : 36336
aida08 : 16 10 6 2 3 3 1 3 3 3 6 : 36464
aida09 : 0 3 1 1 1 3 4 2 3 3 6 : 36360
aida10 : 8 1 10 0 2 2 2 4 2 3 6 : 35912
aida11 : 2 2 2 3 3 4 3 3 2 3 6 : 35928
aida12 : 12 7 4 1 3 3 2 3 3 3 6 : 36616
|
|
322
|
Mon May 17 23:17:16 2021 |
DSJ | Tuesday 18 May 0.00-8.00 shift | 0.00 - Taking over from Lewis
00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2)
01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4)
01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6)
02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8)
Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11)
System Wide Checks:
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 0, Failed 16
Base Current Difference
aida05 fault 0x4da : 0x4ee : 20
White Rabbit error counter test result: Passed 15, Failed 1
FPGA Timestamp error counter test result: Passed 16, Failed 0
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 43 12 6 4 1 4 2 2 1 3 7 : 38444
aida02 : 21 12 4 4 2 3 3 3 3 3 6 : 36980
aida03 : 23 8 4 2 2 4 2 3 3 3 6 : 36764
aida04 : 34 12 10 5 1 4 1 4 3 3 6 : 37224
aida05 : 18 5 5 1 2 3 2 2 1 3 7 : 38112
aida06 : 40 12 5 7 2 4 2 4 3 3 6 : 37552
aida07 : 40 10 6 2 2 4 2 2 2 2 7 : 37392
aida08 : 19 11 3 3 1 4 3 3 3 3 6 : 36980
aida09 : 43 6 2 4 3 4 3 3 3 3 6 : 37180
aida10 : 32 14 10 3 4 1 2 4 1 3 7 : 39280
aida11 : 40 7 2 5 4 3 1 4 3 3 6 : 37144
aida12 : 37 14 13 2 2 3 2 4 3 3 6 : 37396
aida13 : 18 9 3 1 1 5 3 3 3 3 6 : 37024
aida14 : 36 13 6 4 2 3 2 3 1 3 7 : 38872
aida15 : 32 11 7 4 3 4 2 3 2 4 6 : 38024
aida16 : 26 9 4 1 0 3 1 4 3 3 6 : 36752
02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13)
03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15)
03.05 - No beam, problem with SIS, don't know how long will be off
03.14 - Beam is back
03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17)
04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19)
Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22)
System Wide Checks:
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 0, Failed 16
Base Current Difference
aida05 fault 0x4da : 0x4f2 : 24
White Rabbit error counter test result: Passed 15, Failed 1
FPGA Timestamp error counter test result: Passed 16, Failed 0
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 11 5 6 1 4 2 2 1 3 7 : 38416
aida02 : 16 9 5 4 2 4 2 2 3 3 6 : 36312
aida03 : 9 7 4 4 2 4 2 3 3 3 6 : 36764
aida04 : 36 11 8 4 3 4 1 3 4 3 6 : 37800
aida05 : 20 4 3 2 2 3 2 2 1 3 7 : 38112
aida06 : 22 13 10 7 2 4 3 3 4 3 6 : 38336
aida07 : 28 7 7 2 2 4 2 2 2 2 7 : 37336
aida08 : 2 3 4 4 1 3 2 3 3 3 6 : 36512
aida09 : 40 7 1 4 4 3 3 3 3 3 6 : 37096
aida10 : 26 9 12 2 3 2 2 4 1 3 7 : 39280
aida11 : 30 7 1 5 4 3 1 3 2 4 6 : 37600
aida12 : 15 10 6 4 2 3 2 4 3 3 6 : 37228
aida13 : 1 3 2 1 2 4 3 2 3 3 6 : 36316
aida14 : 34 12 7 4 2 3 2 3 1 3 7 : 38872
aida15 : 18 11 5 3 4 4 2 3 2 4 6 : 37968
aida16 : 22 7 4 2 0 3 1 4 3 3 6 : 36752
04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24)
05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26)
05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28)
06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30)
Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33)
System Wide Checks:
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 0, Failed 16
Base Current Difference
aida05 fault 0x4da : 0x4f9 : 31
White Rabbit error counter test result: Passed 15, Failed 1
FPGA Timestamp error counter test result: Passed 16, Failed 0
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 9 3 7 1 5 1 3 1 3 7 : 38744
aida02 : 18 3 5 2 3 3 3 2 3 3 6 : 36400
aida03 : 21 9 4 2 2 4 2 4 3 3 6 : 37276
aida04 : 32 12 7 3 3 4 1 3 4 3 6 : 37744
aida05 : 12 8 3 2 2 4 2 2 1 3 7 : 38240
aida06 : 28 14 4 8 2 4 3 3 3 3 6 : 37280
aida07 : 40 12 5 2 2 4 1 3 2 2 7 : 37648
aida08 : 13 8 4 4 1 3 2 3 3 3 6 : 36596
aida09 : 35 5 2 4 3 4 2 4 3 3 6 : 37396
aida10 : 6 10 4 1 1 3 2 2 1 3 7 : 38024
aida11 : 41 9 1 4 4 3 2 4 3 3 6 : 37372
aida12 : 29 11 9 4 2 3 2 4 3 3 6 : 37340
aida13 : 24 10 5 1 2 4 3 2 3 3 6 : 36512
aida14 : 34 16 5 4 2 3 2 3 1 3 7 : 38872
aida15 : 42 8 6 2 4 4 2 3 2 4 6 : 38024
aida16 : 8 9 6 1 0 3 1 4 3 3 6 : 36712
06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35)
07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37)
07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39) |
|
588
|
Mon Apr 22 15:17:07 2024 |
DSJ | 16.00-0.00 22/04/24 | First checks of shift. All looks ok.
Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.
Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)
checks at 16.00 - merger has crashed - aida01 dropped out - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09
Reset 01, restarted DAQ, and restarted as R16.
16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.
Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms
RUN17 STARTED 16.51|
HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.
20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift
21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps
22.48 - things apear ok. (Attachments 23-26) |
|
151
|
Sat Mar 14 07:13:59 2020 |
DK, TD, LS, CG | Saturday 14th March 08:00 - 24:00 | 8:15
All system wide checks passed
Leak currents are continuing to creep up - attachment #1
FEE temperatures fine - attachment #2
Stats about where they have been - attachment #3
Implants look as previously (5ish counts per channel in SSD1 and SSD2) - attachment #4
Merger fine, 3 Mega items / sec
TapeServer still writing about 25 MB/s
09.37 analysis of data file S480/R12_443
deadtime aida09 c. 5%, all other FEE64s <<1%
See attachment #5
09.54.51
biases, SSD1 and 2 over 2 uA each now - attachment #6
11:20
All system-wide checks okay
Merger okay, 3 Mega items / sec
TapeServer normal, 27 MB/sec
Biases as attachment #7
Stats as #8
Temps as #9
Histos as #10
14:10
All system wide checks okay
Stats #11
Temps #12
Implants #13
Leak currents are getting a bit higher #14
15:30
All system wide checks passed
Stats attachment #15
Temps attachment #16
Implants attachment #17
Biases / leak currents attachment #18
Merger okay, 3 Mega items / sec
TapeServer okay, writing 27 MB/s
16.12 all online spectra zero'd
16.31 stat & rate spectra - attachments 19-22
1.8.L spectra - attachments 23 & 24
aida03 pulser peak 57 ch FWHM
1.8.H spectra - attachments 25 & 26
NewMerger & TapeServer - attachments 27 & 28
18.29
system wide checks all okay
FEE temperatures normal (attachment 29)
Leakage currents okay and recorded to spreadsheet (attachment 30)
Good event statistics okay (attachment 31)
Merger running at 3M items/sec
Tape server running at 25MB/sec
20.25
system wide checks all okay
FEE temperatures normal (attachment 32)
Leakage currents okay and recorded to spreadsheet (attachment 33)
Good event statistics okay (attachment 34)
Merger running at 3M items/sec
Tape server running at 26MB/sec
as of 20.39 just finished R12_961
21.36 beam is down confirmed by rates histogram (attachment 35), informed others on shift
was down for maybe 5 minutes is back on at 21.40 (attachment 36)
think between R12_1004 to R12_1008
22.10
system wide checks all okay
FEE temperatures normal (attachment 37)
Leakage currents okay and recorded to spreadsheet (attachment 38)
Good event statistics okay (attachment 39)
Merger running at 3M items/sec
Tape server running at 26MB/sec
23.03 someone comes to ask if we see an increase in implantation rate as they do their side
I checked the rates histogram and good events and see nothing that would indicate an increase (attachment 40 and 41)
FEE temperatures and leakage currents do not seem any different either
believe they are calling someone about the issue
23.30 leaving now to catch bus but people on night shift have been told how to check on AIDA and know that CG is on call if needed |
|
105
|
Sat Dec 7 02:39:02 2019 |
DK, NH et al. | 34Si Fragments PID | See attached
Tentative PID (DK) as attachment #3
DESPEC LISE++ file as attachment #4 |
|
140
|
Tue Mar 10 23:17:31 2020 |
DK, LS, PW, TD, CA | Wednesday 11 March 2020 | 00:15 Shift change near AIDA R6_475
We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.
All system-wide checks passed successfully
Temperatures ok (aida01 still a bit hot) - see attachment #1
Stats as normal (SSD3 still noisy) - see attachment #2
Bias and pulser settings remain sane - see attachment #3
Writing data to disk at ~26 MB/s as yesterday - see attachment #4
Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
Rates look not unreasonable for expected implantation in short time window - see attachment #6
2:26
We are near R6_574
All system checks are fine
Temps are fine - attachment #7
Stats look as before - attachment #8
Biases and leaks typical - attachment #9
Merger fine, 3 x 10^6 items / sec (skip screencap)
TapeServer still sits near 25 MB/s (skip screencap)
Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10
2:44 R6_587 or R6_588
The spills seem to have stopped based on the beeping scalars. Confirmed with other group there are no scint counts.
See attachment of AIDA rates where no implants can be seen. See attachment #11
2:46 Spill beeps heard briefly, but they cut again. R6_590
2:49 Beam seems to be back R6_592
Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants. No problem. See attachment #12 for implants.
4:35
System checks all pass (except expected error on master clock)
Temps okay - attachment #13
Biases and leak currents normal - attachment #14
Stats as before - attachment #15
Merger going, 3 mega items per sec
TapeServer going, 25 MB/s
5:59 Beam is gone again, near R6_738. As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
See attachment #16 which shows no implants in the rate.
Contact phones for Control Room: 2222
We called, they are aware of a problem with the magnets and they are working to resolve it. So we will await them.
Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.
We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off
7:14 Beam is still off, but I make some notes
System wide checks all okay
Temps as normal (see attachment #17)
Biases and leak currents okay (see attachment #18)
Stats as normal (see attachment #19)
Merger going at 3 mega cps
TapeServer on, around 25 MB/sec
Implants not seen in histogram rate as expected (see attachment #20)
7:32 We called the operators to confirm that they don't change anything before calling us first.
Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.
8.00 shift change at file R6_823
beam is back on, no settings changed.
particles per spill is similar to before (2k~3k per spill)
implants now being seen in rate histogram (attachment 21)
8.32 system wide checks all okay
FEE temperatures are normal (attachment 22)
leak currents are okay and have been recorded to sheet (attachment 23)
good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
merger rate is running around 3M items/sec
tape service rate is running around 26MB/sec
9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
remaking each histogram and saving layout6 seems to be fixed (attachment 27)
Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
invoked from within
"SpecDetails 1.8.h"
(in namespace inscope "::urn:SpectrumService" script line 1)
invoked from within
====
10.05 beam is being stopped to change some thresholds
10.34 system wide checks all okay
FEE temperatures are normal (attachment 28)
leak currents are okay and have been recorded to spreadsheet (attachment 29)
good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 24MB/sec
implants not seen in histogram rate as expected as beam still down (see attachment 30)
10.42 beam still down but taking data, currently file finishing R6_946
as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
have zeroed all histograms so will take screenshots at 11.00
11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks
11.25 beam back file S480/R6_975
12.00 analysis file S480/R6_1001 - attachment 33
FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
all HEC data rate c. 1kHz
all ADC data rate c. 1.2MHz
12.23 system wide checks all okay
FEE temperatures are normal (attachment 34)
leak currents are okay and have been recorded to spreadsheet (attachment 35)
good event stats are okay (attachment 36)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 26MB/sec
13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
seems to be intermittent. file R6_1051
13.10 beam back on, no one was called. currently on file R6_1061
13.22 beam off as going to higher intensity beam. file R6_1069
14.42 system wide checks all okay
FEE temperatures are normal (attachment 38)
leak currents are okay and have been recorded to spreadsheet (attachment 39)
good event stats are okay (attachment 40)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 24M/sec
14.48 on file R6_1131
16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
file R6_1193
16:35 compression of files R6_[200-999] commenced (nice -20)
16:40 system wide checks all okay
FEE temperatures are normal (attachment 41)
leak currents are okay and have been recorded to spreadsheet (attachment 42)
good event stats are okay (attachment 43) |
|
147
|
Thu Mar 12 23:44:48 2020 |
DK | PID Plots | Labelled PID plots. Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.
There was an error in the printouts for Cd isotopes, but the below attachments are corrected.
The Z-depth (color, not nuclear charge) is log scale. |
|
148
|
Fri Mar 13 08:01:53 2020 |
DK | Friday 13th 8:00 to 16:00 shift | 8:00 - See previous shift's elog entry. All good.
9:00
All system wide checks passed
Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
Tape Server okay - 27 MB/s
Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
Stats as usual - attachment #2
Temps normal - attachment #3
Biases and leaks fine - attachment #4
Disk usage projection
Space remaining: ~2 TB
Fill rate: ~2 GB / 1 minute
-> 1000 minutes =~ 17 hours
Still gzipping from R6_823
10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
Kbytes Written: -1986382400 Kbytes/sec: 26816
Rate is fixed at that value on update.
However, actual data files are being updated so data has been coming.
DK and NH consulted and believe the situation is fine.
We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.
10:25 Now we are on R10 after start and stop the TapeServer. Problem is resolved.
11:30
Implants: attachment #5
Stats: attachment #6
Biases / leaks: attachment #7
Temps: attachment #8
System wide checks performed, situation normal. |
|
236
|
Sat Apr 17 19:39:56 2021 |
DJ, TD | Saturday 17 April 20.00-00.00 |
19.37 per FEE64 rate spectra - attachments 1 & 2
1.8.L spectra - attachments 3 & 4
1.8.H spectra - attachments 5-8
1.8.W spectra - attachments 9 & 10
20.42 DAQ contrinues file NULL/R30_233
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bd7 : 10
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x4 : 4
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
-------
23:48
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bde : 17
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x7 : 7
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 5 6 2 1 3 1 3 3 3 6 : 36336
aida02 : 7 9 10 3 2 2 2 4 2 3 6 : 36068
aida03 : 26 7 8 4 3 4 2 4 2 3 6 : 36448
aida04 : 18 8 2 1 2 2 3 4 2 3 6 : 36168
aida05 : 24 7 9 4 3 1 3 2 2 4 6 : 37352
aida06 : 17 10 3 1 3 5 1 3 3 3 6 : 36644
aida07 : 7 11 2 2 3 2 3 2 4 3 6 : 37268
aida08 : 23 4 1 5 1 1 2 3 3 3 6 : 36332
aida09 : 14 6 3 2 1 3 2 3 3 3 6 : 36504
aida10 : 2 2 2 2 1 2 2 2 3 3 6 : 35768
aida11 : 2 2 1 2 2 2 2 4 2 3 6 : 35816
aida12 : 26 8 5 5 4 3 1 3 3 3 6 : 36632
---- |
|
238
|
Sun Apr 18 07:30:20 2021 |
DJ, TD | Sunday 18th April 08:00-12:00 | ---- 09-26
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
---
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bfc : 47
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
--
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bfc : 47
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
-
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bfc : 47
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
08.45 TD resets baseline for WR and FPGA errors
|
|
263
|
Thu Apr 22 23:17:46 2021 |
DJ | Friday 23rd April 00:00-08:00 | Took over from TD ... All is well.
00:45 System Wide Checks
Clock: 12 passed, 0 failed.
ADC Calibration: Passed 12, 0 failed.
White Rabbit Decoder: Passed 4,failed 8. - See screenshot.
FPGA Passed 11, failed 1. - See screenshot.
FEE64 Linux: See screenshot.
-----
02:49 run closed. Checking beam.
03:00 File 55 opened.
03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.
File 56 opened at 4:24.
---------
04:30 System Wide Checks (See screenshots)
White Rabbit Decoder: Passed 12, failed 0.
FPGA: page not loading properly.
----
05:37 System wide checks. See screenshots
Clock: Status: 11 passed. 1 fail.
ADC calibration: Passed 9, failed 3.
White Rabbit decoder: Passed 12, failed 0.
-----
07:32 System wide checks (See screenshots).
|
|
695
|
Fri Jan 31 09:37:47 2025 |
CC, TD, MP | Friday 31 January | Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000
N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.
n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted
BB7 aida10 (asics #1-2 p+n, #3-4 n+n)
BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream)
10.30 FEE64 power ON
DSSSD bias ON (BB18 only) - attachment 1
leakage current OK
ASIC settings 2024Dec13-17.02.45 restored
p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf
Attachments 2-7 WR timestamp aida05 & aida12, system wide check
aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
WR decoder status aida02
per FEE64 Rate spectra - attachment 8
aida10 high rate - BB7 not biased
rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments
ADC data item stats - attachment 9
rates generally OK
aida10 high - BB7 not biased
aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!
BNC PB-5 settings - attachment 10
WR timestamps OK - attachment 11
FEE64 temperatures OK - attachment 12
12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet
ADC data item stats - attachment 13
all rates higher cf. attachment 9
per FEE64 Rate spectra - attachment 14
per p+n FEE64 1.8.L spectra - attachment 15
aida09 pulser peak width 69 ch FWHM = 49keV FWHM
aida14 pulser peak width 50 ch FWHM = 35keV FWHM
per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17
per n+n FEE64 1.8W spectra - 20us FSR - attachment 18
DSSSD bias & leakage current OK - attachment 19
ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%
FEE64 temperatures OK - attachment 20
DSSSD bias volatge & leakage current OK - attachment 21
CAEN N1419ET ch#0 BB18, ch#1 BB7
Install of bPlas drivers, cabling, grounds complete
ADC data item stats - attachment 22
per FEE64 Rate spectra - attachment 23
all BB18 p+n FEE64s *except* aida05 show very low rates of noise
per p+n FEE64 1.8.L spectra - attachment 24
aida09 pulser peak width 62 ch FWHM = 42keV FWHM
BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD
per FEE64 1.8.W spectra - 20us FSR - attachments 25-26 |
|
697
|
Fri Jan 31 17:26:47 2025 |
CC, TD, MP | Friday 31 January contd. |
18.18 bPlas and BB7 installs complete
Restart AIDA DAQ, Merger, Tape Server and re-test
per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 69 ch FWHM = 42keV FWHM
per FEE64 Rate Spectra - attachment 2
BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK
per FEE64 1.8.W spectra - 20us FSR - attachments 3-5
ADC data item stats - attachment 6
aida01 aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
BB7 aida10 19k
WR timestamps OK - attachment 7
Merger, TapeSever etc - attachments 8-9
disk directpry /TapeData/FEB25
working OK
DSSSD bias & leakage current OK - attachment 10
System wide checks OK - attachment 11-15
note global clock status 6 errors reported earlier today have now gone
FEE64 temperatures OK - attachment 16
18.38 Transition to safe state
DAQ STOP
disable data transfer 1
detector bias OFF
FEE64 power OFF
Can restart as follows
1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps
If all OK can re-connect to Merger/TapeServer as follows
1) DAQ STOP
2) enable data transfer 1
3) DAQ GO
Can disconnect from Merger/TapeServer as follows
1) DAQ STOP
2) disable data transfer 1
|
|
704
|
Fri Jun 13 12:42:39 2025 |
CC, TD | Friday 13 June 2025 |
13.42 ASIC settings 2025Jun12-13.53.04
*all* FEE64s slow comparator 0x64
all fast disc outputs disabled
p+n FEE64s waveform threshold 7000, n+n FEE64s 9000
CC grounding ribbon cable heavy duty braid to adaptor PCB and AIDA support assembly ground
'top' p+n FEE64s and snout to be completed
16.00 DSSSD bias & leakage current - attachment 1
FEE64 temps OK - attachment 2
All system wide checks OK *except* WR decoder status aida02 and aida06 - attachment 3
WR timestamps OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
per p+n FEE64 1.8.L spectra - attachment 7
aida09 pulser peak width 117 ch FWHM - no change since yesterday
per FEE64 1.8.W spectra - 20us FSR - attachments 8-10
|
|
709
|
Tue Jul 22 18:00:45 2025 |
CC, NK, MP | FWHM aida FEE | AIDA05 completely disconnected, all the others FEEs64 are fully connected
Attachment 1: aida05, FWHM= 95 ch, 63 keV
Attachment 2: aida15, FWHM= 108 ch, 74 keV |
|
716
|
Thu Jul 24 10:28:04 2025 |
CC, NK, MP | Comparison of channels in the same FEE64 and ASIC 05.1 and with AIDA13.1.8L | BB7 is still connected but not biased (same configuration as the previous elog). The channel of the CAEN power supply #0 is connected to ASIC08 and biased at 100V. The channel of the CAEN power supply #1 is disconnected but still biased at 100V. AIDA04 (chained to AIDA05) is biased via an unshielded cable to an iseg NHQ 246L power supplied, biased at 100 V with 15 uA.
All the outside drain wires of bplast are now taped to the snout.
The red markers of the cursors of the previous test have been left for ease of comparison.
Attachment 1: AIDA05.1.8L --> FWHM 67 channels
Attachment 2: AIDA05.1.1L --> FWHM 69 channels
Attachment 3: AIDA05.1.4L --> FWHM 79 channels
Attachment 4: AIDA05.1.12L --> FWHM 79 channels
Attachment 5: AIDA05.1.15L --> FWHM 79 channels
Attachment 6: AIDA13.1.8L --> FWHM 236 channels, chosen because same place than AIDA05 but for the dowstream detector that is still biased with the CAEN module
Attachment 7: AIDA12.1.8L --> FWHM 91 channels, chosen because same place than AIDA05 but of the bottom of upstream detector that is still biased with the iseg module
Attachment 8: AIDA16.1.8L --> FWHM 143 channels, chosen because same place than AIDA05 of the bottom of dowstream detector that is still biased with the CAEN module. This curve has 2 humps |
|
710
|
Wed Jul 23 09:37:34 2025 |
CC, NK | Comparison pulser FWHM with/without HT and detector cables | This elog is the first reliable. Before, the histograms have not been zero'd
Each test are done and compared for AIDA05.1.8L, compared with the pulser (caracteristics, see attachment1)
Test 1: with HT cables shielded
Attachment 2: HT = 100V; cable AIDA05 connected --> FWHM = 77 channels (= 51.3 keV) with a top of the curve at 100 cts (AIDA9.1.8L has also been checked and was at 89 channels)
Attachment 3: HT = 0V; cable AIDA05 connected --> FWHM = 1757 channels with a top of the curve at 15 cts after almost 40 min because the noise had a baseline of 6500 channels
Attachment 4: HT = 0V; cable AIDA05 disconnected --> FWHM = 16 channels with a top of the curve at 110 cts
Attachment 5: HT = 100V; cable AIDA05 disconnected --> FWHM = 17 channels with a top of the curve at 110 cts
Test 2: with HT cables not shielded
Attachment 6: HT = 0V; cable AIDA05 connected --> FWHM = 1452 channels (968 keV) with a top of the curve at 23 cts after 1:15 hours because the noise had a baseline of 7000 channels
Attachment 7: HT = 100V; cable AIDA05 connected --> FWHM = 73 channels with a top of the curve at 100 cts
Attachment 8: HT = 0V; cable AIDA05 disconnected --> FWHM = 15 channels with a top of the curve at 120 cts
Attachment 9: HT = 100V; cable AIDA05 disconnected --> FWHM = 16 channels with a top of the curve at 125 cts
Conclusion: with shielded or unshielded cables, the noise situation seems to be the same, proving that mot of the noise don't come from the cables
Other thing that have been noticed is that the peak of the pulser is shifted when the cables are connected compared to when they are not connected: connected: peak at around channel 21675; disconnected: peak at around channel 21925 |
|
711
|
Wed Jul 23 13:08:30 2025 |
CC, NK | Comparison of channels in the same FEE64 and ASIC 05.1 | Biased at 100, AIDA05 connected, HT cables not shielded CAEN HT
Attachment 1: AIDA05.1.8L --> FWHM 75 channels
Attachment 2: AIDA05.1.1L --> FWHM 70 channels; peak of the pulser shifted to the right (red lines on the left: cursor for the 1.8L)
Attachment 3: AIDA05.1.4L --> FWHM 85 channels; peak of the pulser completely shifted to the left (10 000 on the left)
Attachment 4: AIDA05.1.12L --> FWHM 78 channels
Attachment 5: AIDA05.1.15L --> FWHM 79 channels, peak the most shifted to the right of the 5 curves tested |
|
712
|
Wed Jul 23 13:59:37 2025 |
CC, NK | Shielding of the snout | Attachment1: AIDA05.1.8L when AIDA05 connected, biased at 100 V with not-shielded cables, CAEN HT, after the base of the snout (screws) has been connected to the ground via cupper sheets
FWHM: 76 channels (to be compared with the 2 previous elogs 73 channels and 75 channels)
Conclusion: no differences can be observed. |
|
715
|
Wed Jul 23 17:02:22 2025 |
CC, NK | Comparison of channels in the same FEE64 and ASIC 05.1 and with AIDA13.1.8L | BB7 is still connected but not biased (same configuration as the previous elog). The channel of the CAEN power supply #0 is connected to ASIC08 and biased at 100V. The channel of the CAEN power supply #1 is disconnected but still biased at 100V. AIDA04 (chained to AIDA05) is biased via a shielded cable to an iseg NHQ 246L power supplied, biased at 101 V with 16 uA.
All the outside drain wires of bplast are now taped to the snout.
Attachment 1: AIDA05.1.8L --> FWHM 60 channels
Attachment 2: AIDA05.1.1L --> FWHM 51 channels
Attachment 3: AIDA05.1.4L --> FWHM 57 channels
Attachment 4: AIDA05.1.12L --> FWHM 65 channels
Attachment 5: AIDA05.1.15L --> FWHM 69 channels
Attachment 6: AIDA13.1.8L --> FWHM 277 channels, chosen because same place than AIDA05 but for the dowstream detector that is still biased with the CAEN module
Attachment 7: AIDA12.1.8L --> FWHM 88 channels, chosen because same place than AIDA05 but of the bottom of upstream detector that is still biased with the iseg module
Attachment 8: AIDA16.1.8L --> FWHM 133 channels, chosen because same place than AIDA05 of the bottom of dowstream detector that is still biased with the CAEN module. This curve has 2 humps |
|