 Fri Apr 23 07:09:57 2021, LJW, Friday 23rd April 08:00-12:00 9x
|
08:30
System Checks
Clock Status error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 1
Temperatures - See attachment 2
Bias & Leakage Currents - See attachment 3
10:02
System Checks:
Clock Status Error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration Error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status Error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 4
Temperatures - See attachment 5
Bias & Leakage Currents - See attachment 6
10:45
System Checks
Clock Status Error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration check Error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status Error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 7
Temperatures - See attachment 8
Bias & Leakage Currents - See attachment 9
|
|
Mon May 17 15:14:35 2021, LJW, Monday 17th May 2021 16:00-20:00 9x
|
16:25
Informed at shift handover that there has been no beam.
Statistics - See attachment 1
Temperatures - See attachment 2
Bias & Leakage currents - See attachment 3
System Wide checks - All passed, EXCEPT:
ADC Calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit decoder status:
Base Current Difference
aida05 fault 0x4da : 0x4dd : 3
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
18:00
Usecb scalars now showing DSSD implant rates.
Statistics - See attachment 4
Temperatures - See attachment 5
Bias & Leakage currents - See attachment 6
System Wide checks - All passed, EXCEPT:
ADC Calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit decoder status
Base Current Difference
aida05 fault 0x4da : 0x4e1 : 7
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx
19:45
Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.
Statistics - See attachment 7
Temperatures - See attachment 8
Bias & Leakage currents - See attachment 9
System Wide checks - All passed, EXCEPT:
ADC Calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit decoder status:
Base Current Difference
aida05 fault 0x4da : 0x4e1 : 7
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes
19:55 - Beam now back again - DSSD implants showing
|
|
Sat Apr 17 11:27:38 2021, JS, TD, Saturday 17th April 12:00-16:00 10x
|
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
13:55 CEST
Statistics : ok elog:234/4
Temp : ok elog:234/5
Bias : ok elog:234/6
ucesb : ok
DB: No faults found
ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and
Control browser page to rerun calibration for that module
White Rabbit Check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
14:05 no beam
14:10 beam back
14:15 no beam, beam current being optimised, going to thicker degrader when beam returns
14:37 CEST
Statistics : ok
Temp : ok
Bias : ok (ch3 now over 6uA)
ucesb : ok
DB: No faults found
15:03 CEST
Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found
15:31 CEST
Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found
15:50
Statistics : ok elog:234/7
Temp : ok elog:234/8
Bias : ok elog:234/9
ucesb : ok elog:234/10
DB: No faults found
ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last |
|
Tue Apr 20 19:00:00 2021, JS, Tuesday 20th 20:00-00:00 12x
|
20:00
Taking over from Liam, all systems good.
20:30
DB No faults.
Stats good
ucesb ok
20:58
DB No faults.
Stats good
Temps good
ucesb ok
21:31
DB No faults.
Stats good
Temps good
ucesb ok
22:02
DB No faults.
Stats good elog:254/3
Temps good elog:254/4
ucesb ok elog:254/1
Current ok elog:254/2
ADC Calibration check:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
22:06 ucesb - DAQ Error - Onsite team checking daq crash
Tape server icon stopped spinning
Stats Rates 0, Counter aida01 zeroed but others normal. elog:254/5
Called Tom. Tried resetting in midas, no luck. Full reboot of DAQ and midas required.
According to log at 22:07 aida01 reset
Data taking stopped at 22:04 file R45_193
22:50
After a full power cycle all appear to be up and running well. Starting full checks.
23:00
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
Check FPGA is having a page error
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k
4k
aida01 : 4 8 2 2 1 3 1 4 2 3 7 :
39792
aida02 : 13 9 2 4 2 3 2 3 2 3 7 :
39708
aida03 : 19 5 4 3 1 3 1 4 2 3 7 :
39892
aida04 : 14 5 2 2 0 5 2 3 2 3 7 :
39744
aida05 : 16 11 3 2 2 4 3 2 1 4 7 :
40584
aida06 : 15 9 5 2 2 2 1 4 2 3 7 :
39828
aida07 : 15 9 3 3 2 2 1 4 2 3 7 :
39828
aida08 : 21 4 3 4 2 2 1 4 2 3 7 :
39844
aida09 : 11 6 1 2 3 4 2 3 2 3 7 :
39788
aida10 : 0 6 4 2 0 3 2 3 2 3 7 :
39472
aida11 : 16 7 2 0 1 3 2 3 2 3 7 :
39512
aida12 : 3 7 4 1 3 2 2 3 2 3 7 :
39524
DB No faults.
Stats good elog:254/6
Temps good elog:254/7 aida01 had been high during reboot issue, >70, but now is back down
Current ok elog:254/8
ucesb ok
23:25
DB No faults.
Stats good
Temps good
ucesb ok
23:42 No Beam
23:48 Beam Back
23:57
DB No faults.
Stats good elog:254/11
Temps good elog:254/12
ucesb ok elog:254/9
Current ok elog:254/10
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
*FPGA check page error*
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 9 5 2 1 2 1 4 2 3 7 : 39812
aida02 : 13 9 5 2 2 2 1 4 2 3 7 : 39820
aida03 : 21 8 3 2 2 2 1 4 2 3 7 : 39812
aida04 : 13 4 3 1 1 5 2 3 2 3 7 : 39780
aida05 : 24 5 3 2 2 4 2 3 2 3 7 : 39800
aida06 : 16 5 5 2 2 2 1 4 2 3 7 : 39800
aida07 : 22 6 2 2 1 3 1 4 2 3 7 : 39848
aida08 : 21 8 2 3 2 2 1 4 2 3 7 : 39828
aida09 : 18 7 1 1 3 4 2 3 2 3 7 : 39792
aida10 : 19 3 3 1 0 3 2 3 2 3 7 : 39476
aida11 : 17 4 4 0 1 3 2 3 2 3 7 : 39524
aida12 : 13 4 3 2 3 1 2 3 2 3 7 : 39428
00:04 Handing over to Corrigan |
|
Thu Apr 22 11:03:57 2021, JS, Thursday 22nd April: 12:00-16:00 12x
|
12:00 Taking over from Liam, all is well.
12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz
12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they
need to check and will need apprx. 15 min.")
12:30
ucesb ok (no beam)
DB No faults
Stats ok.
12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.
13:00
We are taking intermittent beam while they work on the FRS
DB No faults
ucesb ok elog:261/1
Current ok elog:261/2
Stats ok elog:261/3
Temps ok elog:261/4
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
Base Current Difference
aida12 fault 0x0 : 0x46 : 70
FPGA Timestamp error counter test result: Passed 11, Failed 1
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 6 5 3 2 2 2 3 2 3 7 : 39580
aida02 : 21 7 4 2 2 3 2 3 2 3 7 : 39692
aida03 : 20 10 3 3 0 3 2 3 2 3 7 : 39600
aida04 : 8 7 4 5 2 4 3 2 2 3 7 : 39608
aida05 : 17 14 1 2 3 3 4 2 2 3 7 : 39748
aida06 : 17 10 2 3 2 2 4 2 2 3 7 : 39572
aida07 : 27 10 8 7 2 3 3 2 2 3 7 : 39708
aida08 : 9 6 2 3 2 1 3 3 2 3 7 : 39636
aida09 : 1 2 1 4 0 3 3 2 2 3 7 : 39204
aida10 : 5 8 4 4 2 2 1 3 2 3 7 : 39316
aida11 : 6 4 5 3 2 3 2 2 2 3 7 : 39144
aida12 : 15 5 8 1 0 4 1 3 2 3 7 : 39428
13:32
ucesb ok
DB No faults
Stats ok
14:01
ucesb ok
DB No faults
Stats ok
14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
cat /var/log/messages | grep /MIDAS - attachments 6 & 7
Grafana DSSSD bias & leakage currents for last 7 days - attachment 8
14:55
DB No faults
ucesb ok
Current ok elog:261/9
Stats ok elog:261/10
Temps ok elog:261/11
FPGA Timestamp Check Error:
Base Current Difference
aida12 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 11, Failed 1
15:30
ucesb ok
DB No faults
Stats ok
15:59
ucesb ok elog:261/12
DB No faults
Stats ok |
|
Wed May 12 18:53:38 2021, JS, Wed 12 20:00-00:00
|
19:51 Shift change
Stats ok
ucesb ok, Max implants ~1500 Hz
20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)
20:51
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.
20:59
Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
Base Current Difference
aida05 fault 0x36ca : 0x36ce : 4
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
Base Current Difference
aida13 fault 0xa : 0xb6 : 172
FPGA Timestamp error counter test result: Passed 15, Failed 1
--- both errors above same module as earlier shift
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 3 2 0 1 3 3 2 3 3 6 : 36092
aida02 : 6 4 4 2 2 3 1 3 3 3 6 : 36280
aida03 : 6 3 2 0 0 3 3 2 3 3 6 : 36048
aida04 : 10 3 1 1 3 3 1 3 3 3 6 : 36272
aida05 : 19 8 2 0 2 3 1 3 3 3 6 : 36268
aida06 : 2 2 0 2 3 4 2 2 3 3 6 : 36120
aida07 : 6 7 3 0 3 4 3 2 3 3 6 : 36416
aida08 : 7 5 4 0 3 2 2 3 3 3 6 : 36420
aida09 : 21 6 3 2 2 4 2 3 2 3 6 : 35700
aida10 : 13 3 1 1 3 3 1 4 2 3 6 : 35772
aida11 : 25 6 4 2 1 4 2 4 2 3 6 : 36180
aida12 : 0 4 0 3 2 3 3 3 2 3 6 : 35712
aida13 : 18 5 2 3 2 3 2 4 2 3 6 : 36080
aida14 : 2 1 1 1 2 3 2 4 2 3 6 : 35904
aida15 : 0 1 1 3 0 4 2 2 3 3 6 : 35960
aida16 : 4 5 3 2 3 4 1 3 2 3 6 : 35432
ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4
21:36
ucesb ok
Stats ok
22:01
ucesb ok 1500 Hz peak implant
Stats ok
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.
22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler
23:40 FRS magnet issue, possibly power, no beam
00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah |
|
Fri May 14 19:01:09 2021, JS, Friday 14th May 20:00-00:00 27x
|
20:00 Taking over from CB
20:33 Doing full checks.
Temps ok elog:319/8
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit -
Base Current Difference
aida01 fault 0xf932 : 0xf934 : 2
aida02 fault 0x62ec : 0x62ee : 2
aida03 fault 0x8679 : 0x867b : 2
aida04 fault 0xf0e4 : 0xf0e6 : 2
aida05 fault 0x9db8 : 0x9dc5 : 13
aida06 fault 0x7f18 : 0x7f1a : 2
aida07 fault 0xdd2c : 0xdd2e : 2
aida08 fault 0x1557 : 0x1559 : 2
White Rabbit error counter test result: Passed 8, Failed 8
FPGA Timestamp errors -
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 35 34 30 8 1 4 1 2 2 3 6 : 35772
aida02 : 37 30 28 5 3 1 2 4 2 3 6 : 36644
aida03 : 41 29 30 16 4 3 1 5 1 3 6 : 36588
aida04 : 13 6 23 12 4 4 4 4 2 2 6 : 35412
aida05 : 20 32 23 10 2 2 2 2 1 4 6 : 36736
aida06 : 14 5 10 5 1 3 2 3 2 3 6 : 35680
aida07 : 44 23 22 9 4 3 1 3 2 3 6 : 36200
aida08 : 39 30 28 9 0 3 2 3 3 2 6 : 35308
aida09 : 41 31 28 10 2 5 1 3 1 3 6 : 35484
aida10 : 43 23 27 7 3 1 1 3 3 3 6 : 36916
aida11 : 39 35 17 10 3 3 2 2 2 3 6 : 35908
aida12 : 37 36 15 10 2 2 2 4 3 2 6 : 35684
aida13 : 47 32 17 7 4 2 3 4 3 2 6 : 36012
aida14 : 23 33 14 10 3 4 3 4 3 2 6 : 36164
aida15 : 44 28 24 12 3 2 4 3 3 2 6 : 35920
aida16 : 22 17 24 17 3 4 2 3 1 3 6 : 35648
Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27
Resume info #3 - elog:319/3 same as elog:317/26
Pause info #2 - elog:319/4 same as elog:317/32
AIDA disk info #6 - elog:319/5 same as elog:317/31
AIDA ADC data items - elog:319/6 same as elog:317/30
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see
clearly once uploaded the screenshots]
Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%
21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay
21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit -
Base Current Difference
aida01 fault 0xf932 : 0xf934 : 2
aida02 fault 0x62ec : 0x62ee : 2
aida03 fault 0x8679 : 0x867b : 2
aida04 fault 0xf0e4 : 0xf0e6 : 2
aida05 fault 0x9db8 : 0x9dc5 : 13
aida06 fault 0x7f18 : 0x7f1a : 2
aida07 fault 0xdd2c : 0xdd2e : 2
aida08 fault 0x1557 : 0x1559 : 2
White Rabbit error counter test result: Passed 8, Failed 8
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 39 34 29 8 1 4 1 2 2 3 6 : 35772
aida02 : 46 29 28 5 3 2 2 3 2 3 6 : 36288
aida03 : 39 25 29 16 4 2 2 5 1 3 6 : 36660
aida04 : 14 6 19 12 4 4 3 4 2 2 6 : 35096
aida05 : 20 30 24 10 2 2 2 2 1 4 6 : 36736
aida06 : 26 4 11 5 2 4 3 2 2 3 6 : 35672
aida07 : 46 22 22 9 4 3 1 3 2 3 6 : 36200
aida08 : 28 32 28 9 0 2 2 4 3 2 6 : 35664
aida09 : 32 30 29 10 3 5 2 3 1 3 6 : 35776
aida10 : 39 23 25 7 3 1 1 3 3 3 6 : 36868
aida11 : 40 31 21 10 3 3 1 4 1 3 6 : 35688
aida12 : 17 31 12 10 2 2 3 4 3 2 6 : 35772
aida13 : 35 28 22 7 4 2 3 4 3 2 6 : 36012
aida14 : 30 33 14 10 3 4 3 4 3 2 6 : 36192
aida15 : 34 26 24 12 3 2 4 3 3 2 6 : 35864
aida16 : 38 24 27 17 3 4 2 3 1 3 6 : 35816
Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14
Aida Correlation Info #8 - elog:319/15 - rates showed zero, checked again and normal elog:319/17
Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%
22:00
Temps ok
Stats ok
ucesb ok
22:35
Temps ok
Stats ok
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up
again elog:319/17
22:54 They are closing the file, we are on R5_155
23:00 |
|
Wed Jun 5 08:34:47 2024, JB. AM, MP, MY, Detector bias tests 14x
|
We are trying to bias each system on its own and in sequence biasing each other detector system to see the interdependence.
HV OK. TEMP OK. Attachment 1 &2.
9.43 bPlast unbiased, BB7 unbiased. Attachments 3-6 is the noise situation with only AIDA biased (bPlast was biased but turned off after AIDA was biased.)
10/16 FEEs < 20k noise. Max noise is 250k. Downstream AIDA appears to be seeing additional noise since S100.
11:21 Returning after an hour the noise has increased significantly in many channels. Scale changed to 10000 in histograms, it appears that the problem is from a couple hot channels.
11:41 'Power' cycled AIDA turned off. bPlast turned on, on its own, then BB7. bPlast doesn't seem to notice anything.
12:21 bPlast and BB7 on. AIDA ON. Rates and histograms given by attachment 7-8. More FEEs noisy, but same level as 11:21. There appears to be the same noisy channel in all of the FEEs (?) Some grounding not properly accounted for, or a wire touching somewhere (?)
13:55h: AIDA & BB7 on. Rates and histograms given in attachments 9-10. Same level as bplast & AIDA on. Same noisy channels in FEEs. According to Nic, these correspond to the end of the ribbon cables & is normal.
13:59 AIDA, bPlast & BB7 on before lunch. Same levels of noise as in previous combinations (attachments 11-12).
So it seems we can run with the three sub-systems biased simultaneously.
|
|
Thu Apr 11 08:27:21 2024, JB, TD, 11 April noise tests 48x
|
TO-DO for 11.04.2024
- Try bringing bPlast drain wire ground back to the PSU ground for PN 300 and R&SRMP4040.
- Recheck the downstream detector bias and ground scheme.
-
9:27 Restarted AIDA and hit go at 9:54. Situation the same as it was in the evening of 10.04.2024. 10 out of 16 FEE64s in good condition. Waveforms of aida16,
TEMP OK - HV OK. See attachments 1-6.
10:20 We grounded the output of the R&SRMP4040 to to the drain wire ground of the bPlast ribbon cables and the output of the PN 300 PSU that is powering the booster boards. Results given by attachments 7 -11. We only saw a marginal improvement in the overall noise condition.
10:32 We connected the ground of the frame to the drain wire ground, that is also mutually connected to the ground of the R&SRMP4040 output ground and PN 300 output ground. Results given by attachments ???. We did not see any improvement in the overal noise condition.
11.00 DSSSD HV OFF
FEE64 power OFF
Check seating of all adaptor PCBs and drain/ground wires secured - generally OK
Restart
1.8.W spectra - 20us FSR - attachments 15-16
per FEE64 Rate spectra - attachment 17
10/16 < 20k, max 110k
ADC data item stats - attachment 18
Incremental improvement.
11.45 DSSSD HV OFF
FEE64 power OFF
test - daisy chain removed
Restart
1.8.W spectra - 20us FSR - attachments 20-21
per FEE64 Rate spectra - attachment 19
10/16 < 20k, max 210k
ADC data item stats - attachment 22
overall somewhat worse
12.20 DSSSD HV OFF
FEE64 power OFF
Re-install test - daisy chain, Tighten aida04 DSSSD ribbon cable drain wire.
Restart
1.8.W spectra - 20us FSR - attachments 23-24
per FEE64 Rate spectra - attachment 25
ADC data item stats - attachment 26
10/16 < 20k, max 110k
Status quo ante
aida04 & aida08 1*W & 2*W spectra - 200us FSR - attachments 27-31
large transients observed for aida04 1*W - cable or ASIC fault?
aida16 **W spectra - 200us FSR - attachments 32-35
all channels apopear to be working but mix of high/low noise channels for asics #3-4, asics #1-2 all appear high noise
14.00 DSSSD HV OFF
FEE64 power OFF
Disconnect ribbon cables from aida04 adaptor PCB
Restart
n+n FEE64 1.8.W spectra - 20us FSR - attachments 37
per FEE64 Rate spectra - attachment 36
ADC data item stats - attachment 38
Implies origin of large transients observed in aida04 asic#1 is downstream of the FEE64 adaptor PCB, i.e. ribbon cable or DSSSD.
14.40 DSSSD HV OFF
FEE64 power OFF
re-connect ribbon cables from aida04 adaptor PCB
1x pin J2 slightly bent - straightended with screwdriver - FFSD connector insertion OK
Restart
n+n FEE64 1.8.W spectra - 20us FSR - attachments 40-41
per FEE64 Rate spectra - attachment 42-43
ADC data item stats - attachment 39
10x < 20k, max 120k
aida11 asic#4 1.8.W spectra - 200us FSR - attachment
no large transients observed
DSSSD #1 & #2 bias from -120V to -100V
ADC data item stats - attachment 40
9x < 20k, max 220k (aida08)
per FEE64 Rate spectra - attachment 41
17.15 bPlas ON -
Current bPlast ground configuration:
bPlas current ground configuration - drainwires of all ribbon cables excluding short side (cont. with snout) are grounded back to the bPlast R&SRMP4040 PSU, on the 29 V output. 4 V PSU to booster boards are floating, output of PN 300 is grounded to the PSU. The mesytec PSU, that also powers bPlast SiPMs at
29V is not grounded to anything but the frame. Snout is currently light tight, I suspect internal radiation from FATIMA.
It is noted that bPlast current draw fluctuated significantly (+/- 300 mV) as a result of thresholds set to the detector. I tried to set the thresholds HIGH to stabilise the bPlast detector as the power draw is fluctuating greatly.
AIDA noise very good
ADC data item stats - attachment 42
12x < 20k, max 83k (aida04)
per FEE64 Rate spectra - attachment 43
To Do list
- separate 29V (low current) and 4V (high current) return paths/ground refs
- The mesytec PSU that powers the bPlast detector is running to frame, it might be an idea to ground ribbon c
|
|
Thu Apr 18 08:57:48 2024, JB, TD, Thursday 18 April 36x
|
09.52 DAQ continues file S100_alpha/R5
note merger not functioning correctly due to aida03, aida04 & aida09 out of WR sequence
DSSSD bias & leakage current OK - attachment 1
FEE64 temps OK - attachment 2
*except* aida02 ASIC temp which is known to be u/s
WR timestamps - attachment 3
aida03, aida04 & aida09 out of sequence
All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 4-6
ADC data item stats - attachment 7
per FEE64 Stat spectra - attachment 8
per FEE64 1.8.L spectra - attachments 9-10
per FEE64 1.8.W spectra - 20us FSR - attachments 11-12
Merger, TapeServer & Link Data Rates - attachments 13-15
Grafana - attachment 16
DSSSD bias/leakage current trends OK
19.15 alpha background file S100_alpha/R10
DSSSD bias & leakage current OK - attachment 17
FEE64 temps OK - attachment 18
*except* aida02 ASIC temp which is known to be u/s
All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 19-21
WR timestamps OK - attachment 22
ADC data item stats - attachment 23
Merger, TapeServer & Meger stats - attachments 24-26
data rate c. 2.8Mb/s
per FEE64 1.8.W spectra - 20us FSR - attachments 27-28
correlation scaler stats & data link stats - attachments 29-30
data rate dominated by MACB scaler inputs aida03 & aida04 ( c. 200kHz each )
22.16 DAQ continues Ok file S100_alpha/R10_35
Merger, TapeServer & Meger stats - attachments 31-33
data rate c. 2.8Mb/s
ADC data item stats - attachment 34
FEE64 temps OK - attachment 35
*except* aida02 ASIC temp which is known to be u/s
DSSSD bias & leakage current OK - attachment 36 |
|
Mon Mar 11 15:19:44 2024, JB, NH, Priyanka, Michael Armstrong, Helena Albers, To Do: AIDA PCB tests
|
To do:
1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs
- When the p+n ribbon cables were disconnected for both the upstream and downstream AIDA, the pins and ribbon cables did not appear to have any blemishes.
- Both sets of adaptor PCBs were biased to 100 V and only had an apparent fluctuation in current at c. 0.01 uA. At low voltages (~10V) the current cycled between 0 and 1 uA.
2) Check that all ribbon cables are properly seated in the adaptor PCBs
- When inspected all ribbon cables seemed to be properly seated. A decent press was applied to confirm the seating. The PCBs themselves were inspected and no noticeable damage was observed.
3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables
- Done for the p+n side, can decide if this is needed for the n+n side.
4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.
- Will have to decide further.
5) From email discussion: Check the SHV connector is grounded if one unplugs the cable from the back of the HV module. If yes: We are touching a ground somewhere. If no: it's via the HV (fine).
- When HV#0 was disconnected from the HV module, we used a multimeter on the SHV connecter and frame and read zero resistance 0L on the meter. While when completing the circuit for HV#1 (still) plugged into the HV module we read a resistance of > 1.7 Ohm.
- The HV is touching ground somewhere. This should not be a problem however as the snout is isolated, and this was observed by connecting the SHV to the snout and reading 0L.
6) Properly cover snout with black cloth and bias upstream.
- Snout was covered with black cloth and a black bag. The downstream detector p+n side was biased at 50V and charging/discharging was observed. This is probably due to a short connection somewhere.
Summary: from the p+n side test the voltage-current break down appears to come from inside the snout. |
|
Thu Mar 14 13:00:23 2024, JB, NH, MA, AM, GA, Mounting and biasing DSSD 2
|
new Downstream DSSD2: 3208-2/3208-5/3208-8
Covered with black cloth.
| Voltage (V) |
Current (uA) |
| 10 |
3.1 |
| 20 |
3.8 |
| 30 |
|
| 40 |
|
| 50 |
4.515 |
| 60 |
|
| 70 |
4.2 |
| 80 |
4.545 |
| 90 |
5.5 |
| 100 |
7.940 |
| 110 |
|
| 120 |
|
|
Voltage (V)
|
Current (uA) |
| 10 |
|
| 20 |
|
| 30 |
|
| 40 |
|
| 50 |
|
| 60 |
|
| 70 |
|
| 100 |
7.6 |
| 110 |
9.5 |
| 120 |
Breakdown |
| |
|
| |
|
Wafer 3 (beam left) shows breakdown issues at 90V... curious
We replace with november beam triple: 3208-3/3208-21/3208-22
We must speak to Micron about why so many 3208 wafers seem to have issues at >90V
It seems unlikely to be other issues |
|
Thu Mar 14 12:13:17 2024, JB, NH, MA, Mounting and biasing DSSD 1
|
Upstream bPlast mounted
new Upstream DSSD: 3208-2/3208-5/3208-8
Covered with black cloth.
| Voltage (V) |
Current (uA) |
| 10 |
3.345 |
| 20 |
3.6 |
| 30 |
|
| 40 |
|
| 50 |
4.5ish |
| 60 |
|
| 70 |
5.4 |
| 80 |
|
| 90 |
|
| 100 |
5.6 |
| 110 |
|
| 120 |
5.7 |
| |
|
V-I behaviour found to be nominal. |
|
Wed Mar 13 13:02:19 2024, JB, NH, HA, MA, DSSD 1 biasing tests
|
Dismounted Snout and biased DSSSD1 channels
leftmost waifer working
middle waifer reaches 90V then current ramps up
rightmost waifer reaches 80V then current ramps up
DSSSD2 had fingerprint near connections, as shown in the attached image some had been squashed. |
|
Thu Mar 7 15:15:29 2024, JB, NH, CC, AIDA HV Bias Test IV Curve
|
AIDA I-V Test
| Voltage (V) |
Current (uA) |
| 10 |
1.43 |
| 20 |
2.065 |
| 30 |
2.415 |
| 40 |
2.64 |
| 50 |
2.825 |
| 60 |
2.99 |
| 70 |
3.185 |
| 80 |
3.58 |
| 90 |
5.01 |
| 100 |
9.4 |
- Voltage-Current test of newly installed AIDA snout. Breakdown observed at around ~92 V. Probably caused my light leakage into the snout, will investigate further.
- Tried to cover the snout with a black cloth. This did not change the breakdown behaviour. |
|
Wed Mar 27 14:22:35 2024, JB, NH, Installing FEE64s of DSSSD2 14x
|
Mounted on frame:
DSSD 1 (Upstream) : 3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22
Current mapping has been redone for better management.
| AIDA - FEE Mapping |
| DSSSD 1 |
DSSSD2 |
| FEE |
MAC |
FEE |
MAC |
| aida01 |
41:ba:8a |
aida06 |
41:05:15 |
| aida02 |
41:f6:b7 |
aida07 |
41:f6:5a |
| aida03 |
41:d8:21 |
aida08 |
41:d7:cd |
| aida04 |
41:a0:71 |
aida10 |
41:d0:0E |
| aida05 |
41:cf:ac |
aida13 |
41:d8:2b |
| aida09 |
41:ee:10 |
aida14 |
42:0d:15 |
| aida15 |
41:b4:0c |
aida11 |
41:EE:0f |
| aida12 |
41:ba:89 |
aida16 |
41:f6:ed |
Going to try optimising noise now.
DHCP updated
new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)
New layouts: /home/npg/LayOut/GSI_Triple_S100
New layout.txt
Firmware of aida11 updated from 0xea40704 to 0x3350706
Temps GOOD fig 5
Rates fig 6, 7
Check adapter alignment aida14 and aida16
bPlas left/right cables are not insulated and shorting to the snout
Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor
From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!
White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)
Turn off bPlas
We see the noise drop a lot |
|
Mon Jun 3 14:40:34 2024, JB, NH, 03 June 2024 10x
|
15:40 reseated FEE64s adaptors of aida01, aida09, aida13 & aida16.
TEMP OK. HV OK. System wide checks all passed excl. aida07 ADC calibration, see attached.
Histograms and rates look same as previous, reseating aida01 seems to have worked. aida16 still unhappy.
re-reseated aida16. 16:19 - appears to have fixed the problem in the channel.
Pulser test + connected.
bPlast ground removed from frame.
To-do:
- Need clarification about links connected on AIDA n+n FEE64 adaptor boards. Currently, the situation is given by the attachment 8 & 9.
- Fix BB7 grounding, continuity between snout and frame observed.
- Pulser walkthrough for n+n, can do p+n when convienient.
18.30 Noted FEE64 ADC stats now significantly higher ( attachments 9-10 ) cf. earlier this afternoon (attachments 1-2)
DSSSD bias OFF. FEE64 power OFF.
|
|
Mon Dec 16 13:07:04 2024, JB, MP, CC, AIDA timing test 16x
|
https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG
Day2:
10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3
FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9
ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.
The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10
No signal was observed, threshold was then set to 0x190 (400 keV) 677/11
13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.
|
 Tue Dec 17 12:45:36 2024, JB, MP, CC, AIDA timing test
|
| Quote: |
|
https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG
Day2:
10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3
FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9
ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.
The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10
No signal was observed, threshold was then set to 0x190 (400 keV)
13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.
|
678/1 shows the AIDA OR64 trigger from the fast comparator of aida10.
678/2-4 show the bPlast accepted trigger after the bPlast DAQ has been triggered by the AIDA OR64. The images show a signal coming fast in time after the pulser and also a signal at around 80 us that appears to be coming from pile up.
The bPlast DAQ also now runs with the AIDA OR64 trigger after the NIM out signal from the MACB was sent to an octal discriminator to fix the width of the signal and also the pulse width which from TAMEX should be kept at or above 100 ns. The NIM signal from the MACB was mostly 100 ns but also jumping to signals with a 20ns pulse width.
|
|
Wed Jan 8 16:47:05 2025, JB, MP, CC, AIDA timing test
|
| Quote: |
|
https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG
Day2:
10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3
FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9
ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.
The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10
No signal was observed, threshold was then set to 0x190 (400 keV) 677/11
13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.
|
The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.
These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.
This was set to have the HitRate in aida10 to be just above the noise. |
|