 Thu Jun 17 11:14:04 2021, TD, Thursday 17 June 8x
|
12.15 DAQ continues OK - no storage mode
12.15 System checks OK *except*
Base Current Difference
aida01 fault 0x8944 : 0x8963 : 31
aida02 fault 0x9b36 : 0x9b55 : 31
aida03 fault 0xf26b : 0xf28a : 31
aida04 fault 0x57a0 : 0x57bf : 31
aida05 fault 0x4d3c : 0x4d55 : 25
aida05 : WR status 0x10
aida06 fault 0x640c : 0x6425 : 25
aida07 fault 0x255e : 0x2577 : 25
aida08 fault 0xffbd : 0xffd6 : 25
aida09 fault 0x87e : 0x895 : 23
aida10 fault 0xc557 : 0xc56e : 23
aida11 fault 0x683e : 0x6855 : 23
aida12 fault 0x1c35 : 0x1c4c : 23
aida13 fault 0x8f41 : 0x8f58 : 23
aida14 fault 0x1b47 : 0x1b5e : 23
aida15 fault 0x4e4 : 0x127e : 3482
aida16 fault 0xf570 : 0xf587 : 23
White Rabbit error counter test result: Passed 0, Failed 16
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida01 fault 0x0 : 0x2 : 2
aida02 fault 0x0 : 0x2 : 2
aida05 fault 0x0 : 0x2 : 2
aida06 fault 0x0 : 0x2 : 2
aida09 fault 0x0 : 0x2 : 2
aida10 fault 0x0 : 0x2 : 2
aida11 fault 0x0 : 0x2 : 2
aida12 fault 0x0 : 0x3e03 : 15875
aida13 fault 0x0 : 0x32d6 : 13014
aida14 fault 0x0 : 0x2 : 2
aida15 fault 0x0 : 0x2 : 2
aida16 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 31 11 9 4 4 4 2 2 2 4 6 : 37604
aida02 : 30 10 10 3 3 4 3 2 2 4 6 : 37768
aida03 : 35 11 8 6 5 3 2 2 3 3 6 : 36580
aida04 : 33 9 4 4 3 4 2 3 3 3 6 : 36940
aida05 : 41 9 7 3 4 4 2 2 2 4 6 : 37564
aida06 : 38 6 5 3 2 3 2 3 4 3 6 : 37752
aida07 : 25 12 6 5 4 2 3 2 2 4 6 : 37572
aida08 : 32 7 10 5 4 3 3 2 2 4 6 : 37752
aida09 : 41 8 6 4 2 3 3 2 2 4 6 : 37572
aida10 : 31 10 11 5 5 3 3 3 3 3 6 : 37340
aida11 : 29 7 12 3 4 4 3 3 1 4 6 : 37324
aida12 : 42 8 7 5 3 4 3 3 1 4 6 : 37304
aida13 : 43 10 4 5 5 3 3 3 1 4 6 : 37276
aida14 : 40 15 4 5 4 4 3 3 3 3 6 : 37368
aida15 : 13 5 8 2 1 2 3 2 1 4 7 : 40284
aida16 : 39 9 6 6 5 3 3 3 1 4 6 : 37316
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Sat Jun 12 15:46:33 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1025 Last changed Sat Jun 12 15:48:28 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
12.15 Hot HEC channels - ASIC check
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - 20us FSR - attachments 3 & 4
ADC data items - attachment 5
FEE64 temperatures OK - attachment 6
DSSSD bias & leakage currents - attachment 7
significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C
Merger/Tape Server/Merger statistics - attachment 8
no merger errors reported since previous restart |
|
Thu Apr 15 22:38:18 2021, OH, Thursday 15th April - 16th April
|
23:38 They are testing the finger detector
It is a lot of U they are putting into it which is passing through to AIDA ~3500Hz
Most seems to be punching through
23:45 Rate seems to be reduced to peaking at 1000Hz
00:06 DAQ Running to no storage
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xa
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse |
|
Thu Apr 15 13:28:07 2021, TD, Thursday 15 April 19x
|
13.28 System wide checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0x44c2 : 0x44c5 : 3
aida02 fault 0xc81f : 0xc822 : 3
aida03 fault 0x9342 : 0x9344 : 2
aida04 fault 0xdb3f : 0xdb41 : 2
aida05 fault 0x4baf : 0x4bb3 : 4
aida06 fault 0xd2a4 : 0xd2a8 : 4
aida07 fault 0xc51c : 0xc51f : 3
aida08 fault 0xe853 : 0xe856 : 3
aida09 fault 0x5212 : 0x5216 : 4
aida10 fault 0xf81d : 0xf81f : 2
aida11 fault 0x810 : 0x812 : 2
aida12 fault 0xa279 : 0xa27a : 1
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x5 : 5
aida12 fault 0x0 : 0xc : 12
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 57 33 12 15 3 3 3 2 3 2 6 : 35276
aida02 : 15 3 5 2 1 4 1 3 3 3 6 : 36388
aida03 : 13 4 2 1 4 3 1 3 3 3 6 : 36372
aida04 : 21 5 2 1 2 4 4 2 3 3 6 : 36668
aida05 : 10 5 6 3 2 3 3 3 1 4 6 : 36880
aida06 : 22 12 1 0 3 3 1 3 3 3 6 : 36360
aida07 : 19 3 3 0 3 3 2 4 1 4 6 : 37076
aida08 : 21 6 2 4 2 3 1 3 3 3 6 : 36388
aida09 : 2 7 5 3 2 4 2 3 2 3 6 : 35696
aida10 : 7 3 1 3 1 3 3 3 2 3 6 : 35684
aida11 : 1 2 1 3 2 5 2 2 3 3 6 : 36228
aida12 : 17 10 4 2 3 2 2 4 2 3 6 : 36052
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida02 => Options file size is 2114 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
13.30 Output of ~/oh/OptionsCheck.py
02:25:00 PM
FAULTS!
Unknown option in FEE2 Options database
Unknown entry starts:
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&ASIC.se
ttings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Unknown option in FEE2 Options database
Unknown entry starts:
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Unknown option in FEE2 Options database
Unknown entry starts:
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
13.37 Detector bias & leakage currents OK - attachment 1
FEE64 temps OK - attachments 2
Statistics - good events, ADC data, disc data, info code 4 & 5, pause & resume, correlation - attachments 3-10
per FEE64 rate spectra - attachments 11 & 12
per FEE64 1.8.L spectra - attachments 13 & 19
aida5 peak width 70 ch FWHM, aida06 peak width 89ch FWHM
per FEE64 1.8.W spectra 20us FSR - attachments 14 & 15
meger & tape server - attachments 16-18
|
|
Thu Mar 12 08:04:24 2020, OH, CA, TD, Thursday 12th March 08:00-16:00 14x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
09:00 Bias and leakage currents ok - attachment 1
Statistics ok - attachment 2
FEE temperatures ok - attachment 3
System wide checks all ok
09:08 No beam, not sure exactly how long it has been down.
FRS team are working on it
09:28 still no beam. issues with SIS.
DAQ set to NoStorage mode
most recent file R7_522
10.27 beam is back
writing to file R9_0
10.43 attachment 4 - hit rate spectra
11:09 all system wide checks ok
FEE64 temps ok - attachment 5
good event statistics ok - attachment 6
detector bias and leakage currents - attachment 7
11.18 merger ok, ~ 2.7 million data items per second
TapeServer ok, ~ 25 Mb/s data rate
Data forwarding to MBS ok
11.20 writing to file R9_39
11.22 beam off
11.24 beam back online (R9_42)
11.41 FRS has reverted to an older setting
13:04 hit rate spectra - attachment 8
13:19 all system wide checks ok
13:30 FEE64 temps ok - attachment 9
good event statistics ok - attachment 10
detector bias and leakage currents ok - attachment 11
15:31 good event statistics ok - attachment 13
detector bias and leakage currents ok - attachment 14
FEE64 temps ok - attachment 15
all system wide checks ok
beam seems to have stopped during 2pm meeting
15.35 TapeServer ok, data rate ~ 23 Mb/s
Merger ok, rate of 2.5 - 3 million data items per second
data forwarding to MBS ok
15.38 AIDA writing to file R9_233
16:02 beam back - AIDA writing to file R9_250
|
|
Wed Mar 11 23:44:27 2020, OH, Thursday 12th March 00:00-08:000 10x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
00:44 Implantation range in AIDA - attachment 1
01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
the AIDA boxes and begin moving files onto that
01:35 System wide checks all ok
Bias and leakage current ok - attachment 2
Stats all ok - attachment 3
FEE temperatures ok - attachment 4
TapeServer rate around 25-29MB per second
Merger rate around 2.6-3.2 million items per second
04:11 System wide checks all ok
Bias and leakage currents ok - attachment 5
Stats all ok - attachment 6
FEE temp ok - attachment 7
06:38 System wide checks all ok
Bias and leakage currents ok - attachment 8
Stats all ok - attachment 9
FEE temp ok - attachment 10 |
|
Thu May 12 07:09:43 2022, TD, Thursday 12 May 08:00-24:00 27x
|
08.10 AIDA DAQ status
14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
AD9222 OFF - waveforms disabed
disc - all disabled
ASIC settings 2021Apr29-13-16-00
slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23
BNC PB-5
amplitude 1.000V
attenuation x1
tau_d 1ms (tail pulse)
polarity +
frequency 22Hz
08.14 all histograms zero'd
2
08.15 system wide check counter baseline
08.17 check ASIC control - all FEE64s all ASICs
Attachment 1 - grafana DSSSD bias & leakage current - OK
Attachment 2 - per FEE64 stat spectra
Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips
Attachments 5 & 6 - per FEE64 1.8.L spectra
pulser peak width aida01 124ch FWHM, aida04 293ch FWHM
Attachment 7 - per FEE64 rate spectra
Attachments 8-12 - system wide checks - all OK
Attachments 13-16 - statistics - adc, pause, resume & correlation data
Attachment 17 - FEE64 temps OK
Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends
Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages
08.44 DAQ continues - file NOTAPE/R18 - no storage mode
08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc
08.53 no storage mode enabled
Attachment 21 - analysis of file NOTAPE/R18_20
Attachment 22- analysis of file NOTAPE/R19_19
AIDA12 slow comp up to 0xd improved deadtime drastically for FEE
10:54 Beam on detector. Take another run to test deadtime - attachment 23
Deadtimes ranging from 10-40% in n+n FEEs
Attachment 23 & 24 aida06 aida02 1.*.H spectra
14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime) |
|
Thu May 12 00:37:52 2022, OH, AM, Thursday 12 May 13x
|
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
That is now complete.
Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc
R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
Will lower 2 and 4 to 200 and raise 6 and 8 to 300
R17 2 good, 4 shakey 6 and 8 poor - attachment 2
R18 2 and 4 at 220 and 6 and 7 at 350
2 and 8 low deadtime, 4 and 6 could be better. - attachment 3
Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
Statistics at this rate - attachment 4
02:18 Statistics - attachment 7
Temperatures - attachment 6
Bias & leakage current - attachment 5
Clock Status - OK
White Rabbit - OK
The other 3 checks currently aren't working
04:11 Statistics - attachment 10
Temperatures - attachment 9
Bias & leakage current - attachment 8
Clock Status - OK
White Rabbit - OK
06:05 Statistics - attachment 13
Temperatures - attachment 12
Bias & leakage current - attachment 11
Clock Status - OK
White Rabbit - OK |
|
Thu Jun 12 13:15:25 2025, GB, CC, NK, MP, TD, Thursday 12 June 2025
|
EMC tests with NK (Orsay)
Stack configuration (upstream->downstream)
bPlas
AIDA DSSSD #0
AIDA DSSSD #1
bPlas
3x MSL type BB7(DS)-1000 (1x aida10, 2x Mesytec preamplifiers)
AIDA DSSSD #0 bias -100V leakage current -13.23uA
AIDA DSSSD #1 bias -100V leakage current -9.68uA
BNC PB-5 settings
Amplitude 1.0V
Attenuator x1
Frequency 25Hz
Polarity +
tau_d 1ms
Tail pulse
14.39 per p+n FEE64 1.8.W spectra - 20us FSR - attachment 1
per p+n FEE64 1.8.L spectra - attachment 2
Pulser peak widths ch FWHM
DSSSD #0 (DSSSD - FEE64 cables with heavy duty braid - electrically connected to ground of adaptor PCB and internally connected to ground within snout)
9 1 5
120 201 21
15 3 12
114 180 broad
DSSSD #1 (DSSSD - FEE64 cables with mesh - currently not connected i.e. floating to adaptor PCB and internally connected to ground within snout)
10 14 13
- 22 374
11 7 16
broad 417 20
Pulser peak widths c. 20 ch FWHM probably indicate cable from DSSSD *not* connected to FEE64 adaptor PCB
|
|
Thu Mar 11 07:09:25 2021, OH, LS, Thursday 11th of March 29x
|
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
N.B. wrong date given to the screenshots by accident (It's early)
Statistics - attachment 1
FEE temperatures (AIDA10 unavailable) - attachment 2
FEE temperature - attachment 3
Bias and leakage currents ok - attachment 4
08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
Restarted but no stats in aida04
It did not recover from a reset
Telnet in and did a reboot command
09:10 Finally recovered issue was it was not linking to merger
There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
Manually set the ASIC settings for aida12. Rates look as expected after
Will try to reset once beam is back
10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
Analysis of R30_21 looks fine
10:38 System wide checks ok
Statistics - attachment 5
Temperature - attachment 6
Bias and leakage - attachment 7
12.10 (LS)
System wide checks all okay, no fails
Statistics (attachment8)
Rate spectra (attachment9)
FEE temps (attament10)
Leakage currents written to sheet (attachment11)
Merger ~45M items/s
Tapeserver ~14MB/s
13.45 no beam (around file R30_98)
14.00 System wide checks all okay, no fails
Statistics (attachment12)
FEE temps (attament13)
Leakage currents written to sheet (attachment14)
Merger ~44M items/s
Tapeserver ~14MB/s
Beam still down
14.11 beam back aida file R30_112
14.13 beam back down, back at 14.16
14.20 no beam
14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
double by analysing R30_101,102,103 no timewarps
beam is back but low intensity at the moment
14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)
16.00 System wide checks all okay, no fails
Statistics (attachment16)
Rate spectra (attachment17)
FEE temps (attament18)
Leakage currents written to sheet (attachment19)
Merger ~45M items/s
Tapeserver ~14MB/s
18.00 System wide checks all okay, no fails
Statistics (attachment20)
Rate spectra (attachment21)
FEE temps (attament22)
Leakage currents written to sheet (attachment23)
Merger ~45M items/s
Tapeserver ~15MB/s
bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
now see error in MBS relay!
"Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
It triggered 63 times.
I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser
19:10 MBS DAQ stopped to allow FRS tests
19:38 MBS DAQ is back
20:15 Aida07 fails FPGA check
Base Current Difference
aida07 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
All other system wide checks ok
Statistics - attachment 24 (Again ignore the year in the filename)
Temperatures - attachment 25
Bias and leakage currents - 26
21:36 They are resetting the time sorter due to FATIMA issues
22:38 System wide checks error on WR in addition to the previous FPGA error
Base Current Difference
aida05 fault 0x5cc9 : 0x5cca : 1
aida06 fault 0xde76 : 0xde77 : 1
aida07 fault 0xa566 : 0xa567 : 1
aida08 fault 0x6ab0 : 0x6ab1 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Statistics - attachment 27
Temperature - attachment 28
Bias and leakage current - attachment 29
23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain. |
|
Thu Jun 10 11:32:27 2021, TD, Thursday 10 June 9x
|
12.26 DAQ continues file S496/R32_428
High data rate from c. 06.00 this morning
ASIC check
System wide checks OK *except*
Base Current Difference
aida01 fault 0x8944 : 0x8947 : 3
aida02 fault 0x9b36 : 0x9b39 : 3
aida03 fault 0xf26b : 0xf26e : 3
aida04 fault 0x57a0 : 0x57a3 : 3
aida05 fault 0x4d3c : 0x4d3d : 1
aida05 : WR status 0x10
aida06 fault 0x640c : 0x640d : 1
aida07 fault 0x255e : 0x255f : 1
aida08 fault 0xffbd : 0xffbe : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 35 11 6 3 4 4 2 2 2 4 6 : 37540
aida02 : 30 8 7 3 4 4 3 2 2 4 6 : 37768
aida03 : 38 8 13 3 4 3 1 3 3 3 6 : 36744
aida04 : 37 9 7 2 3 4 2 3 3 3 6 : 36940
aida05 : 29 5 9 3 3 3 1 3 2 4 6 : 37580
aida06 : 38 6 2 3 3 3 2 3 2 4 6 : 37768
aida07 : 31 9 2 7 4 2 3 2 2 4 6 : 37572
aida08 : 34 6 3 5 4 4 3 2 2 4 6 : 37768
aida09 : 29 2 12 5 1 3 3 2 2 4 6 : 37540
aida10 : 24 8 15 3 5 3 3 3 1 4 6 : 37296
aida11 : 35 4 15 3 3 4 3 3 1 4 6 : 37308
aida12 : 36 5 3 5 6 4 3 3 1 4 6 : 37384
aida13 : 31 10 4 3 6 4 3 3 1 4 6 : 37356
aida14 : 37 7 1 4 5 4 3 3 1 4 6 : 37276
aida15 : 34 10 6 3 4 1 2 4 2 4 6 : 38168
aida16 : 31 9 3 3 5 4 3 3 1 4 6 : 37268
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Fri Jun 04 09:30:42 CEST 2021
FEE : aida02 => Options file size is 1025 Last changed Sun May 23 00:19:21 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Fri May 14 16:54:56 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Mon May 17 06:25:41 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Sun May 23 00:16:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 202
Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
Lost activity monitor - attachment 2
1.8.W spectra - 20us FSR - attachments 3 & 4
ADC data items - attachment 5
FEE64 temperatures OK - attachment 6
DSSSD bias & leakage currents OK - attachment 7
Merger/Tape Server/Merger statistics - attachment 8
no merger errors reported since previous restart
15.07 analysis of file S496/R32_431 - attachment 9
max dead time aida04 0.33% |
|
Thu May 20 01:00:30 2021, Muneerah, Thu May 20 00:00-08:00 13x
|
02:00 System check
The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!
The rates on ucesb got back to work by around 01:12
Deat Time check ok R14_870, attachment 1
Statistics are ok attachment 2
Temperatures are ok attacment 3
Voltages are ok attachment 4
system wide check
All ok except
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbite
1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 8 8 5 0 6 1 2 2 2 7 : 37296
aida02 : 19 8 11 2 3 4 2 2 3 3 6 : 36412
aida03 : 14 6 3 4 1 3 1 3 3 3 6 : 36312
aida04 : 25 12 12 4 3 4 1 4 3 3 6 : 37316
aida05 : 4 8 3 4 1 2 2 4 3 3 6 : 36928
aida06 : 42 12 8 7 2 3 3 4 3 3 6 : 37736
aida07 : 24 6 7 3 3 3 1 3 2 2 7 : 37536
aida08 : 10 15 7 2 1 4 2 3 3 3 6 : 36752
aida09 : 29 13 6 6 4 3 3 3 3 3 6 : 37244
aida10 : 18 12 7 2 2 3 3 2 2 4 6 : 37464
aida11 : 20 8 0 5 5 4 2 3 3 3 6 : 36976
aida12 : 7 2 4 3 2 3 2 2 4 3 6 : 37068
aida13 : 2 5 5 4 2 3 2 3 3 3 6 : 36608
aida14 : 9 13 8 2 2 2 3 3 2 4 6 : 37836
aida15 : 24 12 12 1 0 3 2 3 3 3 6 : 36640
aida16 : 12 9 9 2 1 3 2 4 3 3 6 : 37128
6:45 System check
Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9
System wide check are all ok except:
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit 1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 11 3 5 0 6 1 2 3 2 7 : 38264
aida02 : 14 9 9 2 3 3 3 2 3 3 6 : 36496
aida03 : 13 7 3 5 2 3 2 3 3 3 6 : 36668
aida04 : 30 15 13 4 3 4 1 3 4 3 6 : 37888
aida05 : 17 9 3 3 1 2 2 4 3 3 6 : 36956
aida06 : 40 16 5 6 2 3 2 3 2 4 6 : 37936
aida07 : 3 2 3 2 3 4 2 2 2 2 7 : 37196
aida08 : 17 10 6 2 1 3 3 3 3 3 6 : 36852
aida09 : 29 11 7 6 3 3 2 4 3 3 6 : 37436
aida10 : 1 10 5 1 1 3 3 2 2 4 6 : 37252
aida11 : 30 9 0 5 4 4 2 3 3 3 6 : 36960
aida12 : 19 10 5 3 2 3 2 2 4 3 6 : 37196
aida13 : 22 9 6 3 2 3 3 3 3 3 6 : 36960
aida14 : 19 6 9 2 2 3 3 2 2 4 6 : 37452
aida15 : 6 8 11 2 1 3 2 3 3 3 6 : 36616
aida16 : 19 14 5 3 1 3 1 4 3 3 6 : 36908
07:33 system check
Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13
System wide check
all ok except
ADC all failed
White rabbit
1 failed
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 9 4 6 0 5 1 2 2 2 7 : 37152
aida02 : 19 6 13 1 3 3 4 2 3 3 6 : 36780
aida03 : 23 5 6 4 2 3 1 3 3 3 6 : 36452
aida04 : 24 11 14 3 3 4 2 3 3 3 6 : 37048
aida05 : 17 3 6 3 1 2 2 3 3 3 6 : 36444
aida06 : 36 15 4 9 2 4 2 4 3 3 6 : 37608
aida07 : 20 8 5 3 3 4 2 2 2 2 7 : 37376
aida08 : 21 11 10 2 1 4 2 2 2 4 6 : 37324
aida09 : 27 14 7 6 3 3 2 4 3 3 6 : 37452
aida10 : 25 9 5 2 2 3 3 2 2 4 6 : 37436
aida11 : 30 14 2 4 5 3 2 3 3 3 6 : 36936
aida12 : 22 3 5 3 2 3 2 3 3 3 6 : 36640
aida13 : 10 7 1 4 2 3 3 2 3 3 6 : 36336
aida14 : 13 9 12 2 3 3 2 3 2 4 6 : 37820
aida15 : 20 8 11 3 0 3 2 3 3 3 6 : 36640
aida16 : 23 6 6 2 0 2 1 3 4 3 6 : 37164
|
|
Thu Mar 21 15:41:49 2024, NH, AM, MP, CC, Thu Mar 21 6x
|
Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
Fig 4-6: After AM and MP turn off Mesytec Preamps
No difference |
 Fri Mar 22 08:22:43 2024, NH, AM, MP, CC, Thu Mar 21
|
> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
>
> Fig 4-6: After AM and MP turn off Mesytec Preamps
>
> No difference
I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off. |
|
Thu Feb 20 16:35:55 2020, TD, NH, Thu 20 February 2020 12x
|
DSSD#3 (most downstream) was replaced
Noted that old DSSD wafer was not sitting properly - detached - in PCB, PCB flexing and/or inadequate epoxy gluing
New DSSD#3 - 2998-22 - 1006 um - 100 V - ~4 uA
Old Detector removed was 3208-14
Fig 1. HV Control
Fig 2. Temps
Fig 3. Stats
Fig 4. Rates
Fig 5-6: Waves
Fig 7-8: Pulser Peaks
Start alpha run at 17:48, 20FEB20, R1
Slow comparator 0x64
Pulser off
Fig 9 : Alpha stats
Fig 10: Merger stats
Fig 11: Merger main page
Fig 12: Timestamp errors in merger console |
|
Thu May 13 17:05:42 2021, Muneerah, MArc, Thu 13 May 04:00-00:00
|
19:08 System Check
Rates from ucesb below 1 kHz
Atachment 1 Stats
Atachment 2 Temperatures
Atachment 3 Voltages
System wide checks are all ok except
ADC calibration all failed
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dba : 2
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA all passed
Memory information
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 6 2 2 1 2 4 3 4 2 3 7 : 40424
aida02 : 3 4 2 2 1 2 1 3 3 3 7 : 40140
aida03 : 10 10 0 3 2 4 2 4 2 3 7 : 40280
aida04 : 3 2 0 2 2 3 2 3 1 4 7 : 40540
aida05 : 4 9 2 3 2 4 3 3 1 4 7 : 41048
aida06 : 8 6 1 1 2 3 2 2 3 3 7 : 40064
aida07 : 1 4 0 1 1 3 1 3 3 3 7 : 40196
aida08 : 1 4 2 2 2 4 3 4 2 3 7 : 40452
aida09 : 8 5 2 2 1 2 1 3 3 3 7 : 40168
aida10 : 22 5 4 2 1 3 1 4 2 3 7 : 39872
aida11 : 2 1 2 1 2 2 3 3 2 3 7 : 39632
aida12 : 8 5 2 2 1 3 1 4 2 3 7 : 39784
aida13 : 18 8 2 1 0 2 3 3 2 3 7 : 39624
aida14 : 23 12 2 2 1 2 2 4 2 3 7 : 40028
aida15 : 3 5 3 2 2 3 1 4 2 3 7 : 39844
aida16 : 0 2 1 2 1 4 2 4 2 3 7 : 40096
|
|
Thu May 13 00:51:40 2021, Muneerah and OH, Thu 13 May 00:00-08:00 7x
|
01:50 Beam on
The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing.
Tempetratures OK
Rates OK
Attachment 1 Voltages
Atachment 2 Rates
Atachment 3 Temperatures
Attachment 4 ucesb rates less than 1 kHz
System wide check are ok except:
Base Current Difference
aida05 fault 0x36ca : 0x36ce : 4
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida13 fault 0xa : 0xe9 : 223
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760
aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644
aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188
aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404
aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104
aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128
aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500
aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672
aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588
aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460
aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348
aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000
aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840
aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996
aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280
aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716
<strong>05:00 OH Took over at 04:00</strong>
System wide checks ok except
Base Current Difference
aida05 fault 0x36ca : 0x36d0 : 6
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x1 : 1
aida13 fault 0xa : 0xe9 : 223
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
05:41 AIDA 05 has dropped out. Will restart the system
05:58 DAQ recovered from powercycle
p+n strips at 0xa
n+n strips at 0x20
ADC control register at 0xFF
All waveforms turned off in LED
Statistics - attachment 5
Correlation scaler in FEE 3 and 4 are mostly tracking 1:1
06:34 Local users have been unable to recover MBS
And have been unable to contact local experts
07:22 Bias and leakage currents ok - attachment 6
Stats attachment -7 |
|
Mon Dec 9 12:28:01 2024, NH, CC, MP, Test of BB7 on AIDA Electronics
|
BB7 connected to aida07, sitting in a cardboard box
Update Tcl/Tml files in ASIC, ASIC4 and sys.tml frm PJCS email to allow setting per-ASIC polarity on FEE64
Restart MIDAS@8015
Power on all FEEs
Note nnrpi is not responding to ssh... MP investigates if it ahs been unplugged
No, but a powercycle worked and fixed it
See error: /MIDAS/FEE_ASIC/aida06/ASICs.txt not found, create the folders
We see some data in the L spectra, and changing the settings from one ASIC seems to work properly
Stopped for CC/MP to determine the which ASICs correspond to p+n and n+n
Follow up: Alpha data overnight or source? |
|
Wed Jul 23 15:56:00 2025, CC, MP, NK, Test of AIDA05 with new bias module
|
Attachment1: AIDA05.1.8L when AIDA05 connected, biased at 101 V with shielded cables. BB7 is still connected but not biased (same configuration as the previous elog). The channel of the CAEN power supply #0 is connected to ASIC08 and biased at 100V. The channel of the CAEN power supply #1 is disconnected but still biased at 100V. AIDA04 (chained to AIDA05) is biased via a shielded cable to an iseg NHQ 246L power supplied, biased at 101 V with 16 uA.
All the outside drain wires of bplast are now taped to the snout.
The red markers of the cursors of the previous test have been left for ease of comparison.
FWHM: 21699-21644 = 55 channels (to be compared with the previous elogs between 73 channels and 79 channels)
Conclusion: a significant improvement can be observed with a cleaner power supply. |
|
Wed Jul 23 15:04:26 2025, CC, MP, NK, Test of AIDA05 with BB7 connected to Mesytec but not biased
|
Attachment1: AIDA05.1.8L when AIDA05 connected, biased at 100 V with not-shielded cables, after the DEGAS platform has been closed, the 2 tiles of BB7 have been connected to the mesytec preamps. BB7 is not biased. The mesytec preamps have been fully connected and supplied but 6/8 black cables are not connected on the other side. 4X and 4Y are connected to the 2 FEBEX cards.
The 2 drain wires of bplast that were hanging on the bottom are now taped to the snout, not making any antena effect anymore.
FWHM: 79 channels (to be compared with the previous elogs between 73 channels and 76 channels)
Conclusion: no significant differences can be observed when BB7 is connected. The tail on the left side of the spectrum in twice as small, most likely because on the now-taped drain wires of bplast |
|
Sun May 9 19:29:23 2021, TD, Sunday 9 May 14x
|
|
|