 Tue Jul 5 08:53:20 2022, TD, To Do
|
In no particular order
1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)
observe +/- 6V, 12V, 24V lines with/without load
try new CAEN NIM bin and/or NIM bin of different type
2) Measure actual voltages at FEE64 power connector input
OH suggests fab of power adaptor for safe observation - contact EW
3) rev B adaptor PCB
invert 125 way ERNI - check for mech conflicts
paired HV input (avoid Lemo-00 T pieces)
consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
straight jumpers
shrouded Samtec headers - consider mech issues/consequences of using eject clips too
re-visit HV filtering & separate trace ground
4) isolation transformer
as practical matter may be necessary to operate all platform from isolation transformer
consider hire of appropriate unit
need method to measure isolation - will require permit to work or equiv
5) investigate S4 area ac mains
NH discussing with GSI electricians
6) Systematic measurement of AIDA PSU noise
Spec linear AIDA FEE64 PSU
7) Redesign of snout
Return to 1mm welded box Al for lower stage of snout for added rigidity
8) Revisit calculation of cable lengths. Particularly for the triple
9)
|
|
Wed Jun 29 10:48:26 2022, NH, OH, AIDA Dismounted
|
All detectors removed from single and triple AIDA snouts
Empty snouts *and* DSSDs (in boxes) stored in NH office |
Tue Jun 28 10:11:35 2022, OH, NH, MIDAS Data Aq V10 
|
11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics
Patrick has produced V10 which removes these.
When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
WR items for 0xE - attachment 1
WR items for 0x7 - attachment 2
However we see in the NewMerger terminal the message shown in attachment 3 frequently.
Also we note that the merger time error counter is also going up.
Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
Are we having dead time issues which is causing time warps?
Does each buffer from the MIDAS Data Acq start with a full WR timestamp?
aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way? |
|
Tue Jun 28 09:37:19 2022, NH, Tues 28 June 08:00-
|
Experiment over
10:37 - Stop DAQ & Tape
S4 enters controlled access and they uncable bPlas
Will dismount AIDA snout after |
|
Mon Jun 27 23:11:47 2022, TD, Tuesday 28 June 00:00-08:00 30x
|
00:07 Zero stats & all histograms
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 -> 0xa
all waveform AD9252 ADCs disabled
all fast discs disabled
BNC PB-5 settings (to p+n FEE64s only)
amplitude 1.0V
attenuator x1
decay time 1ms
polarity +
frequency 22Hz
analysis of file S505/R5_896 - attachment 1
zero timewarps
deadtime all FEE64s << 1%
All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3
adc data item stats - attachment 4
FEE64 temps OK - attachment 5
DSSSD bias & leakage currents OK - attachment 6 & 7
00:15 Check ASIC control all FEE64s, all ASICs
02:03
analysis of file S505/R5_914 - attachment 8
zero timewarps
deadtime all FEE64s << 1%
per FEE64 1.8.H spectra - attachments 9 & 10
per p+n FEE64 1.8.L spectra - attachment 11
aida01 pulser peak width 94 ch FWHM
per FEE64 stat & rate spectra - attachments 12 & 13
All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15
adc data item stats - attachment 16
FEE64 temps OK - attachment 17
DSSSD bias & leakage currents OK - attachment 18
03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today
03:41
analysis of file S505/R5_926 - attachment 19
zero timewarps
deadtime all FEE64s << 1%
All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21
adc data item stats - attachment 22
FEE64 temps OK - attachment 23
DSSSD bias & leakage currents OK - attachment 24
07:00
analysis of file S505/R5_954 - attachment 25
zero timewarps
deadtime all FEE64s << 1%
All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27
adc data item stats - attachment 28
FEE64 temps OK - attachment 29
DSSSD bias & leakage currents OK - attachment 30 |
|
Mon Jun 27 17:03:09 2022, MA, Monday 27th June 16:00-00:00 6x
|
16:00 Took over the shift from OH no beam yet.
18:00 Still no beam yet.
Statistics, Temperature, Current are checked and attached 1-3
system wide checks same as last updated in the previoues shift.
22:00 The beam is back but not taking data yet! FRS team doing some checkings
Statistics, Temperature, Current are checked and attached 4-6
system wide checks same as last updated in the previoues shift.
23:30 beam is back and taking data |
|
Mon Jun 27 06:45:22 2022, OH, Monday 27th June 08:00-16:00
|
07:45 Spoke to David and the beam has been gone since about 05:30
Reason for the loss of beam is a vacuum issue before the FRS
They are waiting for the experts
08:31 Statistics ok - attachment 1
Temperature ok - attachment 2
Bias and leakage currrents ok - attachment 3
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
Currently on file R5_771
Analysis of file R5_770 (No beam) - attachment 4
Around 0.25% deadtime on AIDA02 rest even less
09:02 Current free HDD space 965 GB
Current tape server rate 4979 kB/s
Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours
11:41 Statistics ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5cf : 146
aida08 fault 0xf1be : 0xf2ba : 252
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
FPGA Timestamp error counter test result: Passed 7, Failed 1
Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently? |
|
Sun Jun 26 23:04:30 2022, Marc, new shift - Monday 27 June 0:00 to 8:00 12x
|
0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).
Stats & Temperatures (VIRTEX,PSU, ASICs) all ok.
At 0:30
Stats ok - Attachment 1
Temp ok - Attachement 2
HV-LC -Attachment 3
At 2:20
Stats ok - Attachment 4
Temp ok - Attachement 5
HV-LC -Attachment 6
Wide Checks:
Clock status test result: Passed 8, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration (same as before):
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
Calibration test result: Passed 0, Failed 8
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
WR decoder status:
Base Current Difference
aida07 fault 0xc53d : 0xc5c9 : 140
aida08 fault 0xf1be : 0xf2b2 : 244
White Rabbit error counter test result: Passed 6, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp check:
Base Current Difference
aida07 fault 0x2a : 0x41 : 23
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
At 4:15:
Stats ok - Attachment 7
Temp ok - Attachement 8
HV-LC -Attachment 9
Wide Checks: No change
At 7:15: (no beam since ~6am -> background run)
Stats ok - Attachment 10
Temp ok - Attachement 11
HV-LC -Attachment 12
Wide Checks: No change |
|
Sun Jun 26 08:51:20 2022, OH, NH, Sunday 26 June 08:00-24:00 27x
|
09:51 Taken over from Tom following the night shift
Experiment is still running smoothly
Compression of the files is complete up to the start of R5.
Have started compression of files in R5 which should run up to R5_499
Currently on R5_528
Statistics ok - attachment 1
Temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
System wide checks:
Base Current Difference
aida07 fault 0xc53d : 0xc594 : 87
aida08 fault 0xf1be : 0xf271 : 179
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3b : 17
FPGA Timestamp error counter test result: Passed 7, Failed 1
Current merger rate is 2E6-4E6 events per second
Current tapeserver rate is 5800 kB/s
Free HDD space is 1.1 TB
At current rates will last 54 hours which should see out the experiment
Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
Analysis of R5_528 - attachment 4
Deadtime of AIDA02 currently around 15%
11:56 Statistics ok - attachment 5
Temps ok - attachment 6
Bias and leakage currents ok - attachment 7
System wide checks:
Clocks all ok
Base Current Difference
aida07 fault 0xc53d : 0xc59c : 95
aida08 fault 0xf1be : 0xf27a : 188
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3b : 17
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of file R5_458 - attachment 8
Deadtime in AIDA02 only 11.5% in this file
TapeServer rate still 5.5 MB/s
14:36 Has been no beam for the last while or so.
With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
Stats- attachment 9
Temp - attachment 10
Bias and leakage currents ok - attachment 11
System wide checks:
Clock ok
Base Current Difference
aida07 fault 0xc53d : 0xc59d : 96
aida08 fault 0xf1be : 0xf27c : 190
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3c : 18
FPGA Timestamp error counter test result: Passed 7, Failed 1
13:30 Beam taken to change an ion source
14:57 Beam back
16:21 Statistics ok - attachment 12
Temperatures ok - attachment 13
Bias and leakage currents ok - attachment 14
System wide checks - ASIC clocks ok
Base Current Difference
aida07 fault 0xc53d : 0xc5a3 : 102
aida08 fault 0xf1be : 0xf285 : 199
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3e : 20
FPGA Timestamp error counter test result: Passed 7, Failed 1
Currently on file R5_592
Analysis of R5_591 - Attachment 15
Deadtime of AIDA02 sitting at 17%
[NH taking over for OH so he can get home]
18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
Statitics ok - attachement 16
Temps ok - attachement 17
Bias & leakage ok - attachement 18
System wide checks -
Clocks OK
ADC Calibration N/A
WR Decoder -
Base Current Difference
aida07 fault 0xc53d : 0xc5ac : 111
aida08 fault 0xf1be : 0xf28c : 206
White Rabbit error counter test result: Passed 6, Failed 2
FPGA -
Base Current Difference
aida07 fault 0x2a : 0x3f : 21
PLL OK
Currently on file R5_621
Analysis of R5_620 - Attachement 19
aida02 deadtime only 5%, all others negligible
20:20 Stats ok - attachment 20
Temps ok - attachment 21
Bias and leakage currents ok - attachment 22
System wide checks:
Clock ok
Base Current Difference
aida07 fault 0xc53d : 0xc5ad : 112
aida08 fault 0xf1be : 0xf293 : 213
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3f : 21
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R6_637 - attachment 23
22:13 Statistics ok - attachment 24
Temperatures ok - attachment 25
Bias and leakage currents ok - attachment 26
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5ba : 125
aida08 fault 0xf1be : 0xf2a2 : 228
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x40 : 22
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_658 - attachment 27 |
|
Sat Jun 25 22:48:57 2022, TD, Sunday 26 June 00:00-08:00 30x
|
23.43 Check ASIC control
Zero stats & all histograms
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 -> 0xa
all waveform AD9252 ADCs disabled
all fast discs disabled
BNC PB-5 settings (to p+n FEE64s only)
amplitude 1.0V
attenuator x1
decay time 1ms
polarity +
frequency 22Hz
All system wide checks OK *except* WR & FPGA errors - attachments 1 & 2
DSSSD bias & leakage currents OK - attachment 3
adc data item stats - attachment 4
FEE64 temps OK - attachment 5
DSSSD bias & leakage currents OK - attachment 6
analysis of file S505/R5_415 - attachment 7
zero timewarps
max deadtime aida02 c. 17%, all other FEE64s < 1%
02:36
analysis of file S505/R5_449 - attachment 8
zero timewarps
max deadtime aida02 c. 15%, all other FEE64s < 1%
per FEE64 rate & stat spectra - attachments 9 & 10
per p+n FEE64 1.8.L spectra - attachment 11
aida01 pulser peak width 99 ch FWHM
per FEE64 1.8.H spectra - attachments 12 & 13
All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15
adc data item stats - attachment 16
FEE64 temps OK - attachment 17
DSSSD bias & leakage currents OK - attachment 18
05:38
analysis of file S505/R5_483 - attachment 19
zero timewarps
max deadtime aida02 c. 16%, all other FEE64s < 2%
All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21
adc data item stats - attachment 22
FEE64 temps OK - attachment 23
DSSSD bias & leakage currents OK - attachment 24
07:27
analysis of file S505/R5_502 - attachment 25
zero timewarps
max deadtime aida02 c. 16%, all other FEE64s < 2%
All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27
adc data item stats - attachment 28
FEE64 temps OK - attachment 29
DSSSD bias & leakage currents OK - attachment 30 |
|
Sat Jun 25 17:06:42 2022, BA, AA, Saturday 25 June 2022 16:00-00:00 9x
|
Took over the night shift from Magda
18:00 : attachments 1-3
20:00 : attachments 4-6
22:00 : attachments 7-9
00:00 : attachments 10-12
|
|
Sat Jun 25 06:55:07 2022, MS, Saturday 25 June 2022 8:00-16:00 15x
|
Took over the night shift from Tom.
7:00 The beam is back.
8:00 : attachments 1-3
10:00 : attachments 4-6
12:00 : attachments 7-9
14:00 : attachments 10-12
16:00 : attachments 13-15
|
|
Fri Jun 24 22:36:45 2022, TD, Saturday 25 June 00:00-08:00 31x
|
23:36 Check ASIC contorol
Zero stats & all histograms
ASIC settings 2021Apr29-13-16-00
slow comparator 0x64 -> 0xa
all waveform AD9252 ADCs disabled
all fast discs disabled
BNC PB-5 settings (to p+n FEE64s only)
amplitude 1.0V
attenuator x1
decay time 1ms
polarity +
frequency 22Hz
analysis of file S505/R5_154 - attachment 1
zero timewarps
max deadtime aida02 c. 17%, all other FEE64s < 5%
stats - attachments 2-6
per FEE64 1.8.H spectra - attachments 7 & 8
per p+n FEE64 1.8.L spectra - attachment 9
aida01 pulser peak width 101 ch FWHM
per FEE64 stat & rate spectra - attachments 10 & 11
adc data item stats - attachment 12
FEE64 temps OK - attachment 13
DSSSD bias & leakage currents OK - attachments 14 & 15
00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17
03:03
analysis of file S505/R5_189 - attachment 18
zero timewarps
max deadtime aida02 c. 17%, all other FEE64s < 5%
all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20
adc data item stats - attachment 21
FEE64 temps OK - attachment 22
DSSSD bias & leakage currents OK - attachment 23
03:30
per FEE64 1.8.H spectra - attcahments 24 & 25
06:09 no beam
analysis of file S505/R5_221 - attachment 26
zero timewarps
all dead times << 1%
all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28
adc data item stats - attachment 29
FEE64 temps OK - attachment 30
DSSSD bias & leakage currents OK - attachment 31 |
|
Fri Jun 24 15:04:06 2022, Marc, Friday 24th June - evening shift 11x
|
16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.
Next wide check will be in about an hour.
17:00
Stats -ok - attachment 1
Temperatures ok - attachement 2
Leakage current ok but - attachment 3
ucesb screen-shot - attachment 4
Wide check completed. Nothing different.
WR status decoder status:
Base Current Difference
aida07 fault 0xc53d : 0xc547 : 10
aida08 fault 0xf1be : 0xf1e9 : 43
White Rabbit error counter test result: Passed 6, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp::
Base Current Difference
aida07 fault 0x2a : 0x2c : 2
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.
19:20 -
Stats ok - attachment 6
Temp ok - attachment 7
Leakage current ok - attachement 8
Wide check completed and same output as above.
22:00
Stats ok - attachment 9
Temp ok - attachment 10
Leakage current ok - attachement 11
Wide check completed and same output as above.
|
|
Fri Jun 24 06:56:06 2022, OH, Friday 24 June 008:00-16:00 16x
|
07:56 Took over from AM. No issues reported overnight
Stats ok - attachment 1
Temperature ok - attachment 2
Bias and leakage current ok - attachment 3
System wide checks:
Clocks ok
WR
Base Current Difference
aida07 fault 0xc4fe : 0xc53b : 61
aida08 fault 0xf0e9 : 0xf1b7 : 206
White Rabbit error counter test result: Passed 6, Failed 2
FPGA Errors
Base Current Difference
aida07 fault 0x11 : 0x29 : 24
aida08 fault 0x1a : 0x2c : 18
FPGA Timestamp error counter test result: Passed 6, Failed 2
Analysis of R3_359 - attachment 4
Dead time still around 15.9%
Merger item rate around 2E6-4E6
Tape server rate at 7750 kB/s
Current HDD free space 1.7 TB
Time left in HDD 2.59 days
Will run out of space at some point on Sunday
Have started compression of uncompressed raw data on the HDD. Using nice +10
08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
Am regularly checking the statistics
Was able to stop the DAQ by relaunching the merger.
Reset the FEEs and recovered without a powercylce
Started R4 following this stop
08:58 They have taken the beam to change the ion source for maybe 2 hours
Will stop writing data but continue forwarding to MBS
No storage ticked. Following break will be on R5
11:10 Beam is starting to come back R5 started
Stats ok - attachment 6
Temps ok - attachment 7
Bias and leakage currents ok - attachment 8
13:05 Statistics ok - attachment 9
Temps ok - attachment 10
Bias and leakage curents ok - attachment 11
System wide checks:
WR
Base Current Difference
aida07 fault 0xc53d : 0xc540 : 3
aida08 fault 0xf1be : 0xf1d0 : 18
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_33 - attachment 12
15:28 Statistics ok - attachment 13
Temperature ok - attachment 14
Bias and leakage current ok - attachment 15
System wide checks:
Base Current Difference
aida07 fault 0xc53d : 0xc546 : 9
aida08 fault 0xf1be : 0xf1e0 : 34
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x2b : 1
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_63 - attachment 16 |
|
Fri Jun 24 00:34:55 2022, AM, Friday 24 June 00:00-08:00 20x
|
01:30 Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good
03:30 Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good
05:30 Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good
07:30 Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good |
|
Thu Jun 23 15:10:17 2022, Philippos Papadakis, Thursday 23 June 16:00-24:00 15x
|
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc525 : 39
aida08 fault 0xf0e9 : 0xf166 : 125
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x1c : 11
aida08 fault 0x1a : 0x23 : 9
Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)
18:05
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc527 : 41
aida08 fault 0xf0e9 : 0xf177 : 142
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x1c : 11
aida08 fault 0x1a : 0x24 : 10
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)
20:22
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc529 : 43
aida08 fault 0xf0e9 : 0xf17f : 150
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x20 : 15
aida08 fault 0x1a : 0x25 : 11
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)
21:56
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc529 : 43
aida08 fault 0xf0e9 : 0xf186 : 157
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x21 : 16
aida08 fault 0x1a : 0x26 : 12
Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)
23:50
System wide checks.
Clock status OK
White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc52e : 48
aida08 fault 0xf0e9 : 0xf18c : 163
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x21 : 16
aida08 fault 0x1a : 0x27 : 13
FPGA Timestamp error counter test result: Passed 6, Failed 2
Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15) |
|
Thu Jun 23 06:56:02 2022, OH, Thursday 23 June 08:00-16:00 11x
|
07:56 Taken over from AM
Current White Rabbit Error status:
Base Current Difference
aida07 fault 0xc4fe : 0xc51a : 28
aida08 fault 0xf0e9 : 0xf132 : 73
White Rabbit error counter test result: Passed 6, Failed 2
FPGA error status:
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
FPGA Timestamp error counter test result: Passed 6, Failed 2
Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link
Statistics ok - attachment 1
FEE temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
Analysis of file R3_93 - attachment 4
Still deadtime in aida02 of ~15%
Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
Currently 2.2 TB of space on /media/SecondDrive
Current data rate of 6.23MB/s
At current data rate this should last 4 days
08:27 Currently no beam
Problem with Alvarez 3 now
Tomorrow will have a 1 hour break due to changing of the source
09:36 Beam back
09:59 System wide checks
Base Current Difference
aida07 fault 0xc4fe : 0xc51b : 29
aida08 fault 0xf0e9 : 0xf138 : 79
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x11 : 0x18 : 7
aida08 fault 0x1a : 0x20 : 6
Stats ok - attachment 5
Temperatures ok - attachment 6
Bias and leakage currents ok - attachment 7
11:14 FRS group is changing from Hg setting back to Pt
11:36 Back on Pt R3_129
14:14 Stats ok - attachment 8
Temperature ok - attachment 9
Bias and leakage currents ok - attachment 10
System wide checks:
WR:
Base Current Difference
aida07 fault 0xc4fe : 0xc523 : 37
aida08 fault 0xf0e9 : 0xf156 : 109
White Rabbit error counter test result: Passed 6, Failed 2
FPGA
Base Current Difference
aida07 fault 0x11 : 0x1b : 10
aida08 fault 0x1a : 0x20 : 6
Analysis of R3_156 - attachment 11 |
|
Wed Jun 22 23:18:45 2022, AM, OH, TD, MA, Thursday 23 June 19x
|
00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
Attempted to recover restarting merger but this caused 01 to drop out.
00:10 DAQ recovered and AIDA01 recovered with a reboot
01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good
03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good
05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good
07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good |
|
Wed Jun 22 15:09:37 2022, MA, Wednesday 22 June 16:00-00:00 12x
|
16:00 Take over from Magda mrning shift
Beam was off of sometime for tunning to Mercurey setting.
17:00 attached 1-3
19:00 attached 4-6
21:00 attached 7-9
23:30 10-12
|