 Thu Jun 12 13:15:25 2025, GB, CC, NK, MP, TD, Thursday 12 June 2025 
|
EMC tests with NK (Orsay)
Stack configuration (upstream->downstream)
bPlas
AIDA DSSSD #0
AIDA DSSSD #1
bPlas
3x MSL type BB7(DS)-1000 (1x aida10, 2x Mesytec preamplifiers)
AIDA DSSSD #0 bias -100V leakage current -13.23uA
AIDA DSSSD #1 bias -100V leakage current -9.68uA
BNC PB-5 settings
Amplitude 1.0V
Attenuator x1
Frequency 25Hz
Polarity +
tau_d 1ms
Tail pulse
14.39 per p+n FEE64 1.8.W spectra - 20us FSR - attachment 1
per p+n FEE64 1.8.L spectra - attachment 2
Pulser peak widths ch FWHM
DSSSD #0 (DSSSD - FEE64 cables with heavy duty braid - electrically connected to ground of adaptor PCB and internally connected to ground within snout)
9 1 5
120 201 21
15 3 12
114 180 broad
DSSSD #1 (DSSSD - FEE64 cables with mesh - currently not connected i.e. floating to adaptor PCB and internally connected to ground within snout)
10 14 13
- 22 374
11 7 16
broad 417 20
Pulser peak widths c. 20 ch FWHM probably indicate cable from DSSSD *not* connected to FEE64 adaptor PCB
|
|
Fri Mar 13 15:48:11 2020, Friday 13th 16:00 - 24:00, OH, TD 11x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
16:51 Bias and leakage currents ok but starting to increase in DSSD 1 predominantly - attachment 1
Statistics ok - attachment 2
FEE temp ok - attachment 3
System wide checks all ok
16:57 Histograms zeroed
17:17 ASIC control check done
17:40 Layout 1 - attachment 4
Layouts 3 and 4 - attachments 4 and 5
Layouts 5 and 6 - attachments 6 and 7
19:06 Earlier in the shift two extra 8TB drives were formatted and mounted
Named SecondDrive and ThirdDrive
TapeData folder created on SecondDrive
Plan is before the end of this shift to move the symbolic link from the filling up drive to the new drive
Current prediction is drive would otherwise be full by 12:20 tomorrow.
Playing it safe and switching early
21:57 Bias and leakage ok - attachment 8
Statistics ok - attachment 9
FEE Temps ok - attachment 10 |
|
Thu Jun 13 23:04:01 2024, Dan Judson, TD, 0.00-8.00 14/6/24 40x
|
0.00 checks - all looks ok
Voltage/currents - attachment 1
Rates attachment 2 + 3
Temps - attachment 4
Merger - attachment 5
ucesb - attachment 6
Grafana - attachment 7
1.30 -aida1 in merger Links, aida02 in rate spectra, has stopped - resetting DAQ.
Datafile R7_961 being written when stopped
1.40 - seems ok
1.50 checks - seems ok
Voltage/currents - attachment 8
Rates attachment 9 + 10
Temps - attachment 11
Merger - attachment 12
ucesb - attachment 13
Grafana - attachment 14
3.30 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.
Datafile R7_1010 being written when stopped
3.38 - seems ok
3.40 checks - all looks ok
Voltage/currents - attachment 15
Rates attachment 16 + 17
Temps - attachment 18
Merger - attachment 19
ucesb - attachment 20
Grafana - attachment 21
Zeroed spectra - attachments 22-28
5.39 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.
Datafile R7_1062 being written when stopped
AIDA out of the main data
Could not get it to restart folowing the notes on the elog - SOAP errors as shown in attachment 29. called Tom who''s logged in
6.10 - seems to be working again. Not sure what happened. Tom thinks multiple restarts fixed it
Started data file R8
AIDA back in the main data
FRS shifters report a problem with the AIDA white rabbit time stamp - attachment 30
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 17D8C4ED 8F46CF7B , WR/10=> 2627A17C18714BF, Readout Time => 2627A17C2A74000
aida02 : White Rabbit=> 17D8C4ED A0A1B623 , WR/10=> 2627A17C3435F03, Readout Time => 2627A17C4410000
aida03 : White Rabbit=> 17D8C4ED AFDACE68 , WR/10=> 2627A17C4C914A4, Readout Time => 2627A17C5D3C000
aida04 : White Rabbit=> 17D8C4ED C0516FC2 , WR/10=> 2627A17C66E8B2D, Readout Time => 2627A17C7AC4000
aida05 : White Rabbit=> 17D8C4ED D3B93DAB , WR/10=> 2627A17C85F52F7, Readout Time => 2627A17C9AEC000
aida06 : White Rabbit=> 17D8C4ED E8C9B243 , WR/10=> 2627A17CA7A91D3, Readout Time => 2627A17CB704000
aida07 : White Rabbit=> 17D8C4ED FFE475FD , WR/10=> 2627A17CCCA0BCC, Readout Time => 2627A17CDD94000
aida08 : White Rabbit=> 17D8C4EE 103232A6 , WR/10=> 2627A17CE6B6B77, Readout Time => 2627A17CF7BC000
aida09 : White Rabbit=> 17D8C4EE 20A1F612 , WR/10=> 2627A17D0103235, Readout Time => 2627A17D112C000
aida10 : White Rabbit=> 17D8C4EE 307D8FE7 , WR/10=> 2627A17D1A627FD, Readout Time => 2627A17D2958000
aida11 : White Rabbit=> 17D8C4EE 4402829D , WR/10=> 2627A17D399D9DC, Readout Time => 2627A17D4570000
aida12 : White Rabbit=> 17D8C4EE 6288B04D , WR/10=> 2627A17D6A744D4, Readout Time => 2627A17D7D94000
aida13 : White Rabbit=> 17D8C4EE 751E1306 , WR/10=> 2627A17D88301E7, Readout Time => 2627A17D9C34000
aida14 : White Rabbit=> 17D8C4EE 894E8806 , WR/10=> 2627A17DA87DA67, Readout Time => 2627A17DBFF4000
aida15 : White Rabbit=> 17D8C4EE 9EEEB997 , WR/10=> 2627A17DCB178F5, Readout Time => 2627A17DDC18000
aida16 : White Rabbit=> 17D8C4EE AF0BC923 , WR/10=> 2627A17DE4DFA83, Readout Time => 2627A17DF5C8000
Tom thinks everything appears ok with AIDA
Have converted WR timestamp to data/time - see attachment 40
At the ~second timescale AIDA WR timestamp looks OK, i.e. no gross errors. Will need to check correlations with other detector sub-systems to confirm WR on the ~us time timescale..
However, ~10us offset reported by DESPEC online crew is correct (AIDA timestamps data at a later point in the signal processing cycle than other detector sub-systems). Have suggested online crew contact Nic or Calum to check what the AIDA WR timediff spectra should look like.
7.00 checks - all looks ok
Voltage/currents - attachment 31
Rates attachment 32 + 33
Temps - attachment 36
Merger - attachment 37
ucesb - attachment 39
Grafana - attachment 38
Handed over to Tom
|
|
Mon Jun 10 07:15:48 2024, Dan, 08.00-16.00 Monday 10th June 50x
|
8am checks
temps ok - attachment 2
rates ok - attachment 8
voltage ok - attachment 5
merger ok - attachment 6
9am checks
temps ok - attachment 10
rates ok - attachment 9
voltage ok - attachment 12
merger ok - attachment 11
(Ignore Grafana plots 13-43 inc - not refreshed)
9.16 beam stopped to optimise beam intensity
10.00 checks - still no beam
temps ok - attachment 17
rates ok - attachment 15+16
voltage ok - attachment 14
merger ok - attachment 18
11.00 checks - still no beam
temps ok - attachment 23
rates ok - attachment 21+22
voltage ok - attachment 20
merger ok - attachment 24
12pm checks
temps ok - attachment 28
rates ok - attachment 26+27
voltage ok - attachment 29
merger ok - attachment 30
12.45 - beam back
temps ok - attachment 34
rates ok - attachment 32+33
voltage ok - attachment 35
merger ok - attachment 36
2pm checks
temps ok - attachment 40
rates ok - attachment 38+39
voltage ok - attachment 41
merger ok - attachment 42
/tmp/R4_276 added - attachment 44
max deadtime 10.5% (aida08)
n+n FEE64s deadtime
3pm checks
temps ok - attachment 48
rates ok - attachment 46+47
voltage ok - attachment 45+50
merger ok - attachment 49 |
|
Fri Apr 16 15:52:08 2021, DSJ , 16 April 16.00 shift 12x
|
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 10 3 1 1 2 1 3 3 3 6 : 36156
aida02 : 25 10 4 3 1 2 2 3 3 3 6 : 36500
aida03 : 27 8 5 2 3 4 3 3 2 3 6 : 36092
aida04 : 30 4 4 5 0 2 4 2 3 3 6 : 36472
aida05 : 19 5 7 1 3 2 2 2 2 4 6 : 37060
aida06 : 18 13 1 2 1 3 2 3 3 3 6 : 36544
aida07 : 24 7 3 2 2 3 1 3 2 4 6 : 37384
aida08 : 15 11 2 5 2 4 1 4 3 3 6 : 37076
aida09 : 7 3 4 1 5 6 5 4 3 2 6 : 36308
aida10 : 8 3 13 4 3 3 1 4 2 3 6 : 36040
aida11 : 18 11 17 7 4 4 4 3 2 3 6 : 36752
aida12 : 12 7 7 7 4 4 2 3 2 3 6 : 36024
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
17:27. Checked system stats, temps, leakage etc. looks ok
18:49. Tape server writing to disk /NULL/R21_9+
/TapeData points to /media/SecondDrive/ - 3.6Tb free
19.15. checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 9 16 2 2 2 1 3 3 3 6 : 36424
aida02 : 1 3 5 4 1 3 3 2 3 3 6 : 36268
aida03 : 24 13 11 6 3 3 2 4 2 3 6 : 36472
aida04 : 5 4 4 4 3 3 3 2 3 3 6 : 36404
aida05 : 17 8 4 2 2 2 2 2 2 4 6 : 36996
aida06 : 26 11 3 2 0 3 2 3 3 3 6 : 36528
aida07 : 24 7 2 1 1 1 2 3 2 4 6 : 37272
aida08 : 13 12 6 3 3 4 1 4 3 3 6 : 37140
aida09 : 22 15 19 2 6 5 4 4 3 2 6 : 36416
aida10 : 16 11 6 4 3 3 1 4 2 3 6 : 36024
aida11 : 12 4 5 4 2 3 3 3 2 3 6 : 35872
aida12 : 18 12 9 4 4 4 2 3 2 3 6 : 36024
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
19.53. Checked system stats, temps, leakage etc. looks ok
20:30. Checked system stats, temps, leakage etc. looks ok
20:59. Checked system stats, temps, leakage etc. looks ok
21:31 System checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb40 : 6
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 21 9 17 3 2 2 1 3 3 3 6 : 36492
aida02 : 5 1 1 6 0 2 2 3 3 3 6 : 36332
aida03 : 2 10 11 6 4 4 3 3 2 3 6 : 36296
aida04 : 27 12 3 4 0 2 3 2 3 3 6 : 36220
aida05 : 19 9 7 2 1 2 2 2 2 4 6 : 36996
aida06 : 28 5 0 4 0 3 1 3 3 3 6 : 36248
aida07 : 19 13 10 6 2 2 1 3 2 4 6 : 37524
aida08 : 18 9 6 0 1 4 1 3 3 3 6 : 36400
aida09 : 17 19 12 3 6 5 4 4 3 2 6 : 36348
aida10 : 23 9 7 4 2 3 1 4 2 3 6 : 35988
aida11 : 19 10 3 3 2 3 3 3 2 3 6 : 35884
aida12 : 12 6 0 3 2 4 2 3 2 3 6 : 35648
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
22.06. Checked system stats, temps, leakage etc. looks ok
22.18CEST BNC PB-5 amplitude changed from 1V to 2V
fioler NULL/R21_67
22.23CEST All histograms zero'd
22.43. Checked system stats, temps, leakage etc. looks ok
23.21. Checked system stats, temps, leakage etc. looks ok
23.46 system checks
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb40 : 6
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 17 12 7 0 3 3 2 4 2 3 6 : 36180
aida02 : 35 3 5 2 0 2 1 3 3 3 6 : 36148
aida03 : 22 6 10 2 1 3 2 4 2 3 6 : 36136
aida04 : 27 9 3 5 2 4 3 3 2 3 6 : 36100
aida05 : 20 9 3 3 2 2 2 2 2 4 6 : 37032
aida06 : 25 11 1 2 0 3 1 3 3 3 6 : 36236
aida07 : 20 6 6 2 3 3 2 4 1 4 6 : 37216
aida08 : 25 10 0 2 1 3 1 3 3 3 6 : 36276
aida09 : 10 4 3 4 4 5 4 4 3 2 6 : 35960
aida10 : 26 9 3 0 1 3 1 4 2 3 6 : 35744
aida11 : 2 3 4 2 2 3 3 3 2 3 6 : 35744
aida12 : 4 9 7 2 2 4 2 3 2 3 6 : 35720
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Fri Apr 16 01:00:12 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021 |
|
Fri Apr 23 15:13:28 2021, DSJ, 23 April 16.00-20.00 shift 6x
|
16:00 taking over from Muneerah
16:15 - temp, ucesb, database, stats etc. look ok.
16:41 - Nic restarting MBS
17:05 - temp, ucesb, database, stats etc. look ok.
17:43 - temp, ucesb, database, stats etc. look ok.
18.10 - system wide checks. All same as previous checks - eg elog 264 + 266
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Check FPGA Timestamp Error - HTML errors
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 7 4 7 4 3 2 4 2 3 6 : 36388
aida02 : 1 2 5 2 0 4 3 4 3 4 3 : 27044
aida03 : 28 4 4 4 4 4 2 2 3 3 6 : 36432
aida04 : 5 5 2 6 2 3 3 3 3 3 6 : 36892
aida05 : 24 4 7 3 3 3 2 2 3 3 6 : 36240
aida06 : 21 6 6 3 3 4 2 2 3 3 6 : 36356
aida07 : 15 7 4 3 3 4 2 2 3 3 6 : 36308
aida08 : 23 6 7 1 3 4 1 3 3 3 6 : 36572
aida09 : 1 8 2 1 1 2 4 2 3 3 6 : 36292
aida10 : 3 4 6 2 2 2 3 3 2 3 6 : 35660
aida11 : 15 7 4 1 2 3 2 4 2 3 6 : 36052
aida12 : 25 13 5 2 2 3 2 3 3 3 6 : 36700
18:45 - temp, ucesb, database, stats etc. look ok.
19:03 - temp, ucesb, database, stats etc. look ok.
19:26 - temp, ucesb, database, stats etc. look ok.
TapeData (df -h) - 7.2T 3.9T 2.9T 58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
19:30 - beam being shared with R3B, implantation rates fluctuation by~ 50% back to normal 19:40
19:50 system checks - all as previous
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Check FPGA Timestamp Error - HTML errors
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 25 6 4 7 4 3 2 4 2 3 6 : 36404
aida02 : 20 7 5 1 0 4 2 3 2 3 4 : 27384
aida03 : 28 6 3 5 3 5 2 2 3 3 6 : 36528
aida04 : 19 8 4 6 2 4 4 2 3 3 6 : 36876
aida05 : 7 3 8 3 4 3 1 3 3 3 6 : 36500
aida06 : 25 8 4 3 3 4 2 3 3 3 6 : 36868
aida07 : 18 7 3 4 3 4 2 2 3 3 6 : 36336
aida08 : 16 10 6 2 3 3 1 3 3 3 6 : 36464
aida09 : 0 3 1 1 1 3 4 2 3 3 6 : 36360
aida10 : 8 1 10 0 2 2 2 4 2 3 6 : 35912
aida11 : 2 2 2 3 3 4 3 3 2 3 6 : 35928
aida12 : 12 7 4 1 3 3 2 3 3 3 6 : 36616
|
|
Mon May 17 23:17:16 2021, DSJ, Tuesday 18 May 0.00-8.00 shift 39x
|
0.00 - Taking over from Lewis
00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2)
01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4)
01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6)
02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8)
Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11)
System Wide Checks:
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 0, Failed 16
Base Current Difference
aida05 fault 0x4da : 0x4ee : 20
White Rabbit error counter test result: Passed 15, Failed 1
FPGA Timestamp error counter test result: Passed 16, Failed 0
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 43 12 6 4 1 4 2 2 1 3 7 : 38444
aida02 : 21 12 4 4 2 3 3 3 3 3 6 : 36980
aida03 : 23 8 4 2 2 4 2 3 3 3 6 : 36764
aida04 : 34 12 10 5 1 4 1 4 3 3 6 : 37224
aida05 : 18 5 5 1 2 3 2 2 1 3 7 : 38112
aida06 : 40 12 5 7 2 4 2 4 3 3 6 : 37552
aida07 : 40 10 6 2 2 4 2 2 2 2 7 : 37392
aida08 : 19 11 3 3 1 4 3 3 3 3 6 : 36980
aida09 : 43 6 2 4 3 4 3 3 3 3 6 : 37180
aida10 : 32 14 10 3 4 1 2 4 1 3 7 : 39280
aida11 : 40 7 2 5 4 3 1 4 3 3 6 : 37144
aida12 : 37 14 13 2 2 3 2 4 3 3 6 : 37396
aida13 : 18 9 3 1 1 5 3 3 3 3 6 : 37024
aida14 : 36 13 6 4 2 3 2 3 1 3 7 : 38872
aida15 : 32 11 7 4 3 4 2 3 2 4 6 : 38024
aida16 : 26 9 4 1 0 3 1 4 3 3 6 : 36752
02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13)
03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15)
03.05 - No beam, problem with SIS, don't know how long will be off
03.14 - Beam is back
03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17)
04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19)
Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22)
System Wide Checks:
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 0, Failed 16
Base Current Difference
aida05 fault 0x4da : 0x4f2 : 24
White Rabbit error counter test result: Passed 15, Failed 1
FPGA Timestamp error counter test result: Passed 16, Failed 0
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 11 5 6 1 4 2 2 1 3 7 : 38416
aida02 : 16 9 5 4 2 4 2 2 3 3 6 : 36312
aida03 : 9 7 4 4 2 4 2 3 3 3 6 : 36764
aida04 : 36 11 8 4 3 4 1 3 4 3 6 : 37800
aida05 : 20 4 3 2 2 3 2 2 1 3 7 : 38112
aida06 : 22 13 10 7 2 4 3 3 4 3 6 : 38336
aida07 : 28 7 7 2 2 4 2 2 2 2 7 : 37336
aida08 : 2 3 4 4 1 3 2 3 3 3 6 : 36512
aida09 : 40 7 1 4 4 3 3 3 3 3 6 : 37096
aida10 : 26 9 12 2 3 2 2 4 1 3 7 : 39280
aida11 : 30 7 1 5 4 3 1 3 2 4 6 : 37600
aida12 : 15 10 6 4 2 3 2 4 3 3 6 : 37228
aida13 : 1 3 2 1 2 4 3 2 3 3 6 : 36316
aida14 : 34 12 7 4 2 3 2 3 1 3 7 : 38872
aida15 : 18 11 5 3 4 4 2 3 2 4 6 : 37968
aida16 : 22 7 4 2 0 3 1 4 3 3 6 : 36752
04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24)
05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26)
05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28)
06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30)
Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33)
System Wide Checks:
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 0, Failed 16
Base Current Difference
aida05 fault 0x4da : 0x4f9 : 31
White Rabbit error counter test result: Passed 15, Failed 1
FPGA Timestamp error counter test result: Passed 16, Failed 0
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 16 9 3 7 1 5 1 3 1 3 7 : 38744
aida02 : 18 3 5 2 3 3 3 2 3 3 6 : 36400
aida03 : 21 9 4 2 2 4 2 4 3 3 6 : 37276
aida04 : 32 12 7 3 3 4 1 3 4 3 6 : 37744
aida05 : 12 8 3 2 2 4 2 2 1 3 7 : 38240
aida06 : 28 14 4 8 2 4 3 3 3 3 6 : 37280
aida07 : 40 12 5 2 2 4 1 3 2 2 7 : 37648
aida08 : 13 8 4 4 1 3 2 3 3 3 6 : 36596
aida09 : 35 5 2 4 3 4 2 4 3 3 6 : 37396
aida10 : 6 10 4 1 1 3 2 2 1 3 7 : 38024
aida11 : 41 9 1 4 4 3 2 4 3 3 6 : 37372
aida12 : 29 11 9 4 2 3 2 4 3 3 6 : 37340
aida13 : 24 10 5 1 2 4 3 2 3 3 6 : 36512
aida14 : 34 16 5 4 2 3 2 3 1 3 7 : 38872
aida15 : 42 8 6 2 4 4 2 3 2 4 6 : 38024
aida16 : 8 9 6 1 0 3 1 4 3 3 6 : 36712
06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35)
07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37)
07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39) |
|
Mon Apr 22 15:17:07 2024, DSJ, 16.00-0.00 22/04/24 26x
|
First checks of shift. All looks ok.
Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.
Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)
checks at 16.00 - merger has crashed - aida01 dropped out - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09
Reset 01, restarted DAQ, and restarted as R16.
16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.
Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms
RUN17 STARTED 16.51|
HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.
20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift
21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps
22.48 - things apear ok. (Attachments 23-26) |
|
Sat Mar 14 07:13:59 2020, DK, TD, LS, CG, Saturday 14th March 08:00 - 24:00 41x
|
8:15
All system wide checks passed
Leak currents are continuing to creep up - attachment #1
FEE temperatures fine - attachment #2
Stats about where they have been - attachment #3
Implants look as previously (5ish counts per channel in SSD1 and SSD2) - attachment #4
Merger fine, 3 Mega items / sec
TapeServer still writing about 25 MB/s
09.37 analysis of data file S480/R12_443
deadtime aida09 c. 5%, all other FEE64s <<1%
See attachment #5
09.54.51
biases, SSD1 and 2 over 2 uA each now - attachment #6
11:20
All system-wide checks okay
Merger okay, 3 Mega items / sec
TapeServer normal, 27 MB/sec
Biases as attachment #7
Stats as #8
Temps as #9
Histos as #10
14:10
All system wide checks okay
Stats #11
Temps #12
Implants #13
Leak currents are getting a bit higher #14
15:30
All system wide checks passed
Stats attachment #15
Temps attachment #16
Implants attachment #17
Biases / leak currents attachment #18
Merger okay, 3 Mega items / sec
TapeServer okay, writing 27 MB/s
16.12 all online spectra zero'd
16.31 stat & rate spectra - attachments 19-22
1.8.L spectra - attachments 23 & 24
aida03 pulser peak 57 ch FWHM
1.8.H spectra - attachments 25 & 26
NewMerger & TapeServer - attachments 27 & 28
18.29
system wide checks all okay
FEE temperatures normal (attachment 29)
Leakage currents okay and recorded to spreadsheet (attachment 30)
Good event statistics okay (attachment 31)
Merger running at 3M items/sec
Tape server running at 25MB/sec
20.25
system wide checks all okay
FEE temperatures normal (attachment 32)
Leakage currents okay and recorded to spreadsheet (attachment 33)
Good event statistics okay (attachment 34)
Merger running at 3M items/sec
Tape server running at 26MB/sec
as of 20.39 just finished R12_961
21.36 beam is down confirmed by rates histogram (attachment 35), informed others on shift
was down for maybe 5 minutes is back on at 21.40 (attachment 36)
think between R12_1004 to R12_1008
22.10
system wide checks all okay
FEE temperatures normal (attachment 37)
Leakage currents okay and recorded to spreadsheet (attachment 38)
Good event statistics okay (attachment 39)
Merger running at 3M items/sec
Tape server running at 26MB/sec
23.03 someone comes to ask if we see an increase in implantation rate as they do their side
I checked the rates histogram and good events and see nothing that would indicate an increase (attachment 40 and 41)
FEE temperatures and leakage currents do not seem any different either
believe they are calling someone about the issue
23.30 leaving now to catch bus but people on night shift have been told how to check on AIDA and know that CG is on call if needed |
|
Sat Dec 7 02:39:02 2019, DK, NH et al., 34Si Fragments PID
|
See attached
Tentative PID (DK) as attachment #3
DESPEC LISE++ file as attachment #4 |
|
Tue Mar 10 23:17:31 2020, DK, LS, PW, TD, CA, Wednesday 11 March 2020 43x
|
00:15 Shift change near AIDA R6_475
We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.
All system-wide checks passed successfully
Temperatures ok (aida01 still a bit hot) - see attachment #1
Stats as normal (SSD3 still noisy) - see attachment #2
Bias and pulser settings remain sane - see attachment #3
Writing data to disk at ~26 MB/s as yesterday - see attachment #4
Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
Rates look not unreasonable for expected implantation in short time window - see attachment #6
2:26
We are near R6_574
All system checks are fine
Temps are fine - attachment #7
Stats look as before - attachment #8
Biases and leaks typical - attachment #9
Merger fine, 3 x 10^6 items / sec (skip screencap)
TapeServer still sits near 25 MB/s (skip screencap)
Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10
2:44 R6_587 or R6_588
The spills seem to have stopped based on the beeping scalars. Confirmed with other group there are no scint counts.
See attachment of AIDA rates where no implants can be seen. See attachment #11
2:46 Spill beeps heard briefly, but they cut again. R6_590
2:49 Beam seems to be back R6_592
Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants. No problem. See attachment #12 for implants.
4:35
System checks all pass (except expected error on master clock)
Temps okay - attachment #13
Biases and leak currents normal - attachment #14
Stats as before - attachment #15
Merger going, 3 mega items per sec
TapeServer going, 25 MB/s
5:59 Beam is gone again, near R6_738. As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
See attachment #16 which shows no implants in the rate.
Contact phones for Control Room: 2222
We called, they are aware of a problem with the magnets and they are working to resolve it. So we will await them.
Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.
We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off
7:14 Beam is still off, but I make some notes
System wide checks all okay
Temps as normal (see attachment #17)
Biases and leak currents okay (see attachment #18)
Stats as normal (see attachment #19)
Merger going at 3 mega cps
TapeServer on, around 25 MB/sec
Implants not seen in histogram rate as expected (see attachment #20)
7:32 We called the operators to confirm that they don't change anything before calling us first.
Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.
8.00 shift change at file R6_823
beam is back on, no settings changed.
particles per spill is similar to before (2k~3k per spill)
implants now being seen in rate histogram (attachment 21)
8.32 system wide checks all okay
FEE temperatures are normal (attachment 22)
leak currents are okay and have been recorded to sheet (attachment 23)
good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
merger rate is running around 3M items/sec
tape service rate is running around 26MB/sec
9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
remaking each histogram and saving layout6 seems to be fixed (attachment 27)
Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
invoked from within
"SpecDetails 1.8.h"
(in namespace inscope "::urn:SpectrumService" script line 1)
invoked from within
====
10.05 beam is being stopped to change some thresholds
10.34 system wide checks all okay
FEE temperatures are normal (attachment 28)
leak currents are okay and have been recorded to spreadsheet (attachment 29)
good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 24MB/sec
implants not seen in histogram rate as expected as beam still down (see attachment 30)
10.42 beam still down but taking data, currently file finishing R6_946
as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
have zeroed all histograms so will take screenshots at 11.00
11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks
11.25 beam back file S480/R6_975
12.00 analysis file S480/R6_1001 - attachment 33
FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
all HEC data rate c. 1kHz
all ADC data rate c. 1.2MHz
12.23 system wide checks all okay
FEE temperatures are normal (attachment 34)
leak currents are okay and have been recorded to spreadsheet (attachment 35)
good event stats are okay (attachment 36)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 26MB/sec
13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
seems to be intermittent. file R6_1051
13.10 beam back on, no one was called. currently on file R6_1061
13.22 beam off as going to higher intensity beam. file R6_1069
14.42 system wide checks all okay
FEE temperatures are normal (attachment 38)
leak currents are okay and have been recorded to spreadsheet (attachment 39)
good event stats are okay (attachment 40)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 24M/sec
14.48 on file R6_1131
16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
file R6_1193
16:35 compression of files R6_[200-999] commenced (nice -20)
16:40 system wide checks all okay
FEE temperatures are normal (attachment 41)
leak currents are okay and have been recorded to spreadsheet (attachment 42)
good event stats are okay (attachment 43) |
|
Thu Mar 12 23:44:48 2020, DK, PID Plots
|
Labelled PID plots. Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.
There was an error in the printouts for Cd isotopes, but the below attachments are corrected.
The Z-depth (color, not nuclear charge) is log scale. |
|
Fri Mar 13 08:01:53 2020, DK, Friday 13th 8:00 to 16:00 shift 8x
|
8:00 - See previous shift's elog entry. All good.
9:00
All system wide checks passed
Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
Tape Server okay - 27 MB/s
Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
Stats as usual - attachment #2
Temps normal - attachment #3
Biases and leaks fine - attachment #4
Disk usage projection
Space remaining: ~2 TB
Fill rate: ~2 GB / 1 minute
-> 1000 minutes =~ 17 hours
Still gzipping from R6_823
10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
Kbytes Written: -1986382400 Kbytes/sec: 26816
Rate is fixed at that value on update.
However, actual data files are being updated so data has been coming.
DK and NH consulted and believe the situation is fine.
We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.
10:25 Now we are on R10 after start and stop the TapeServer. Problem is resolved.
11:30
Implants: attachment #5
Stats: attachment #6
Biases / leaks: attachment #7
Temps: attachment #8
System wide checks performed, situation normal. |
|
Sat Apr 17 19:39:56 2021, DJ, TD, Saturday 17 April 20.00-00.00 13x
|
19.37 per FEE64 rate spectra - attachments 1 & 2
1.8.L spectra - attachments 3 & 4
1.8.H spectra - attachments 5-8
1.8.W spectra - attachments 9 & 10
20.42 DAQ contrinues file NULL/R30_233
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bd7 : 10
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x4 : 4
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
-------
23:48
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bde : 17
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x7 : 7
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 26 5 6 2 1 3 1 3 3 3 6 : 36336
aida02 : 7 9 10 3 2 2 2 4 2 3 6 : 36068
aida03 : 26 7 8 4 3 4 2 4 2 3 6 : 36448
aida04 : 18 8 2 1 2 2 3 4 2 3 6 : 36168
aida05 : 24 7 9 4 3 1 3 2 2 4 6 : 37352
aida06 : 17 10 3 1 3 5 1 3 3 3 6 : 36644
aida07 : 7 11 2 2 3 2 3 2 4 3 6 : 37268
aida08 : 23 4 1 5 1 1 2 3 3 3 6 : 36332
aida09 : 14 6 3 2 1 3 2 3 3 3 6 : 36504
aida10 : 2 2 2 2 1 2 2 2 3 3 6 : 35768
aida11 : 2 2 1 2 2 2 2 4 2 3 6 : 35816
aida12 : 26 8 5 5 4 3 1 3 3 3 6 : 36632
---- |
|
Sun Apr 18 07:30:20 2021, DJ, TD, Sunday 18th April 08:00-12:00 7x
|
---- 09-26
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
---
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bfc : 47
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
--
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bfc : 47
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
-
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bfc : 47
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
08.45 TD resets baseline for WR and FPGA errors
|
|
Thu Apr 22 23:17:46 2021, DJ , Friday 23rd April 00:00-08:00 27x
|
Took over from TD ... All is well.
00:45 System Wide Checks
Clock: 12 passed, 0 failed.
ADC Calibration: Passed 12, 0 failed.
White Rabbit Decoder: Passed 4,failed 8. - See screenshot.
FPGA Passed 11, failed 1. - See screenshot.
FEE64 Linux: See screenshot.
-----
02:49 run closed. Checking beam.
03:00 File 55 opened.
03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.
File 56 opened at 4:24.
---------
04:30 System Wide Checks (See screenshots)
White Rabbit Decoder: Passed 12, failed 0.
FPGA: page not loading properly.
----
05:37 System wide checks. See screenshots
Clock: Status: 11 passed. 1 fail.
ADC calibration: Passed 9, failed 3.
White Rabbit decoder: Passed 12, failed 0.
-----
07:32 System wide checks (See screenshots).
|
|
Fri Jan 31 09:37:47 2025, CC, TD, MP, Friday 31 January 26x
|
Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000
N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.
n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted
BB7 aida10 (asics #1-2 p+n, #3-4 n+n)
BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream)
10.30 FEE64 power ON
DSSSD bias ON (BB18 only) - attachment 1
leakage current OK
ASIC settings 2024Dec13-17.02.45 restored
p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf
Attachments 2-7 WR timestamp aida05 & aida12, system wide check
aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
WR decoder status aida02
per FEE64 Rate spectra - attachment 8
aida10 high rate - BB7 not biased
rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments
ADC data item stats - attachment 9
rates generally OK
aida10 high - BB7 not biased
aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!
BNC PB-5 settings - attachment 10
WR timestamps OK - attachment 11
FEE64 temperatures OK - attachment 12
12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet
ADC data item stats - attachment 13
all rates higher cf. attachment 9
per FEE64 Rate spectra - attachment 14
per p+n FEE64 1.8.L spectra - attachment 15
aida09 pulser peak width 69 ch FWHM = 49keV FWHM
aida14 pulser peak width 50 ch FWHM = 35keV FWHM
per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17
per n+n FEE64 1.8W spectra - 20us FSR - attachment 18
DSSSD bias & leakage current OK - attachment 19
ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%
FEE64 temperatures OK - attachment 20
DSSSD bias volatge & leakage current OK - attachment 21
CAEN N1419ET ch#0 BB18, ch#1 BB7
Install of bPlas drivers, cabling, grounds complete
ADC data item stats - attachment 22
per FEE64 Rate spectra - attachment 23
all BB18 p+n FEE64s *except* aida05 show very low rates of noise
per p+n FEE64 1.8.L spectra - attachment 24
aida09 pulser peak width 62 ch FWHM = 42keV FWHM
BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD
per FEE64 1.8.W spectra - 20us FSR - attachments 25-26 |
|
Fri Jan 31 17:26:47 2025, CC, TD, MP, Friday 31 January contd. 16x
|
18.18 bPlas and BB7 installs complete
Restart AIDA DAQ, Merger, Tape Server and re-test
per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 69 ch FWHM = 42keV FWHM
per FEE64 Rate Spectra - attachment 2
BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK
per FEE64 1.8.W spectra - 20us FSR - attachments 3-5
ADC data item stats - attachment 6
aida01 aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
BB7 aida10 19k
WR timestamps OK - attachment 7
Merger, TapeSever etc - attachments 8-9
disk directpry /TapeData/FEB25
working OK
DSSSD bias & leakage current OK - attachment 10
System wide checks OK - attachment 11-15
note global clock status 6 errors reported earlier today have now gone
FEE64 temperatures OK - attachment 16
18.38 Transition to safe state
DAQ STOP
disable data transfer 1
detector bias OFF
FEE64 power OFF
Can restart as follows
1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps
If all OK can re-connect to Merger/TapeServer as follows
1) DAQ STOP
2) enable data transfer 1
3) DAQ GO
Can disconnect from Merger/TapeServer as follows
1) DAQ STOP
2) disable data transfer 1
|
|
Fri Jun 13 12:42:39 2025, CC, TD, Friday 13 June 2025 10x
|
13.42 ASIC settings 2025Jun12-13.53.04
*all* FEE64s slow comparator 0x64
all fast disc outputs disabled
p+n FEE64s waveform threshold 7000, n+n FEE64s 9000
CC grounding ribbon cable heavy duty braid to adaptor PCB and AIDA support assembly ground
'top' p+n FEE64s and snout to be completed
16.00 DSSSD bias & leakage current - attachment 1
FEE64 temps OK - attachment 2
All system wide checks OK *except* WR decoder status aida02 and aida06 - attachment 3
WR timestamps OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
per p+n FEE64 1.8.L spectra - attachment 7
aida09 pulser peak width 117 ch FWHM - no change since yesterday
per FEE64 1.8.W spectra - 20us FSR - attachments 8-10
|
|
Tue Jul 22 18:00:45 2025, CC, NK, MP, FWHM aida FEE
|
AIDA05 completely disconnected, all the others FEEs64 are fully connected
Attachment 1: aida05, FWHM= 95 ch, 63 keV
Attachment 2: aida15, FWHM= 108 ch, 74 keV |
|