Auther: Muneerah and Betool
02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok
Statistics ok - 210517_0202_Stats
Temperatures ok - 210517_0202_Temp
Bias and leakage ok - 210517_0200_Bias
02:05 System wide checks:
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1552 : 0x155d : 11
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 5 8 3 3 3 5 2 2 2 3 7 : 39460
aida02 : 22 4 5 0 3 2 2 3 2 3 7 : 39560
aida03 : 21 10 2 1 1 3 2 4 2 3 7 : 40100
aida04 : 18 7 2 0 3 3 3 2 2 3 7 : 39392
aida05 : 16 6 5 1 2 4 3 2 2 3 7 : 39520
aida06 : 18 6 6 2 2 2 2 3 2 3 7 : 39576
aida07 : 12 8 6 1 2 3 1 3 2 3 7 : 39408
aida08 : 13 11 4 0 1 3 3 2 2 3 7 : 39308
aida09 : 10 8 4 2 3 4 2 3 2 3 7 : 39848
aida10 : 16 8 2 1 2 4 2 3 1 4 7 : 40768
aida11 : 26 10 2 2 1 2 2 3 2 3 7 : 39512
aida12 : 16 9 2 1 2 4 2 2 2 3 7 : 39240
aida13 : 15 9 3 2 2 4 2 2 2 3 7 : 39284
aida14 : 25 10 2 3 3 3 2 3 2 3 7 : 39796
aida15 : 4 2 2 1 1 3 2 2 2 3 7 : 38944
aida16 : 16 7 3 3 1 3 3 2 2 3 7 : 39368
04:00 Statistics ok - 210517_0403_Stats
Temperatures ok - 210517_0403_Temp
Bias and leakage ok - 210517_0400_Bias
04:05 System wide checks:
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1552 : 0x1560 : 14
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 7 5 2 3 5 2 2 2 3 7 : 39492
aida02 : 23 6 7 2 1 3 2 3 2 3 7 : 39676
aida03 : 17 9 3 2 0 3 2 3 2 3 7 : 39548
aida04 : 16 4 1 2 1 3 3 3 2 3 7 : 39792
aida05 : 22 7 3 3 1 4 3 3 2 3 7 : 40032
aida06 : 18 6 7 3 1 3 2 3 2 3 7 : 39688
aida07 : 18 7 4 1 2 4 1 4 2 3 7 : 40032
aida08 : 19 5 4 1 1 3 3 3 2 3 7 : 39828
aida09 : 10 6 7 1 3 4 2 2 2 3 7 : 39336
aida10 : 12 10 2 3 1 4 2 3 1 4 7 : 40768
aida11 : 10 9 4 3 2 1 2 3 2 3 7 : 39440
aida12 : 8 7 6 1 2 4 2 3 2 3 7 : 39768
aida13 : 11 2 6 1 2 3 2 3 2 3 7 : 39612
aida14 : 27 7 4 2 3 3 2 3 2 3 7 : 39780
aida15 : 4 2 3 1 1 3 2 2 2 3 7 : 38960
aida16 : 18 8 4 2 1 4 2 2 2 3 7 : 39240
06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA
06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again.
07:01 it is working fine.
07:40 Screen shots after the restart
Statistics ok - 210517_0742_Stats
Temperatures ok - 210517_0743_Temp
Bias and leakage ok - 210517_0740_Bias
System wide checks
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x61da : 0x61dd : 3
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 6 3 1 1 3 1 3 3 3 7 : 40336
aida02 : 11 6 0 1 1 5 2 4 2 3 7 : 40252
aida03 : 0 5 2 3 3 3 3 4 2 3 7 : 40424
aida04 : 2 4 2 1 2 4 3 3 2 3 7 : 39912
aida05 : 11 7 5 1 2 2 4 2 3 3 7 : 40532
aida06 : 20 6 0 2 3 2 1 3 3 3 7 : 40320
aida07 : 15 5 1 2 2 3 2 2 3 3 7 : 40116
aida08 : 2 3 4 1 3 3 2 2 3 3 7 : 40128
aida09 : 18 5 1 3 2 4 2 2 3 3 7 : 40288
aida10 : 11 6 3 4 2 3 3 2 2 4 7 : 41484
aida11 : 4 4 5 2 2 2 3 3 2 3 7 : 39744
aida12 : 18 7 2 3 2 2 4 3 2 3 7 : 40064
aida13 : 2 2 1 3 0 3 2 4 2 3 7 : 39944
aida14 : 16 4 5 3 3 3 3 2 2 4 7 : 41552
aida15 : 9 3 1 1 2 3 2 3 3 3 7 : 40556
aida16 : 16 5 1 2 1 2 3 4 2 3 7 : 40184
Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200
***** 08.30 *****
Checked R12_34 through analyser, dead time seems okay:
*** Timestamp elapsed time: 232.154 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.021 0.000
1 2.781 0.000
2 0.849 0.000
3 5.686 0.000
4 0.000 25.816
5 0.017 0.000
6 0.177 0.000
7 1.349 0.000
8 1.791 0.000
9 0.000 91.660
10 6.194 0.000
11 0.320 0.000
12 1.425 0.000
13 0.000 72.999
14 0.052 0.000
15 1.729 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s
***** 09.00 *****
Checked R12_48 through analyser, dead time seems okay:
*** Timestamp elapsed time: 170.459 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.196 0.000
1 6.294 0.000
2 0.992 0.000
3 8.944 0.000
4 0.024 0.000
5 0.001 0.000
6 0.605 0.000
7 2.096 0.000
8 1.654 0.000
9 0.000 31.395
10 6.175 0.000
11 2.524 0.000
12 2.076 0.000
13 0.000 10.337
14 0.103 0.000
15 2.203 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0902_Stats.png)
***** 09.30 *****
AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20
Ran analyser for R13_16
*** Timestamp elapsed time: 168.059 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.047 0.000
1 2.282 0.000
2 0.080 0.000
3 0.806 0.000
4 0.000 0.000
5 0.260 0.000
6 0.030 0.000
7 1.471 0.000
8 67.467 0.000
9 80.552 0.000
10 0.057 0.000
11 0.000 11.671
12 1.191 0.000
13 0.080 0.000
14 0.091 0.000
15 0.202 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision
*** Timestamp elapsed time: 149.082 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 3.580 0.000
2 0.000 0.000
3 2.800 0.000
4 0.000 0.000
5 0.514 0.000
6 0.140 0.000
7 1.372 0.000
8 55.544 0.000
9 66.588 0.000
10 1.065 0.000
11 0.000 10.010
12 0.681 0.000
13 0.044 0.000
14 0.070 0.000
15 0.400 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)
Merger rate ~5M items/s
Tape server ~17 MB/s
Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision
*** Timestamp elapsed time: 122.147 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.027 0.000
1 19.945 0.000
2 0.601 0.000
3 21.555 0.000
4 0.042 0.000
5 3.226 0.000
6 1.211 0.000
7 9.674 0.000
8 52.817 0.000
9 54.972 0.000
10 1.848 0.000
11 0.008 0.000
12 2.034 0.000
13 0.278 0.000
14 0.018 0.000
15 0.634 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
high deatime now also seen in aida02 and aida04
Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check
*** Timestamp elapsed time: 205.006 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 40.176
1 0.438 0.000
2 0.199 0.000
3 2.927 0.000
4 0.000 140.347
5 1.282 0.000
6 0.150 0.000
7 2.097 0.000
8 1.673 0.000
9 0.000 87.749
10 1.150 0.000
11 0.489 0.000
12 0.374 0.000
13 0.048 20.995
14 0.251 0.000
15 2.441 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs
Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200
***** 08.30 *****
Checked R12_34 through analyser, dead time seems okay:
*** Timestamp elapsed time: 232.154 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.021 0.000
1 2.781 0.000
2 0.849 0.000
3 5.686 0.000
4 0.000 25.816
5 0.017 0.000
6 0.177 0.000
7 1.349 0.000
8 1.791 0.000
9 0.000 91.660
10 6.194 0.000
11 0.320 0.000
12 1.425 0.000
13 0.000 72.999
14 0.052 0.000
15 1.729 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s
***** 09.00 *****
Checked R12_48 through analyser, dead time seems okay:
*** Timestamp elapsed time: 170.459 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.196 0.000
1 6.294 0.000
2 0.992 0.000
3 8.944 0.000
4 0.024 0.000
5 0.001 0.000
6 0.605 0.000
7 2.096 0.000
8 1.654 0.000
9 0.000 31.395
10 6.175 0.000
11 2.524 0.000
12 2.076 0.000
13 0.000 10.337
14 0.103 0.000
15 2.203 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
Stats checked seem okay (0902_Stats.png)
***** 09.30 *****
AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20
Ran analyser for R13_16
*** Timestamp elapsed time: 168.059 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.047 0.000
1 2.282 0.000
2 0.080 0.000
3 0.806 0.000
4 0.000 0.000
5 0.260 0.000
6 0.030 0.000
7 1.471 0.000
8 67.467 0.000
9 80.552 0.000
10 0.057 0.000
11 0.000 11.671
12 1.191 0.000
13 0.080 0.000
14 0.091 0.000
15 0.202 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision
*** Timestamp elapsed time: 149.082 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 3.580 0.000
2 0.000 0.000
3 2.800 0.000
4 0.000 0.000
5 0.514 0.000
6 0.140 0.000
7 1.372 0.000
8 55.544 0.000
9 66.588 0.000
10 1.065 0.000
11 0.000 10.010
12 0.681 0.000
13 0.044 0.000
14 0.070 0.000
15 0.400 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)
Merger rate ~5M items/s
Tape server ~17 MB/s
Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision
*** Timestamp elapsed time: 122.147 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.027 0.000
1 19.945 0.000
2 0.601 0.000
3 21.555 0.000
4 0.042 0.000
5 3.226 0.000
6 1.211 0.000
7 9.674 0.000
8 52.817 0.000
9 54.972 0.000
10 1.848 0.000
11 0.008 0.000
12 2.034 0.000
13 0.278 0.000
14 0.018 0.000
15 0.634 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
high deatime now also seen in aida02 and aida04
Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check
*** Timestamp elapsed time: 205.006 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 40.176
1 0.438 0.000
2 0.199 0.000
3 2.927 0.000
4 0.000 140.347
5 1.282 0.000
6 0.150 0.000
7 2.097 0.000
8 1.673 0.000
9 0.000 87.749
10 1.150 0.000
11 0.489 0.000
12 0.374 0.000
13 0.048 20.995
14 0.251 0.000
15 2.441 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs
12:52 Check FPGA Timestamp errors gives following error message:
Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"
Tcl Call Trace
6: DocSubstSystemFile sock8 error 500 can't\ read\ "\;TS_Base_Collected"\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n"\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}"\;\n\ \ \ \ (procedure\ "\;do_TS_errors"\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n"\;do_TS_errors\ "\;\n\ \ \ \ ("\;TSERRORS"\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ..."\;\n\ \ \ \ (procedure\ "\;do_click"\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n"\;do_click"\;\n\ \ \ \ invoked\ from\ within\n"\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ "\;"\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ "\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form..."\;\n\ \ \ \ (file\ "\;Check.tcl"\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n"\;source\ Check.tcl"\;\n\ \ \ \ (in\ namespace\ eval\ "\;::Check"\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n"\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB..."\;\n\ \ \ \ invoked\ from\ within\n"\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n<\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>\;..."\;\n\ \ \ \ ("\;uplevel"\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;uplevel\ #0\ [list\ subst\ \$script\]"\;\n\ \ \ \ (procedure\ "\;SubstFile"\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n"\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}"\;\n\ \ \ \ ("\;uplevel"\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]"\;\n\ \ \ \ (procedure\ "\;Subst_File"\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n"\;Subst_File\ \$template\ \$interp"\;\n\ \ \ \ invoked\ from\ within\n"\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)"\;\n\ \ \ \ (procedure\ "\;Doc_application/x-tcl-template"\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n"\;\$cmd\ \$path\ \$suffix\ \$sock"\;\n\ \ \ \ (procedure\ "\;Doc_Handle"\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n"\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock"\;\n\ \ \ \ (procedure\ "\;DocDomain"\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n"\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml"\;\n\ \ \ \ ("\;eval"\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n"\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]"\;
5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\"
4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE
3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {}
2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml}
1: HttpdReadPostGlobal sock8 Httpds
Dead time check:
[npg@aidas-gsi S496]$ ~/analyser/analyser R14_24
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 252995858 ( 451517.9 Hz)
Other data format: 8924143 ( 15926.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 85 ( 0.2 Hz)
RESUME: 85 ( 0.2 Hz)
SYNC100: 34087 ( 60.8 Hz)
WR48-63: 34087 ( 60.8 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 8855799 ( 15804.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 0 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 560.323 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 349.960
1 0.002 0.000
2 0.000 5.974
3 1.105 0.000
4 0.000 447.454
5 0.000 2.809
6 0.000 16.727
7 0.137 0.000
8 0.090 0.000
9 0.000 429.897
10 0.461 0.000
11 0.018 0.000
12 0.000 8.985
13 0.000 466.311
14 0.000 15.095
15 0.346 0.000
13:24 Beam is off at the moment...
13:43 Changing spill structure from 1.5s/3.5s to 1s.
13:50
[npg@aidas-gsi S496]$ ~/analyser/analyser R14_29
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 251763810 ( 930342.1 Hz)
Other data format: 10156190 ( 37530.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 205 ( 0.8 Hz)
RESUME: 205 ( 0.8 Hz)
SYNC100: 33009 ( 122.0 Hz)
WR48-63: 33009 ( 122.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 10089762 ( 37284.7 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 527077 ( 1947.7 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 270.614 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.034 6.803
1 8.461 0.000
2 0.569 0.000
3 11.664 0.000
4 0.024 26.416
5 0.854 0.000
6 0.069 0.000
7 0.741 0.000
8 1.336 0.000
9 0.000 75.926
10 0.353 0.000
11 0.105 0.000
12 0.070 0.000
13 0.000 93.585
14 0.017 0.000
15 0.121 0.000
15:06 Spill structure back to 1.5 on 2s off. |