00:00 - CA takes over

00:15 - stats/database/ucesb ok - attachment 1

      - another burst of bad timestamp error messages in NewMerger terminal - but all FEE64 still ok - attachment 2

      - Merger/TapeService ok

00:21 - beam still off while UNILAC having problems

00:26 - beam is back - rate spectra - attachment 3

00:32 - ucesb following beam return - attachment 4

      - implant rates reaching highs of ~1kHz

00:35 - Grafana leakage current monitor last 6 hours - attachment 5

01:00 *all* system wide checks ok

      stats ok - attachment 6

      temperatures ok - attachment 7

      detector bias / leakage currents ok - attachment 8

      db check ok / ucesb ok

      still observe periodic bursts of bad timestamp errors in NewMerger terminal - FEE64s still linked with merger ok

01:30 AIDA crashes just as I make check, most FEE64 lost connection -> powercycle and restart MIDAS

01:53 AIDA reset completed - writing to file NULL/R48

01:56 all system wide checks ok *except* 

FPGA error counter

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


02:01 good event statistics ok - attachment 9

      temperatures ok - attachment 10

      detector bias / leakage currents ok - attachment 11

      db check / ucesb ok

02:20 beam stop

02:21 beam back

02:30 stats/ucesb/db checks all ok

03:00 stats/ucesb/db checks all ok - attachment 12

03:30 stats/ucesb/db checks all ok

04:00 system wide checks ok, except;

      ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

     FPGA errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x9 : 	 9  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:03 beam off briefly

04:05 good event statistics ok - attachment 13

04:07 beam back

      temperatures ok - attachment 14

      bias / leakage currents ok - attachment 15

      db check / ucesb ok - attachment 16

      implant rates in AIDA peaking at ~1.5kHz

04:30 stats/ucesb/db checks all ok

04:48 no beam

04:58 beam back

05:00 stats/ucesb/db checks all ok - attachment 17

05:30 stats/ucesb/db checks all ok

06:00 system wide checks ok, except;

      ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

     FPGA errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x9 : 	 9  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

    FEE64 Linux Memory Info

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      5      3      3      2      3      2      3      2      3      7   : 39628
aida02 :     22     11      3      0      2      3      2      3      2      3      7   : 39648
aida03 :     20     10      1      1      2      2      1      4      2      3      7   : 39760
aida04 :     17     10      2      2      3      3      2      3      2      3      7   : 39732
aida05 :      1      8      3      2      3      2      2      3      3      3      7   : 40564
aida06 :      5      6      2      3      2      2      3      3      2      3      7   : 39748
aida07 :     18      4      7      0      1      3      1      4      2      3      7   : 39832
aida08 :     22     10      3      3      2      2      3      3      2      3      7   : 39864
aida09 :      3      6      2      2      0      3      1      4      2      3      7   : 39708
aida10 :      2      4      3      2      1      2      2      3      2      3      7   : 39384
aida11 :     20      8      3      1      1      3      3      2      2      3      7   : 39328
aida12 :     16      6      3      1      0      3      2      3      2      3      7   : 39488

06:02 good event statistics ok - attachment 18

      FEE64 temperatures ok - attachment 19

      detector bias / leakage currents ok - attachment 20

06:06 db check and ucesb ok - attachment 21

      NewMerger terminal hasn't flagged up any more errors since ~ 03:00

06:30 stats/ucesb/db checks all ok

      DESPEC restarts timesorter

07:00 stats/ucesb/db checks all ok

07:30 stats/ucesb/db checks all ok - attachment 22

      grafana leakage currents last 1 hour - attachment 23