system wide checks:

Clock status:
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit

         Base         Current     Difference
aida06 fault      0x679f :      0x67a4 :      5  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error page not working !!

Memory information:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     15      4      6      4      3      2      4      2      3      6   : 36420
aida02 :     36     25     24      6      0      4      2      3      2      3      4   : 28056
aida03 :     17      7      4      3      3      5      2      2      3      3      6   : 36444
aida04 :      4      5      4      5      3      4      2      3      3      3      6   : 36824
aida05 :     23      9      4      2      3      3      1      3      3      3      6   : 36452
aida06 :     26      8      5      1      3      4      2      2      3      3      6   : 36312
aida07 :     21      9      3      3      4      5      1      2      3      3      6   : 36268
aida08 :     27      9      2      2      2      4      1      3      3      3      6   : 36500
aida09 :     13      4      3      3      1      2      4      2      3      3      6   : 36388
aida10 :      1     10      0      3      1      2      3      3      2      3      6   : 35572
aida11 :     23     13      8      3      3      4      3      3      2      3      6   : 36196
aida12 :     19      7      4      3      2      3      2      3      3      3      6   : 36644

AIDA PC crashe completely at ~3:30am CET.

Tom was called and he Restarted the AIDA system at ~4am with Run 52.

Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.

Quick system checks after crash

Temperature ok.

Still cannot connect remotely to /ucesb with my gsi web login and password (?).

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida05 fault      0x31a2 :      0x31a2 :      0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      9      3      4      2      3      2      1      4      2      3      7   : 39804
aida02 :     29      4      2      3      0      3      1      4      2      3      7   : 39828
aida03 :     20      6      1      2      2      3      1      4      2      3      7   : 39888
aida04 :     19      5      4      2      1      3      1      4      2      3      7   : 39860
aida05 :     27      5      4      1      2      3      2      3      1      4      7   : 40692
aida06 :     17     10      3      2      1      3      1      4      2      3      7   : 39876
aida07 :     22      7      2      2      2      3      1      4      2      3      7   : 39920
aida08 :     12      6      5      1      3      2      1      4      2      3      7   : 39824
aida09 :      0      9      1      3      1      3      1      4      2      3      7   : 39800
aida10 :      2      2      2      3      3      2      3      2      2      3      7   : 39256
aida11 :     21      7      4      3      0      3      2      3      2      3      7   : 39596
aida12 :     18      8      3      0      2      3      2      3      2      3      7   : 39608

ucesb is back - see attachment

Some screenshot of spectra attached

Beam has now stopped (end of expriment) and background measurements starts.

Wide check at 3am -

All fine.

Same output as at the beginning of the shift (see my previous entry) and attachments.

System  wide checks at 7am CET

All seems fine, nothing significant to report

Slight increase of dead time noted in FEE 3, 5 and 7.

-----
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

-----

White Rabbit:

         Base         Current     Difference
aida05 fault      0x500 :      0x56e :      110  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     23      5      6      5      0      4      1      3      3      3      6   : 36484
aida02 :     10      3      3      2      3      2      2      2      3      3      6   : 35952
aida03 :      2      5      1      4      5      2      2      3      3      3      6   : 36608
aida04 :      1      3      4      0      4      3      2      3      3      3      6   : 36572
aida05 :     19      9      3      2      2      3      1      3      4      3      6   : 37380
aida06 :     13     11      1      5      3      4      4      2      3      3      6   : 36860
aida07 :     12      2      4      3      3      2      1      3      2      2      7   : 37280
aida08 :     25      5      7      1      1      3      3      3      3      3      6   : 36828
aida09 :     22      5      6      4      1      3      2      3      3      3      6   : 36640
aida10 :     23      8      6      3      2      3      2      2      2      4      6   : 37212
aida11 :     24     14      7      3      3      2      3      3      3      3      6   : 36960
aida12 :     16     10      5      3      1      2      1      2      4      3      6   : 36736
aida13 :     23      6      4      2      2      3      2      3      3      3      6   : 36620
aida14 :     22      7      8      4      3      4      1      2      2      4      6   : 37200
aida15 :     10     12      9      2      1      3      1      3      3      3      6   : 36376
aida16 :     18      5      6      2      0      3      2      4      3      3      6   : 37008

All seems ok  (apart for my virgin network connection usually quite reliable but not so tonight !?!).

Statistics seems ok.

Temperatures appears stable

Leakage current ok.

System Wide Checks:

Clock check: Passed= 14: - Failed=: 0

ADC calibration check:

FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES

White rabbit decode status: Passed=14 ; Failed=0

FPGA timestamp check: Passed=14 ; Failed=0

Memory information: Page loading failed with the error message below:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"

We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).

This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

Took over remotely from CB  about an hour ago

At 0h40 (CET):
      Stats still ok - attach 1
      Leakage still OK - Attach 2
      Temps - attach 3
      Grafana OK
      UCESB rates ok - attach 5
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

AIDA stats ok

Leakage current ok

Temperatires ok

grafana ok

ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.

System wide check done and same results as earlier:

aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Status at 16:45 (CET)

The expreiment continues to run smoothly.

AIDA Stats look ok - Attachment 1

AIDA Temperature ok - Attachment 2

AIDA Leakage current: ok - Attachment 3

System wide check:

Clock: 13 passed, 1 failed (aida09)

ADC Calibration: 9 passed, 5 fialed (aida2,6,9,10,13)

White rabbit decoder: 14 passed, 0 failed.

FPGA timestamp: 14 passed, 0 failed.

Status at 18:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 4

AIDA Temperature ok - Attachment 5

AIDA Leakage current: increase slightly - Attachment 6

System wide check: No changes, same as above

Status at 20:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 7

AIDA Temperature ok - Attachment 8

AIDA Leakage current: increase slightly - Attachment 9

System wide check: No changes, same as above

Status at 22:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 10

AIDA Temperature ok - Attachment 11

AIDA Leakage current: increase slightly - Attachment 12

System wide check: No changes, same as above

AIDA ELog filled out by DESPEC night shift members Matthew Brunet (MB), Shaheen Jazrawi (SJ), and Matthias Rudiger (MR).

0:05-0:08 Beam operators altering settings, beam off/sporadic during this period.

0:46
    All system wide checks passed
    Leak currents lower than previous measurement - attachment #1
    Implants look as previously - attachment #2
    Stats about where they have been - attachment #3
    FEE temperatures fine - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer writing about 25 MB/s

2:46

      All system wide checks passed
     Leak currents similar to previous - attachment #5 
     Implants look as previously - attachment #6
     Stats about where they have been - attachment #7
     FEE temperatures fine - attachment #8
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s
 

4:49

      All system wide checks passed
     Leak currents similar to previous - attachment #9
     Implants look as previously - attachment #10
     Stats good - attachment #11
     FEE temperatures fine - attachment #12
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s

6:49

      All system wide checks passed
     Leak currents no significant change - attachment #13
     Implants look as previously - attachment #14
     Stats good - attachment #15
     FEE temperatures fine - attachment #16
     Merger fine, 3 Mega items / sec
     TapeServer writing about 25 MB/s

14.15 System wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors - reports 'server internal error'

14.16 DSSSD bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachment 2
      good events stats - attachment 3

14.22 DAQ continues file S460/R51_205
      beam intensity 2.2e+09/s


15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6


Clock check

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC check

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



White Rabbit


		 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




FPGA
Loge page error!

Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      6      2      1      2      3      2      4      2      3      6   : 36044
aida02 :     15      3      2      2      0      3      2      3      2      3      4   : 27188
aida03 :     29      4      0      3      2      4      2      2      3      3      6   : 36212
aida04 :      4      6      4      4      2      3      3      3      3      3      6   : 36864
aida05 :     19      5      4      2      3      3      1      3      3      3      6   : 36404
aida06 :     24      9      5      2      3      3      3      2      3      3      6   : 36472
aida07 :     20      4      9      0      2      3      2      2      3      3      6   : 36096
aida08 :     22      9      4      3      1      3      1      3      3      3      6   : 36352
aida09 :     20      7      3      2      2      3      3      2      3      3      6   : 36344
aida10 :     25      9      9      5      2      3      2      3      3      3      6   : 36828
aida11 :     24     11      2      0      1      3      1      3      3      3      6   : 36248
aida12 :     26      5      4      2      1      3      2      2      3      3      6   : 36048

Old ribbon cable assemblies - attachments 1  2

Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors

p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm

Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm



Manufacture of new ribbon cable assemblies - attachment 3

8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables


17.52 from Helena

"These are the detector positions within the snout:

Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame

Upstream AIDA @ 513 mm

Downstream AIDA @ 523 mm

This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is 
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by 
10mm.

The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position."

03:12 System Check

attachment 1 : Current

attachment 2 : Temperature

attachment 3 : Rates

   
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

   
Calibration test result: Passed 16, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

   
             Base         Current         Difference
aida13 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      5      7      2      1      2      2      3      3      3      6   : 36464
aida02 :      9      8      3      3      1      4      1      2      4      3      6   : 36916
aida03 :      5      2      6      2      0      3      2      3      3      3      6   : 36420
aida04 :      6      5      2      3      4      4      2      3      3      3      6   : 36800
aida05 :     17      6      6      2      2      4      1      3      2      4      6   : 37524
aida06 :      7     12      3      4      3      3      1      3      3      3      6   : 36460
aida07 :     17     11      5      1      3      3      3      2      3      3      6   : 36428
aida08 :      3      5      0      1      4      2      2      4      2      3      6   : 35924
aida09 :     27      6      4      2      0      2      2      2      3      3      6   : 35868
aida10 :     16     11      8      0      2      2      1      3      2      4      6   : 37272
aida11 :     15      2      2      3      1      4      3      4      2      3      6   : 36364
aida12 :      1      6      4      3      1      3      2      4      2      3      6   : 35988
aida13 :     22     14     10      2      4      2      2      4      2      3      6   : 36264
aida14 :     26     10      4      3      2      1      1      3      3      3      6   : 36184
aida15 :     14      2      3      2      2      4      1      2      3      3      6   : 35896
aida16 :      7      5      4      0      2      4      2      3      2      3      6   : 35588

06:10 DSSD1 rate high!

Attached 4

Called OH and he woke up to fix it :)

It was a problem with one of the ASIC according to what he says:

One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35

07:03 System check

attachment 5 Spectrum rate

attachment 6 Voltages

attachment 7 Rates

attachment 8 Temperature

System wide clock are all ok except

White rabbit

 Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA

 Base         Current         Difference
aida13 fault      0xa :      0x14 :      10  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last