Still  no beam in CRING.

Motor fully out position.

SW3

- Merger OK

- Screenshot uploaded

- Data rate : 236 kbytes/s

SW4

- current plot uploaded

SW2

- T OK

- system wide checks OK

- Check statistics tab ok

- Spectra ok

SW5

4 plots uploaded

Still  no beam in CRING.

Motor fully out position.

SW3

- Merger OK

- Screenshot uploaded

- Data rate : 217 kbytes/s

SW4

- current plot uploaded

SW2

- T OK

- system wide checks OK

- Check statistics tab ok

- Spectra ok

SW5

4 plots uploaded

'Check Clock Status': Clock status test result: Passed 5, Failed 0

'Check ADC Calibration' : Calibration test result: Passed 5, Failed 0

'Check the White rabbit decoder status':

Base         Current     Difference
aida01 fault      0xab60 :      0xab7f :      31  
aida02 fault      0x88d3 :      0x88f2 :      31  
aida03 fault      0x23aa :      0x23c9 :      31  
aida04 fault      0x15d7 :      0x15f6 :      31  
aida05 fault      0x2f03 :      0x2f0c :      9  
White Rabbit error counter test result: Passed 0, Failed 5

histograms was reset.

'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status' - no changes

'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status'  no changes

detector motors are still out

'Check Clock Status', 'Check ADC Calibration' : Clock status test result: Passed 5, Failed 0

'Check the White rabbit decoder status' :  
         Base         Current     Difference
aida01 fault      0xab60 :      0xab8f :      47  
aida02 fault      0x88d3 :      0x8901 :      46  
aida03 fault      0x23aa :      0x23d9 :      47  
aida04 fault      0x15d7 :      0x1606 :      47  
aida05 fault      0x2f03 :      0x2f0f :      12  
White Rabbit error counter test result: Passed 0, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status' - no changes

'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status': no change

'Check Clock Status', 'Check ADC Calibration' and 'Check the White rabbit decoder status' : no change

no change:

Check Clock Status': Passed 5, Failed 0

Check ADC Calibration: Passed 5, Failed 0

'Check the White rabbit decoder status: Passed 0, Failed 5

target density at S1: 2.1-2.4e-8

temp: 116K

no change:

Check Clock Status': Passed 5, Failed 0

Check ADC Calibration: Passed 5, Failed 0

'Check the White rabbit decoder status: Passed 0, Failed 5

target density at S1: 2.1-2.4e-8

temp: 116K

no change:

Check Clock Status': Passed 5, Failed 0

Check ADC Calibration: Passed 5, Failed 0

'Check the White rabbit decoder status: Passed 0, Failed 5

target density at S1: 2.1-2.3e-8

temp: 116K

beam at injection: 7e7

no change:

Check Clock Status': Passed 5, Failed 0

Check ADC Calibration: Passed 5, Failed 0

'Check the White rabbit decoder status: Passed 0, Failed 5

target density at S1: 2.3-2.7e-8

temp: 110K

beam at injection: 7e7

no change:

Check Clock Status': Passed 5, Failed 0

Check ADC Calibration: Passed 5, Failed 0

'Check the White rabbit decoder status: Passed 0, Failed 5

target density at S1: 2.2-2.7e-8

temp: 110K

beam at injection: 7e7

Steps taken:
- MIDAS_Releases, Embedded and Exports files brought across from the previously working SL6 aida-gsi install. Brought as tar balls to preserve ownership
- dhcpd.conf set up with same IP addresses as used on the SL6 install from aida-gsi
- nfs-utils package installed on centos7
- Paths to directories listed in /etc/exports checked that they point to the correct locations on the centos system
- Firewall disabled to confirm that it is not interfering
- setenforce 0 to confirm selinux is not interfering.


- Attachment 1 Screenshot of FEE console for aida04 as it stops booting.
- Attachment 2 text dump of /var/log/messages during the FEE boot sequence
- Attachment 3 text dump from tcpdump filtered to aida04
- Attachment 4 /var/lib/dhcp/dhcp.leases file as mentioned in the /var/log/messages dump
- Attachment 5 A copy of dhcpd.conf
- Attachment 6 A copy of /etc/hosts
- Attachment 7 A copy of /etc/exports    
- Attachment 8 Text dump of ifconfig -a (Note the FEEs are conencted to p4p1)

From the messages dump it seems there is an issue being encountered with dhcp leases for the static IP addresses of the FEEs. They are then getting assigned new IP addresses which are getting put in the leases file.

10:00

The computer screen controlling the DAQ on carme-gsi appeared to have crashed.

Unable to move throught the different windows etc.

Unable to connect via anydesk. ssh access still available. It appears the x-server had crashed, we noticed earlier in the beam certain functions such as screenshots were glitchy so this was probably expected eventually.

We were able to access the DAQ on another CRYRING machine machine via 192.168.207.143:8015 and 192.168.207.143:8015 in the browser. DAQ is still running. We will try to reboot the x server without disturbing the DAQ.

ssh into carme-gsi and used the command 'systemctl reboot gdml' to attempt to reboot only this server.

This did not have the desired effect, whole of carme-gsi rebooted. We should have turned off the FEEs and stopped the DAQ gracefully but did not.

Restarted carme-gsi and rebooted the FEEs and DAQ system.

All checks OK -> clocks, ADC calibration etc.

ASIC settings loaded are 2025APR08. Only changes made to this settings were reducing the slow comparator threshold from 0x32 (1 MeV) to 0x14 (200 keV). aida03 and aida04 were set to the MEC on these settings, they have now been reset to the LEC channel (0x0). All channels on the LEC.

This is a suggested sequence of actions to help with the current problem with the White Rabbit timestamp distribution. Test the system between each intervention. If a test provides an answer to the problem then proceed accordingly rather than continue with the suggestions.

• Please sketch the current MACB wiring.
• Unplug and replug the connectors of the VETAR HDMI connecting cable. If possible clean the pcb and cable HDMI connectors with contact cleaner.
• Disconnect the VETAR and connect the Emulator. If the clocks are not correct then at the Local and Waveform control page change the entry for LMK03200 control register at offset 5 to 0xB. If clocks are now ok then repeat for all the FEEs. Be aware that this value will be set back to 0x1B during SETUP. If the Emulator fails then continue with VETAR and setting 0x1B.
• Connect one or more FEE64 to the base MACB.
• Change the base MACB and repeat the test above.
• Connect a FEE to the VETAR direct. 

I am at present unable to advise further. However should there be more then I will update this entry.

This is how to update the MACB firmware using the opensource tool xc3sprog


Setup (only has to be done first time)
--------------------
Install xc3sprog and fxload (ubuntu: sudo apt install xc3sprog fxload)

To configure the USB programmer:
Download xilinx.tar.gz and put the hex files in /usr/share

Download the udev rules and put it in /etc/udev/rules.d

Run:
sudo udevadm control --reload-rules

[Taken from https://github.com/timvideos/litex-buildenv/wiki/Xilinx-Platform-Cable-USB-under-Linux]

Plug in programmer and should work

----------------------------------
Programming a MACB

Open MACB side panel and connect JTAG programmer to port 

Plug MACB into NIM bin and power NIM bin on

Run
xc3sprog -c xpc -m -j

Expected output:
XC3SPROG (c) 2004-2011 xc3sprog project $Rev$ OS: Linux
Free software: If you contribute nothing, expect nothing!
Feedback on success/failure/enhancement requests:
        http://sourceforge.net/mail/?group_id=170565 
Check Sourceforge for updates:
        http://sourceforge.net/projects/xc3sprog/develop

JTAG loc.:   0  IDCODE: 0x16d8c093  Desc:                  XC2C128_TQ144 Rev: A  IR length:  8

This confirms the JTAG is working

Then to program  run

xc3sprog -c xpc -m . macb_apr20.jed

The directory should contain the macb_apr20.jed and the xc2c128.map files

Expected output:
XC3SPROG (c) 2004-2011 xc3sprog project $Rev$ OS: Linux
Free software: If you contribute nothing, expect nothing!
Feedback on success/failure/enhancement requests:
        http://sourceforge.net/mail/?group_id=170565 
Check Sourceforge for updates:
        http://sourceforge.net/projects/xc3sprog/develop

Programming row  81                                        
Verify: Success   

Now power off the NIM bin and remove the programmer and put the cover back on

Use the M10 jacking points provided top/bottom and 
left/right on the upstream and downstream sides of the 
flanges - for example see attachments 1 & 2