Included waves for all 12 FEEs

FEES 1, 5, 9 all show this HF noise on it, and are all located at the top of the AIDA mount.
Connections: HV from DSSD (-ve) is in FEE9
Pulser input is in FEE9 (non-inverted)

Resistance between FEE cooling plate and LEMO connector for all 3 FEEs is a few Ohms, as is resistance between copper braid on DSSD cable and the LEMO connector.

All FEEs show some noise but mainly those 3 and all four with DSSD. Will try to check DSSD connections in case something is messed up.

FEE layout for reference: (Beam's perspective)

                              FEE9
                              FEE5
                              FEE1

FEE10 FEE6 FEE2                FEE4 FEE8 FEE12

                              FEE3
                              FEE7
                              FEE11

The water issue has been resolved - merely incorrect temperature guages (replaced). Water is 20 C as required.

S4 conditions at 13:12 CEST

26.5 C / 47.2 % / Td = 14.3 C

Based on email by TD doing the following procedure to  ensure FEEs are OK after water issue

1) disconnect the power cables of *all* FEE64s at the FEE64 PSUs
2) re-connect power cable, power-up and check/test the FEE64s
  *one at a time* until stable operatinmg temperature is achieved
... say 30mins?

13:34 CEST: All be FEE1 unplugged. Powering on!

Vertex temp read slightly warm (67 C) so powered off. Waiting for a while for water to circulate since off for a while...

15:01 CEST: FEE1 Powered on again, Vertex reading 67 C again but it's stable and slowly going down. Watching and seeing but I think it's OK.
15:35 CEST: FEE1 has been stable around 67 C all the time. Other temps fine. Presuming this isn't a major concern. (Perhaps related to reasonably warm/humid conditions?)

15:47 CEST: FEE2 Powered on.
16:02 CEST: Temperature stable, not exceeded 60 C.

16:07 CEST: FEE3 Powered On.
16:17 CEST: Temperature stable, peaked at 58.44 C

16:21 CEST: FEE4 Powered On.
16:40 CEST: Both FPGA and ASIC reading warm: 65.5 C and 58 C. Seeming to not be rising anymore.

16:41 CEST: FEE5 Powered On.
16:53 CEST: Temperature stable, at 64 C

16:56 CEST: FEE6 Powered On.
17:12 CEST: FPGA temp is ok but ASIC warm. Tmax ASIC = 56.88, FPGA = 55.38, PSU = 27.44

17:15 Turning FEEs and Water off for the weekend
Will test 7-12 on Monday.
Might be cooler in S4.

Cooling seems to be working but temps hotter than before.

The humidity sensor hasn't reported humdities under 90% yet. In order to try and help dry the pipes and turn AIDA the water has been turned off the weekend.
Sunday temperatures might reach 40 C which means it may struggle anyway.

Will consider trying to get a temperature/humidity monitor for S4 to check if the environment is fine too.

It occurs to to wonder if there is now water inside the modules. If the metal of the cooling plates is damp on the outside then it is safe to assume there is water on the inside too . Can I suggest that one  of the modules is dismounted, disassembled and checked?

Patrick

Contents of an email from NH to OH and TD about tests performed on an additional bias supply:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

so I looked at the other supply in the lab (in a NIM crate there, cable direct to DSO)

Still see the same noise wave - also it looked a lot worse at 50V but that may be a worse NIM crate or something else - the current also wasn't 0 (calibration?)

More unusually the signals remain even if the NIM crate is off (DSO_Lab_NoP) and when the SHV is removed from the HV completely (DSO_Lab_NoCable)

FAIR phase 0 DESPEC experiments are scheduled March-May 2020 see

https://www.gsi.de/en/work/organisation/scientific_boards/user/beam_time.htm

The current (19.12.19) version of the schedule is v22 

https://www.gsi.de/fileadmin/beamtime/2020/BTS2020_v022_all.pdf

AIDA will used for DESPEC experiments S480, S452, S470 & S460

To be done at GSI

- check DSSSD - FEE64 connection using alpha background data collected pre- and post- Dec 19 test (NH)

- investigate system noise (TD, NH et al.)

- replace aida10 adaptor PCB and/or FEE64 (TD, NH et al.)

Other issues requiring the support of Carl, James, Patrick & Vic are 
in order of our ( = TD + NH ) priority - Carl et al. may have a different
view. If so, we should discuss.

- ASIC settings / Options DB 'corruption' issues (VFEP, CW et al.)

- waveform data (PCS, JL et al.)

- FEE64/ASIC sync (PCS, JL et al.)

ELOG V3.1.3-unknown

ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2

HT cable shielded, HT of upstream AIDA from iseg, HT of downstream AIDA from CAEN. Both channel 0 and 1 of CAEN are on, even though there is just one channel connected. During this test, some parallele tests are being done on BB7 Mesytec. In general, middle tile: detector disconnected from preamps but preamps connected on the other side.  Right tile: detector connected to preamps, cable shielded but preamps disconnected on the other side (supplied, one output is plugged). Those conditions may have changed during the test. The BB7 and bplast crates are also being connected to the cupper sheet of AIDA's structure to be grounded, the carton pieces under the crates have been removed.

AIDA05 connected, HT 0V

Attachment 1: AIDA05.1.8L --> FWHM 1907 channels

Attachment 2: AIDA09.1.9L --> FWHM 1467 channels

AIDA05 connected, HT 100V

Attachment 3: AIDA05.1.8L --> FWHM 74 channels

Attachment 4: AIDA09.1.8L --> FWHM 79 channels

Attachment 5: AIDA13.1.8L --> FWHM 250 channels

AIDA05 disconnected, HT 0V (cable between the detector and the adaptor board)

Attachment 6: AIDA05.1.8L --> FWHM 16 channels

AIDA05 disconnected, HT 100V

Attachment 7: AIDA05.1.8L --> FWHM 16 channels

Attachment 8: AIDA09.1.8L --> FWHM 87 channels

Attachment 9: AIDA13.1.8L --> FWHM 234 channels

Attachment 10: AIDA12.1.8L --> FWHM 116 channels (bplast cable side disconnected) to be compared to 117 channels (screenshot forgotten to be saved but tags left 21600 and 21171) --> no effect

S4 Conditions:

25.2 C, 35.3% RH, Td = 8.8 C

Water temperature: 20 C

-

Rotating the dewpoint sensor on the pipe seems to have made it switch over to solid red = humidity OK.
Waiting confirmation to see if we should check a FEE (as per PJCS suggestion) before water is turned back on.
Will check the outside copper after lunch.

13:04 : Copper seems warm and dry, humidity sensor still solid red.

Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.

The upstream window of the AIDA detector assembly consists of 2x 50um aluminized mylar

In no particular order

1) Options

variables unexpectedly changing or becoming unset/undefined - see https://elog.ph.ed.ac.uk/DESPEC/205

Is this why *.L, *.H and Rate spectra are undefined in some FEE64s?


2) Merger 'Bad Timestamp' error reports - e.g. see https://elog.ph.ed.ac.uk/DESPEC/203

Meaning of report? Significance/impact?


3) Large numbers of PAUSE/RESUME for some (but not all) FEE64s - see e.g. https://elog.ph.ed.ac.uk/DESPEC/203

Significance/impact?

-- NH to add

4) Slow but non-zero increase of White Rabbit errors (mostly timestamp but much rarer ASIC errors)
We know the White Rabbit network in general is reliable but could we see interruption to the 200 MHz clock or 10 kHz timestamp when FATIMA DAQ is busy/restarting
Wasn't uniform over all FEEs implying a more local source though?

-- OH to add
5) The dropping of links between FEEs and the MERGER. Quite often these were correlated with the 'Bad Timestamp' error messages and also the WR errors

---- PJCS to add
6) The addition of the NIM clock-divider to allow better ASIC clock synchronisation.

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

678/1 shows the AIDA OR64 trigger from the fast comparator of aida10.

678/2-4 show the bPlast accepted trigger after the bPlast DAQ has been triggered by the AIDA OR64. The images show a signal coming fast in time after the pulser and also a signal at around 80 us that appears to be coming from pile up.

The bPlast DAQ also now runs with the AIDA OR64 trigger after the NIM out signal from the MACB was sent to an octal discriminator to fix the width of the signal and also the pulse width which from TAMEX should be kept at or above 100 ns. The NIM signal from the MACB was mostly 100 ns but also jumping to signals with a 20ns pulse width.

The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.

These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.

This was set to have the HitRate in aida10 to be just above the noise.

Initial results: copy from DESPEC elog.

We have managed to obtain the time difference between the bPlast White rabbit and the AIDA Fast time. We had to gate out the zero fast-time events and then also condition that we only take the data from aida10. We see this in 684/1. Then if we look at the time difference between the fast time and the bPlast WRT we see a sharp peak at zero, this makes sense as the AIDA fast time discriminator is being triggered by a pulser which is then being used as a trigger for bPlasts DAQ so these events should be virtually arriving without delay. The delay we do see is infact around 750 ns.

The data collected overnight was also analysed as shown in 684/3 this is the data accumulated with the 22Na source close to the snout and a global threshold in the fast discriminator of 0x0f. The centre peak was roughly fitted with a gaussian:    

  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  Constant     1.21277e+03   2.74526e+01   5.66915e-01  -5.61074e-07
   2  Mean        -1.78513e+02   8.50699e+00   1.62947e-01   1.58554e-06
   3  Sigma        3.48571e+02   4.03993e+00   6.26343e-05  -1.86863e-03

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

Yesterday we spent time essentially performing a dry run to get AIDA DAQ and bPlast into the time sorter, there were some issues with the AIDA mbs PC x86l-119 which was related to some boot issue, it was booting to a newer version of debian while the Relay for AIDA to MBS is on an older version (scratch) (the machine is quite old). We spent a bit of time getting the thresholds correct for each of the ASICs. In the end we just elected to have a blanket level of 0x0f for all of the ASICs when we removed the pulser.

Last night we left the setup with the following thresholds on all of the FEEs to collect data. 681/1. The pulser was also turned off and the 22Na source was moved close to the snout.

This morning c. 9:25 we returned and checked the temperatures, HV and statistics and everything seems to be ok. The only problem was that the timesorter has crashed this morning at 8 am.

AIDA DSSSDs and FEE64-DSSSD cabling are electrically isolated (by design) from snout, mounting rods and support assembly.
 FEE64-DSSSD ribbon cables screened by 3M 1245 Cu foil, drain wire to FEE64 adaptor PCB

AIDA FEE64s are electrically isolated (by design) from the AIDA support frame and each other.

BNC PB-5 test signal 
 daisy chained to FEE64 adaptor PCBs via screened RG174 cable/Lemo 00-250 

CAEN N1419ET DSSSD bias
 daisy chained to 1x p+n junction FEE64 adaptor PCB/Si wafer via screened RG174 cable/Lemo 00-250
 CAEN N1419ET outputs floating, locally grounded by 1x n+n Ohmic FEE64 adaptor PCB link/DSSSD

AIDA FEE64s are connected to the DESPEC 19" rack
 power (AIDA FEE64 PSUs),
 timestamp (AIDA NIM MACB modules),
 system console (via serial-USB cables to USB hubs),
 RJ45 network cables (24 port Gbit network switch)

AIDA PSU PowerStax MS1U-6C-222233-01 - see attachments 1-2
 PSU outputs V+/V- and common 
 no direct ground connection

AIDA MACB
 HDMI cabling MACB-FEE64 is screened and shield is grounded

system console
 ?

RJ45 network cables are coupled by ferrite beads to FEE64 PCB and network switch
 no direct ground connection