AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA  ELOG logo
Message ID: 291     Entry time: Tue Jun 7 19:19:34 2016
Author: CG, TD, DK, AE 
Subject: Wednesday 8th June 

(4 mm Al mask was put upstream of AIDA near 20:20 on 7 June)

 

01.15 Run R1219 started on neutron rich setting.

            Temps ok. Leakage currents ok.

             Plate with holes in front of AIDA , after F11 plastic. F11 plastic count rate ~700cps.

             Keeps getting 'Attempting resychronisation' message in merger terminal window.

             driver 0 10205 consuming almost 100% of CPU. Merger ~20% and data links ~1%

              See attachments 1-4

 

02.15 Run R1220 n-rich setting (BigRIPS run 1218, EURICA off)

          EURICA team retires for the night and cannot get it up and running

          Biases and leak shown as attachments 5-6

          Merger info is shown in attachments 7-9

          CPU Load in attachment 10

          Timestamps against the ADCs and also SYNCS, respectively, attachments 11 and 12

          3:31: Run stopped for the operator to tune the beam

          F11 plastic rate was 700 cps (~heavy ion injection rate)

 

3:43 Run R1221 n-rich setting (BigRIPS 1219, no EURICA)

          4 mm Al mask was removed.

          Pause calls vs. Timestamp is attachment 13

          Merger screenshots are shown as attachments 14-16

          F11 plastic rate is 850 or 900 cps after retune

          ADC v Timestamp is shown as attachment 17.  It can be seen the rate was roughly constant before the operators stopped (corresponding to a long gap of no hits, but now  we go up and down quite a lot in a rather ugly way

          CPU stats (driver to write to HDD in green, merger in red) as attachment 18

          At the time of ~90 in attachment 18, the linkers start to consume all available CPU resources, shown as attachment 19

 

04:35 Run R1222 n-rich setting (BigRIPS 1219 / 1220)

          Ask the operators to reduce the beam intensity by "1/2" -> F11 plastic rate goes from 850 cps to 170 cps, so more like 1/4 effectively

          Merger stats are shown as attachments 20, 21, 22. 

          nnaida1 and nnaida24 seemed to often show pauses (among other ones)

          trying to understand what's going on with the display in Midas, because it has looped around to time zero and is putting new data on to the old data...?

         It seems this condition is much better than before, though

         05:33 stop the run

 

05:38 R R1223 n-rich setting (BigRIPS 1221)

           Ask the operators to resume the previous beam intensity.  (Now like 600 to 800 cps at the F11 plastic)

           attachment 23 shows the beam coming on near time zero.  within <100 seconds, the driver has to fight with many link64 instances and cannot operate correctly

           we understood how to clear the midas histograms so that we can make sense of things.  attachment 24 is ADC, and attachment 25 is pause. 

           you can see some data come smoothly until the system locks down

06.11 R1223 stopped.

 

06.30 Started run R1224.

            Moved Pb bricks downstream of F11 plastic inwards, from a separation of 15cm, to 4cm in an effort to reduce rate in AIDA.

            F11 plastic rate ~900cps.

            No change to vis scalar rates and data still being dropped.

06.45 Stopped run R1224.

            MIDAS plot of ADC vs Timestamp (attachment 26) shows characteristic behaviour.

            Upon writing to disk, for the first minute or so link64 processes consume minimal CPU power. But then very quickly they ramp up consumption and ultimately choke the merger (it seems).

            This is reflected in the ADC v TS plot. Continuous ADC data for first ~1min.

 

06.54 Moved Pb bricks to separation of 2cm. Started run R1225.

            AIDA vis scalar rates reduced slightly, but no change to CPU usage and usual happens - data lost, link64 CPU usage spikes and chokes merger (seemingly).

            F11 plastic rate ~900cps.

07.00 Run R1225 stopped.

           SYNCs stopped being produced. Lost contact with nnaida1.

 

07.25 Pb bricks moved to 1cm apart. Run R1226 started.

            No change to CPU usage (link64 still hogging everything). MIDAS online monitor lost in power cycle so cant view in real-time, but anticipate same outcome.

            Sopped producing SYNCs 07.39. Run 1226 stopped.

 

08.20 Some trouble restarting DAQ and merger.

            As nnaida19 has lost the 'H' for histogramming on the run control page, nnaida21 has now lost the 'X' for data transfer (see attachment 27).

           Removed nnaida21 from merger and normal state of play resumed.

           4mm holey plate + the one attached to it we installed dowstream of F11 plastic.

 

08.50 Run R1227 started.

           CPU usage by link64 ~1% and driver ~70%, much more reasonable. When reloading merger web page, channels flash between bright green and olive. Good.

09.09 Changed to reference mass setting. Run R1227 stopped.

 

09.12 Run R1228 started on reference setting with holey plate still in place.

           R1228 stopped 10.32 (for B2F/F11 entry)

 

Attachment 1: merge_rate_resync.png  319 kB  | Hide | Hide all
merge_rate_resync.png
Attachment 2: merge_resync.png  349 kB  | Hide | Hide all
merge_resync.png
Attachment 3: tape_resync.png  299 kB  | Hide | Hide all
tape_resync.png
Attachment 4: disk_driver_CPU_usage.png  363 kB  | Hide | Hide all
disk_driver_CPU_usage.png
Attachment 5: bias1_R1220_0806-03.10.png  12 kB  | Hide | Hide all
bias1_R1220_0806-03.10.png
Attachment 6: bias2_R1220_0806-03.10.png  12 kB  | Hide | Hide all
bias2_R1220_0806-03.10.png
Attachment 7: R1220_MergeControl_0806-03.12.png  108 kB  | Hide | Hide all
R1220_MergeControl_0806-03.12.png
Attachment 8: MergerLinkRates_R1220_0806-03.13.png  130 kB  | Hide | Hide all
MergerLinkRates_R1220_0806-03.13.png
Attachment 9: MergerStats_R1220_08-6-03.13.png  151 kB  | Hide | Hide all
MergerStats_R1220_08-6-03.13.png
Attachment 10: cpuload_R1220_0806-03.16.png  80 kB  | Hide | Hide all
cpuload_R1220_0806-03.16.png
Attachment 11: ADCvTimestamp_R1220_0806-03.17.png  36 kB  | Hide | Hide all
ADCvTimestamp_R1220_0806-03.17.png
Attachment 12: SYNCSvTimestamp_R1220_0806-03.18.png  34 kB  | Hide | Hide all
SYNCSvTimestamp_R1220_0806-03.18.png
Attachment 13: PauseVTimestamp_R1221_0806_03.51.png  34 kB  Uploaded Tue Jun 7 20:52:08 2016  | Hide | Hide all
PauseVTimestamp_R1221_0806_03.51.png
Attachment 14: MergeControl_R1221_0806_03.50.png  107 kB  Uploaded Tue Jun 7 20:52:21 2016  | Hide | Hide all
MergeControl_R1221_0806_03.50.png
Attachment 15: MergerStats_R1221_0806_03.50.png  153 kB  Uploaded Tue Jun 7 20:52:37 2016  | Hide | Hide all
MergerStats_R1221_0806_03.50.png
Attachment 16: MergerRates_R1221_0806_03.49.png  133 kB  Uploaded Tue Jun 7 20:52:59 2016  | Hide | Hide all
MergerRates_R1221_0806_03.49.png
Attachment 17: ADCvT_R1221_0806_04.02.png  44 kB  Uploaded Tue Jun 7 21:03:36 2016  | Hide | Hide all
ADCvT_R1221_0806_04.02.png
Attachment 18: CPULoad_R1221_0806_04.06.png  72 kB  Uploaded Tue Jun 7 21:09:38 2016  | Hide | Hide all
CPULoad_R1221_0806_04.06.png
Attachment 19: TOP_R1221_0806_04.16.png  74 kB  Uploaded Tue Jun 7 21:17:56 2016  | Hide | Hide all
TOP_R1221_0806_04.16.png
Attachment 20: Merger_R1222_0807-04.59.png  108 kB  Uploaded Tue Jun 7 22:01:55 2016  | Hide | Hide all
Merger_R1222_0807-04.59.png
Attachment 21: MergerStats_R1222_0806-05.00.png  152 kB  Uploaded Tue Jun 7 22:02:16 2016  | Hide | Hide all
MergerStats_R1222_0806-05.00.png
Attachment 22: MergerLink_R1222_0806-04.59.png  146 kB  Uploaded Tue Jun 7 22:02:45 2016  | Hide | Hide all
MergerLink_R1222_0806-04.59.png
Attachment 23: CPUStat_R1223_0806-05.58.png  74 kB  Uploaded Tue Jun 7 22:59:17 2016  | Hide | Hide all
CPUStat_R1223_0806-05.58.png
Attachment 24: ADC_v_T_R1223_0806-06.03.png  38 kB  Uploaded Tue Jun 7 23:04:53 2016  | Hide | Hide all
ADC_v_T_R1223_0806-06.03.png
Attachment 25: Pause_v_T_R1223_0806-06.04.png  34 kB  Uploaded Tue Jun 7 23:05:08 2016  | Hide | Hide all
Pause_v_T_R1223_0806-06.04.png
Attachment 26: ADCvTS_R1224_0639.png  67 kB  Uploaded Tue Jun 7 23:54:11 2016  | Hide | Hide all
ADCvTS_R1224_0639.png
Attachment 27: nnaida21_noDataXfer.png  138 kB  Uploaded Wed Jun 8 01:20:27 2016  | Hide | Hide all
nnaida21_noDataXfer.png
ELOG V3.1.4-unknown