Puppetmasterd – High CPU utilization

Logged into a Puppet master server today and noticed that I had some high CPU utilization happening. This is rather odd as the box usually just chugs along with no worries what-so-ever.

Turns out, some others noticed this as well and some had suspicions that it was leap second related. Check out the full thread here.

Looking back at my SAR records, I find that my CPU spiked right around June 30th at 7PM which is July 1, Midnight GMT.

#sar -q -f /var/log/sa/sa30

03:20:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
03:30:01 PM 1 202 0.00 0.00 0.00
03:40:01 PM 1 202 0.00 0.00 0.00
03:50:01 PM 1 202 0.00 0.00 0.00
04:00:01 PM 1 202 0.00 0.00 0.00
04:10:01 PM 2 204 0.00 0.00 0.00
04:20:01 PM 1 202 0.00 0.00 0.00
04:30:01 PM 1 202 0.00 0.00 0.00
04:40:01 PM 1 202 0.00 0.00 0.00
04:50:01 PM 1 202 0.00 0.00 0.00
05:00:01 PM 1 202 0.00 0.00 0.00
05:10:01 PM 2 205 0.00 0.00 0.00
05:20:02 PM 1 202 0.00 0.01 0.00
05:30:01 PM 1 202 0.00 0.00 0.00
05:40:01 PM 1 202 0.00 0.00 0.00
05:50:01 PM 1 202 0.00 0.00 0.00
06:00:01 PM 1 202 0.00 0.00 0.00
06:10:01 PM 2 205 0.00 0.00 0.00
06:20:01 PM 1 202 0.00 0.00 0.00
06:30:01 PM 1 202 0.00 0.00 0.00
06:40:01 PM 1 202 0.00 0.00 0.00
06:50:01 PM 1 202 0.00 0.00 0.00
07:00:01 PM 4 202 0.32 0.07 0.02
07:10:01 PM 4 205 2.45 2.16 1.19

07:10:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
07:20:01 PM 2 202 2.25 2.32 1.73
07:30:01 PM 3 202 2.25 2.42 2.06
07:40:01 PM 2 202 2.42 2.47 2.25
07:50:01 PM 2 202 2.45 2.46 2.33
08:00:01 PM 3 202 2.42 2.48 2.39
08:10:02 PM 3 205 2.25 2.39 2.38
08:20:01 PM 3 202 2.48 2.52 2.45
08:30:01 PM 3 202 2.62 2.49 2.44
08:40:01 PM 2 202 2.49 2.49 2.45
08:50:01 PM 3 202 2.45 2.48 2.46
09:00:01 PM 3 202 2.46 2.51 2.47

Son of a bitch!

The simple solution was to give it a quick reboot. Its just a test puppet installation anyway, but still, pisses me off rebooting boxes like that.

Anyone else have this issue?

<div class="sd-content">
  <ul>
    <li class="share-facebook">
      <a rel="nofollow" data-shared="sharing-facebook-1862" class="share-facebook sd-button share-icon no-text" href="http://usrlocal.com/2012/07/high-puppetmasterd-cpu-utilization/?share=facebook" target="_blank" title="Click to share on Facebook"><span></span><span class="sharing-screen-reader-text">Click to share on Facebook (Opens in new window)</span></a>
    </li>
    <li class="share-linkedin">
      <a rel="nofollow" data-shared="sharing-linkedin-1862" class="share-linkedin sd-button share-icon no-text" href="http://usrlocal.com/2012/07/high-puppetmasterd-cpu-utilization/?share=linkedin" target="_blank" title="Click to share on LinkedIn"><span></span><span class="sharing-screen-reader-text">Click to share on LinkedIn (Opens in new window)</span></a>
    </li>
    <li class="share-google-plus-1">
      <a rel="nofollow" data-shared="sharing-google-1862" class="share-google-plus-1 sd-button share-icon no-text" href="http://usrlocal.com/2012/07/high-puppetmasterd-cpu-utilization/?share=google-plus-1" target="_blank" title="Click to share on Google+"><span></span><span class="sharing-screen-reader-text">Click to share on Google+ (Opens in new window)</span></a>
    </li>
    <li class="share-pinterest">
      <a rel="nofollow" data-shared="sharing-pinterest-1862" class="share-pinterest sd-button share-icon no-text" href="http://usrlocal.com/2012/07/high-puppetmasterd-cpu-utilization/?share=pinterest" target="_blank" title="Click to share on Pinterest"><span></span><span class="sharing-screen-reader-text">Click to share on Pinterest (Opens in new window)</span></a>
    </li>
    <li class="share-twitter">
      <a rel="nofollow" data-shared="sharing-twitter-1862" class="share-twitter sd-button share-icon no-text" href="http://usrlocal.com/2012/07/high-puppetmasterd-cpu-utilization/?share=twitter" target="_blank" title="Click to share on Twitter"><span></span><span class="sharing-screen-reader-text">Click to share on Twitter (Opens in new window)</span></a>
    </li>
    <li class="share-email">
      <a rel="nofollow" data-shared="" class="share-email sd-button share-icon no-text" href="http://usrlocal.com/2012/07/high-puppetmasterd-cpu-utilization/?share=email" target="_blank" title="Click to email this to a friend"><span></span><span class="sharing-screen-reader-text">Click to email this to a friend (Opens in new window)</span></a>
    </li>
    <li class="share-end">
    </li>
  </ul>
</div>

Share this: