Puppetmasterd – High CPU utilization

Logged into a Puppet master server today and noticed that I had some high CPU utilization happening. This is rather odd as the box usually just chugs along with no worries what-so-ever.

Turns out, some others noticed this as well and some had suspicions that it was leap second related. Check out the full thread here.

Looking back at my SAR records, I find that my CPU spiked right around June 30th at 7PM which is July 1, Midnight GMT.

#sar -q -f /var/log/sa/sa30

03:20:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
03:30:01 PM         1       202      0.00      0.00      0.00
03:40:01 PM         1       202      0.00      0.00      0.00
03:50:01 PM         1       202      0.00      0.00      0.00
04:00:01 PM         1       202      0.00      0.00      0.00
04:10:01 PM         2       204      0.00      0.00      0.00
04:20:01 PM         1       202      0.00      0.00      0.00
04:30:01 PM         1       202      0.00      0.00      0.00
04:40:01 PM         1       202      0.00      0.00      0.00
04:50:01 PM         1       202      0.00      0.00      0.00
05:00:01 PM         1       202      0.00      0.00      0.00
05:10:01 PM         2       205      0.00      0.00      0.00
05:20:02 PM         1       202      0.00      0.01      0.00
05:30:01 PM         1       202      0.00      0.00      0.00
05:40:01 PM         1       202      0.00      0.00      0.00
05:50:01 PM         1       202      0.00      0.00      0.00
06:00:01 PM         1       202      0.00      0.00      0.00
06:10:01 PM         2       205      0.00      0.00      0.00
06:20:01 PM         1       202      0.00      0.00      0.00
06:30:01 PM         1       202      0.00      0.00      0.00
06:40:01 PM         1       202      0.00      0.00      0.00
06:50:01 PM         1       202      0.00      0.00      0.00
07:00:01 PM         4       202      0.32      0.07      0.02
07:10:01 PM         4       205      2.45      2.16      1.19

07:10:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
07:20:01 PM         2       202      2.25      2.32      1.73
07:30:01 PM         3       202      2.25      2.42      2.06
07:40:01 PM         2       202      2.42      2.47      2.25
07:50:01 PM         2       202      2.45      2.46      2.33
08:00:01 PM         3       202      2.42      2.48      2.39
08:10:02 PM         3       205      2.25      2.39      2.38
08:20:01 PM         3       202      2.48      2.52      2.45
08:30:01 PM         3       202      2.62      2.49      2.44
08:40:01 PM         2       202      2.49      2.49      2.45
08:50:01 PM         3       202      2.45      2.48      2.46
09:00:01 PM         3       202      2.46      2.51      2.47

Son of a bitch!

The simple solution was to give it a quick reboot. Its just a test puppet installation anyway, but still, pisses me off rebooting boxes like that.

Anyone else have this issue?