Puppetmasterd – High CPU utilization

Logged into a Puppet master server today and noticed that I had some high CPU utilization happening. This is rather odd as the box usually just chugs along with no worries what-so-ever.

Turns out, some others noticed this as well and some had suspicions that it was leap second related. Check out the full thread here.

Looking back at my SAR records, I find that my CPU spiked right around June 30th at 7PM which is July 1, Midnight GMT.

#sar -q -f /var/log/sa/sa30

03:20:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
03:30:01 PM         1       202      0.00      0.00      0.00
03:40:01 PM         1       202      0.00      0.00      0.00
03:50:01 PM         1       202      0.00      0.00      0.00
04:00:01 PM         1       202      0.00      0.00      0.00
04:10:01 PM         2       204      0.00      0.00      0.00
04:20:01 PM         1       202      0.00      0.00      0.00
04:30:01 PM         1       202      0.00      0.00      0.00
04:40:01 PM         1       202      0.00      0.00      0.00
04:50:01 PM         1       202      0.00      0.00      0.00
05:00:01 PM         1       202      0.00      0.00      0.00
05:10:01 PM         2       205      0.00      0.00      0.00
05:20:02 PM         1       202      0.00      0.01      0.00
05:30:01 PM         1       202      0.00      0.00      0.00
05:40:01 PM         1       202      0.00      0.00      0.00
05:50:01 PM         1       202      0.00      0.00      0.00
06:00:01 PM         1       202      0.00      0.00      0.00
06:10:01 PM         2       205      0.00      0.00      0.00
06:20:01 PM         1       202      0.00      0.00      0.00
06:30:01 PM         1       202      0.00      0.00      0.00
06:40:01 PM         1       202      0.00      0.00      0.00
06:50:01 PM         1       202      0.00      0.00      0.00
07:00:01 PM         4       202      0.32      0.07      0.02
07:10:01 PM         4       205      2.45      2.16      1.19

07:10:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
07:20:01 PM         2       202      2.25      2.32      1.73
07:30:01 PM         3       202      2.25      2.42      2.06
07:40:01 PM         2       202      2.42      2.47      2.25
07:50:01 PM         2       202      2.45      2.46      2.33
08:00:01 PM         3       202      2.42      2.48      2.39
08:10:02 PM         3       205      2.25      2.39      2.38
08:20:01 PM         3       202      2.48      2.52      2.45
08:30:01 PM         3       202      2.62      2.49      2.44
08:40:01 PM         2       202      2.49      2.49      2.45
08:50:01 PM         3       202      2.45      2.48      2.46
09:00:01 PM         3       202      2.46      2.51      2.47

Son of a bitch!

The simple solution was to give it a quick reboot. Its just a test puppet installation anyway, but still, pisses me off rebooting boxes like that.

Anyone else have this issue?

Matt Patterson avatar
About Matt Patterson
Husband, Father of 3, Programmer at heart, spends his days running ridiculously large data centers in the midwest.