RRDtool is insanely useful. If you’re looking after lots of servers like I have to, it helps to be able to graph various system stats both to placate management, and to plan for upgrades, failures, etc.
I got the following plate of multicoloured spaghetti by graphing the User CPU of a whole bunch of servers using SNMP, then averaging use across the cluster. It’s interesting to note that although individual servers will often spike up to almost 100% usage, once their process dies and drops back down to zero the others take up the load immediately, quite nicely illustrating how well the load balancer is working! This pattern is repeated across network interfaces and memory utilisation as well.
CPU Utilisation, Averaged across a cluster