Deciding what to measure is hard, and even daunting at first. There’s a ton of code in your project and you don’t want to just slap a gaggle of useless metrics in there. Your measurement should mean something, dammit! On the other hand, it would be nice if it didn’t take forever to get started :)
Don’t worry – getting started doesn’t have to take forever. Your code base is a great place to get inspiration for measurement, so let’s take a look at the usual suspects!
1) Your test suite
You’ve already thought long and hard about what’s worth protecting in your application (or someone has, hopefully). It’s impossible for me to guess the unique and beautiful snowflakes that make up your test suite, but I’m sure you know them well. Look for the code with lots of tests.
Bonus – this is a great opportunity to deal with Heisenbugs and code that has a history of acting differently in production. Nothing makes dealing with hard-to-solve problems easier than a huge body of relevant data.
2) Your “features”
Add a metric for each of the features in your application. On your first pass, keep the features high-level: “created a game”, “skipped turn”, etc. An unexpected change in feature usage can be a good indicator of subtle (or not so subtle) problems. It’s easy to add more detailed monitoring as you figure out which features are the best indicators of application performance.
3) Your code coverage
The inverse of your code coverage, actually. The uncovered code in your application is the interesting part. There’s a high likelihood that the code that isn’t covered by your automated tests falls into one of two categories:
A) Code that’s really inconvenient to test in development
B) Defensive code to protect against worst case scenarios
It’s often much easier to instrument these with your application monitoring tool than to try and figure out a test. While you’re instrumenting these, you should also set up alerts. It’s code that you’re pretty sure works. If it doesn’t, you want to know as soon as possible.
There are many more things that are worth instrumenting, but these three things will have you off to a good start. Remember: good application monitoring is a process, not a product. You can’t think of everything up front. If you start with your best ideas and add to it over time, your application monitoring will grow in value just like a good test suite.