We’ve done a lot of infrastructure work on Instrumental over the last few weeks. Our main goals with this infrastructure work were making Instrumental’s data collection and writing process more resilient and easier to scale. As an application monitoring service, we process billions of data points every single day, and we would love to write more data in less time.
Toku’s marketing page promises a ton of things that sound way better than standard MongoDB – most notably, 20x faster performance (without tuning, no less!) and a 90% reduction in database size.
Sure, let’s try this thing out!
But first, let’s test our assumptions. We don’t actually want to push Toku into production before we’re reasonably sure it will outperform our current implementation of MongoDB.
An important component of successful application monitoring is using data to make better decisions. We’ve already got a graph that tracks how fast our MongoDB install is writing to disk. This gives us a ton of useful historical data, so we can measure Toku’s performance against Mongo’s performance, and prove whether Toku is actually an improvement over our MongoDB implementation.
Side note: we’re not exactly running a standard MongoDB instance here. Our dev team is very familiar with Mongo, and we’ve squeezed quite a bit of performance improvements out of it over the years. Still, Toku seems worth trying out. To the graphs!
Here’s the graph we use to track database write speed for Instrumental, measured in Mongo updates per second, and we’re graphing this number in 5 minute increments. On this graph, higher is better, and if we’re seeing a lot of big valleys, we know we’ve got a problem writing data somewhere.
I mentioned we’re not really running a standard install of MongoDB, so we aren’t really expecting 20x improvement. That would be AWESOME, but it’s not realistic. Still, Toku’s specifically designed to improve performance for high scale applications, so we should see SOME bump. Let’s take a look!
Based on our current graph, we do expect to see some boost in performance. We’re assuming the Toku graph will look similar to our current Mongo graph, just better. Let’s find out!
We dropped in Toku on one of our database boxes, and let it percolate for a day or so to get an idea of what performance would look like. I should note we’re running Mongo and Toku on identical hardware here. This would be a pretty flawed test if we weren’t!
Here’s what this graph looks like, once we start charting Toku performance:
Sure, sometimes Toku is performing better than MongoDB, but do you see how spiky those write speeds are? Way spikier than we’d like, even when compared to how fast MongoDB can write to disk. Total average write speeds were lower: a single process in Mongo could perform around 6,000 upserts per second.
At first, Toku performed very well – we were seeing speeds of 10,000 upserts per second or so, but this slowly dwindled to close to 3,000 to 6,000 upserts per second. After letting this run for a while, we were seeing roughly a 20% decline in write speeds.
We tried different compressions settings – zlib, fastlz, and no compression. Nothing we tried had much impact on upsert speed, although it could be that the effects were delayed since the reindex commands returned near instantaneously. After a day of testing and performance monitoring, it became clear Toku wasn’t right for Instrumental.
This isn’t to say we don’t like Toku – we were drawn in by the performance boosts we thought we could realize, and it was a lot of fun to work with. Admin work was a lot of fun, and we found it to be very fast and responsive.
Still, our assumptions around possible improvements were absolutely worth testing, and we’re glad we had Instrumental to help us realize Toku wasn’t right for us.