Success Via Failure

Posted February 19th, 2014 at 9:37 pm by Matt

Lately, I have found myself learning a lot by screwing up. Specifically, when coding. (Though, as I write this, my wife and I have a 3-month old son, Jack. She might disagree with me scoping this idea to coding. He might as well.)

Recently, I started down the proverbial rabbit-hole trying to speed things up in the financials for LaunchPlan. However, I had to scrap all of my initial code and write off a several hours of effort too. But, I did eventually get to my goal, and I learned quite a bit along the way.

I was trying to speed up a Key-Value store that was currently residing inside a relational SQL database. Actually, I had this thought in the back of my mind for quite a while. The trigger to try to attack this came recently after I read an article comparing Redis, Memcached, and MongoDb. This article got me thinking: Redis could be a perfect solution for this – and I could implement it at the data layer and keep things clean and pretty simple.

I plugged in ServiceStack.Redis and jumped into my code. It took a little while to port over the code in the data layer and setup a legitimate test. I wanted to compare speed in EC2 because in the past I found that local testing for this kind of thing can lead to false positives and other bad things. I also, admittedly, got a bit sidetracked by cleaning up some ugly code that hadn’t been touched in a while. But after a few hours I had things lined up and it was time to compare the old ugliness vs. the new hotness.

I had made things so. much. worse. I mean, not even close. 4x worse! How could that be?

Well, as it turns out I wasn’t comparing apples to apples. I forgot my relational SQL had Memcached in front of it in production. I also quickly learned that a small latency talking to a separate Redis instance multiplied by the number of indexes I had created in an effort to “simplify” my data… = bad. Memcached was on par with Redis for speed already – and my changes + latency * frequency yeilded a much, much slower result.

But, there was a silver lining! I had long since forgotten that during my initial build this KV model was setup to use a complex key – and in SQL I could use “like” queries to extract sets of key/values the way I needed to. This was ugly and yielded more reliance on Memcached that was necessary.

So, frustration set aside, I figured out the best path forward. Keep Memcached in place (as it was already in use elsewhere), break up this complex key and index it’s parts, and keep things in the existing relational database. I avoided adding a new component (Redis) to the stack and still managed to squeeze out a 35% speed increase through better code and logic. Plus, I learned a bunch about Redis along the way!

Later on I discovered my entire caching model was only surface level – and after some more digging I added some deeper business object level caching and grabbed another 20% boost in speed. While my original goal of setting out to implement Redis for speed increases was a total flop – this effort led me down a path that I might not have otherwise arrived at – and taught me something new while yielding the speed increase I was looking for. Not bad!

This certainly isn’t the only time this has happened. It just seems to be commonplace lately. Perhaps this is because I’ve been prematurely mistaking a potential solution for exploration – which can be, but is often not, the same thing. While I am working on defining exploration a bit earlier when I can, I can’t help but to enjoy those times when things neatly come together in the end.

I suppose you could abstract this idea and apply it to life in general. Or perhaps this really only applies to learning new things? Then again, it doesn’t seem right given the stigma failure has. It seems, to me, too hastily discounted and avoided. I’m not the first to say this, but taking the safe route isn’t going to teach you or get you much…beyond a safe destination.

Being a father now, I wonder how best to teach Jack that failure isn’t only an outcome – but often an opportunity? Through this small example, I suppose, I can tell him in some cases you can defuse any fear of failure by labeling it learning or exploration. #lifehack