• Call Ryan
• Email Mom
• Go get prints framed

And every time I look at it, I avoid it.  It’s not like these are complicated, deep-thinking, back-breaking kind of issues.  They’re just reminders to do some basic things that tend to slip through the crack when I spend all day in front of a computer reading or writing or staring at cat pictures.  Yet I still can’t help but reimagine these simple, and sometimes enjoyable tasks, as homework assignments.  The curse of the internet is the allure of the “new” just around every corner.  Oh a new Clojure article? Did Crockford give another interview about speed vs. readability in modern JavaScript? Did a new ROM for my phone come out?

Once in a while I’m able to construct a todo list and not view it as an assault on my personal freedom to do what I want, when I want.  Those few times I feel I’ve truly dedicated myself to the tasks at hand, so it seems fruitless to have expended the energy of making the list at all.  I would have been hard at work regardless of the piece of paper with little scribbles on it.  Maybe I’ll start doing this regularly and it will become more natural, but I’m not overly optimistic since it means the first item at the top of my mental todo list every day will be “write todo list for today.” I think I’ll do that after I have coffee and catch up on any funny cat pictures I missed.

The fog is getting denser every morning

Technical Debt, unlike personal debt, is essentially something your organization owes its future self. Much like government savings bonds purchsed during WWII to finance infrastructure and military buildup, you promise your future self that you’ll do some amount of work in exchange for cutting a corner now. When you look at it not as something the developers are borrowing from someone else, but as something you’re borrowing from yourself, you have to weight it accordingly. There will always be hidden or unexpected costs for having taken the easy way out of some problem, but at least take the time to carefully consider the known problems you’ll likely face. So what happens when you’re faced with insurmountable debt?

The parallel for crushing personal debt is declaring bankruptcy, but is this the right approach? Bankruptcy, in this case, means a rewrite. Maybe not ground up, but it does mean you’ll face a lot of second system problems. Rewrites are so alluring to developers, managers and pretty much everyone. However, they also strand your current product in its current state. If you intend to keep updating the old product, while you build a new replacement, you’ll have to saddle a bunch of your developers, probably some of your best ones, with the enviable task of maintaining a dying code base. One that you’ve decided is so bad that it’s unrecoverable and isn’t worth trying to fix. Truly, these engineers you’ve assigned this task will be grateful for the opportunity to “just keep the lights on” while other teams tackle building the replacement.

The other option is to hunker down and fix bugs. No new features until X bugs have been fixed. No more critical path bugs, no more middle of the night downtime, etc. Usually, management will declare this as the stated goal, and it sounds like such a miserable existence because it is. In the financial parallel, we think of it as the individual paying themselves out of debt month by month. In reality it’s more like a stagnant national economy which isn’t investing in itself. Remember, you owe yourself this time to do things right. Fixing bugs on an individual level isn’t the same as repaying your technical debt. Say you took a shortcut somewhere and built a quick version of a message system to keep two in-house systems in sync. In time, you needed to generalize it, but due to constraints you had to just copy/paste configuration files and code between projects without really abstracting things neatly.

Another aspect of long-term debt is that over time inflation rises can decrease the real burden of debt. If your one-off message bus turned out to be built for a feature users didn’t want in the end, it can die, and your debt is absolved painlessly. You saved the effort, accrued debt and over time, it became trivial that you had accumulated it. Technical debt matters most in high-interest scenarios, e.g. your core products and the libraries that comprise them.

What do you think, have you accumulated meaningful debt? Can you pay down your debt by dropping features that aren’t used or necessary? Or is it time to set aside resources and engineers for reinvestment to offset your debt?

• You (completely) control your environment
  This is one of the things I love most about working from home. You can throw that new LP you wanted to listen to (I’m a big fan of vinyl, which you can’t lug to an office). The temperature is too cold? Turn up the heat a little. You don’t like the shirt you’re wearing today? Go change. Of course, this also means that you can easily end up being drawn into one after another thing to get yourself settled. I can easily spend an hour making coffee, taking a shower, shaving, getting dressed and milling around the house before I actually sit down and start working. So you have to learn to block out the home distractions same as the office distractions.
• No one bothers you
  There’s probably not as many people in your house as a typical office during the daytime. This lack of people results in the surprisingly easy ability to work, uninterrupted, for hours at a time. No one bugs you to go get coffee, or to tell you about their new kitten, or  how adorable their kid was last night in the school play.  These distractions are there in the workplace, all the time, and you can’t avoid them entirely unless you’re physically absent. Once you go a few weeks without this forced interaction you’ll be so psyched at your productivity increase, but you may find yourself somewhat unable to cope with these basic distractions when you do go into an office or other busy environment. Force yourself to get out and go to a coffee shop or coworking space to get that base level of social interaction that we all require to function like human beings!
• Insane productivity
  Getting into the The Zone is difficult in an office, but at home it can be really easy. You’re comfortable, you know your surroundings well, and you can concentrate and really focus like nowhere else.  Then all of a sudden you’re done. Not done in the way you are at the end of the day at work. You’re really done, you’ve burned through two or three days worth of work in an hour and a half. Now you need to get the next thing done. Sure there’s always more work, but I always find myself uninspired to move on to the next thing when my brain is still so fully focused on the last thing.
• You chose what you pay attention
  Procrastination really is lurking around every corner at your house. Instead of getting to those emails, why not play with the cat for a while? Maybe make an epic sandwich for an early lunch? How about watching that TV show you recorded last week? It’s all waiting for you the moment you stop trying to get work done. Your awesome productivity can nicely offset this danger, but it sucks when you realize you wasted half your day waiting to get started because you just didn’t feel like really paying attention to anything work-related.

In general, I feel like it’s an uphill battle and you have to really pay attention to what you’re doing and how on or off task you are. It doesn’t mean you have to be working every moment of the day. The productivity boost means you can actually relax and take a nap, or actually watch that TV show you recorded last week. But if you aren’t aware of what you’re doing, you’ll quickly see your time slip away and you’ll have just spent the day in your pajamas watching all the episodes of BBC’s The Prisoner back-to-back…not that there’s anything wrong with that.

Fast forward to July, my girlfriend and I have been constantly trolling craigslist and we’re finally ready to pack it up and make the move happen.  We have temporary residence with relatives outside San Francisco proper, but that’s temporary.  So we start putting open houses on the calendar, and we’re both excited at the prospect of finding our dream apartment in our dream city. Oh to be young(er) and naive(also -er)!

Turns out, there’s a LOT of other people also trying to get apartments around here!  We were early for a few different showings and each time, the place was rented within 15minutes.  We started to become a big dejected.  Looking at apartments we didn’t like was becoming a necessity because at least the possibility of a home was there, even if it wasn’t perfect.  I think the lowest moment was after we’d been looking for about 10 days, been to a couple of showings in one day and all of them were in big, uncharming buildings with crappy hallways and they had all been cattle calls.  50+ people showed for a 15min look at an apartment in the Valencia corridor in the Mission.  All had checkbooks and pre-filled applications.  The place was probably rented before we event made it up to the apartment to look.

A few days later we see an *amazing* apartment come up on craigslist.  We email minutes after its posted (as did almost 100 other people we later found out).  But we were the first, and the man leasing the flat turned out to be rather principled and fair!  So we saw the apartment (bringing of course, all our credit history, pay stubs, bank info, mortgage statements, DNA profile, family criminal history, proof of residency, proof of whateveryouwantjustgiveusthisapartmentplease). It was worth it, we were prepared and on the cusp of full-throttle desperation we convinced the owner to allow us, our two cats and our rabbit to all move in starting in September!

So a few weeks and a lot of awful apartments and one glimmer of hope at the end of it all can lead you to your dream apartment in San Francisco, but patience and hard work are probably the only real tips other than be prepared I can offer.  Good luck, it’s a jungle out there.

Quick stats:

• 3210 miles in three days
• ~140 gallons of gas
• 23mpg average
• 63 mph average

Now on to the actual apartment search.  We’re staying outside the city until we actually find a spot, but we’re going to start blitzing open houses with all the piles of required documentation (employment letter, credit history, paystubs, rental app, blood type, recent personal internet usage history, etc).

First, take the opportunity to learn more about the code you’re trying to fix. Maybe it’s got that weird syntax for a reason, or maybe you’re processing some funky legacy data. Ask around for help (or try to remember why you wrote it) and you might suddenly find that it’s not broken at all. Look at old commits to your source control repo (you always write useful check-in messages right?). If the logic behind the weird/bad/ugly code turns out to be missing entirely, you might find that it’s just bad code because everyone is afraid to fix it. That’s the cue to dig in and set things right.

Refactor the existing code by splitting big methods up into smaller ones.  Introduce new classes and files to fix broken patterns, but try to leave the existing logic in place where that important business logic is handled. Add comments explaining the code as you see it and more importantly, find our who wrote it (svn blame ftw) and what they intended it to do. Along the way introduce some sane(r) unit tests to actually verify that your code is a drop-in replacement for the existing one. Once you’ve made the world a saner place, you can confidently add that new feature or fix that bug. Best of all, you’ve left it a little better than when you got there. Being a good programmer is often about being a good citizen, and your source code is your community. If you don’t pick up the trash, then who will?

The biggest victory this brings about is that you’ll better understand the intricacies of your product and you’ll have more maintainable and understandable code. This will pay dividends well past the current project or the next few sprints and help improve velocity for anyone touching this code. Most importantly, it tears down imagined barriers that prevent good developers from fixing problems they know about. So don’t be afraid to do the right thing, especially if you have unit tests to back you up

Make an API. Whether you’re creating a new RESTful HTTP service or using ProtoBuf/Thrift it’s worth it to put a layer between your raw data source and your consumers. By creating a layer between your consumers and your data system, you can insulate client code from all sorts of issues related to data management.  You can add cache layers, shard databases, even switch the entire data storage system behind the API while the client stays blissfully unmodified. All of these things put you on the hook for making a fast and reliable API, but the benefits of being able to swap moving parts behind the scenes are invaluable. Furthermore, by making an API for your data, you also make it easier for other developers (internal and external) to make use of your data. Internal consumers will be able to build tighter integration between systems within your product(s). External developers can start to use your systems data to build plugins and add-ons that add value for your customers in ways you hadn’t thought of yet. Of course with all these other developers and teams accessing your data now…

What you expect to happen is not what will happen. In rebuilding some other parts of the Lead storage system my team at also built a RESTful API to modify and retrieve lead data. We thought it would receive relatively moderate usage internally and very light usage externally. A pair of load-balanced tomcat servers could easily handle the 10,000 requests per day we were expecting. Instead we saw internal usage spike well over 100,000 requests in the first few weeks, forcing us to modify and add capacity to the “scalable” system in a variety of ways. Now over a year later we have over 200 HubSpot customers and dozens of internal customers tallying over 1 million requests and serving up a couple gigs worth of data per day. This isn’t what we had expected for this API, and as such it’s got a lot of rough edges, some which have been sanded down, others which still pose hazards to internal and external users.

Over-engineering vs under-delivering. If your system takes a year to design and another to build a first version and its “infinitely” scalable, your business better be capable of making that investment and continuing to run in the meantime. If you’re like just about every other company smaller than Google or Microsoft, your development team can’t wait that long. Google’s BigTable is an amazing work of computer science and software engineering, but I doubt few established companies and even fewer startups can sustain the rest of their business while waiting on a team to deliver such an amazing work. The other side to the coin is under-delivering.  Designing to your current volume of traffic and requests is a sure way to waste your time.  At the very least plan for double, if not an order of magnitude more, requests/traffic/data than your current system sustains. If you’re building something entirely green field, consider what would happen if you opened up access to this data through an API layer to external developers. Would you see 10 requests per day? 100? 100,000? Think big enough to know that you won’t be up until 3am every night patching your system in a desperate attempt to keep it efficient and high-performing as usage grows.

Experiment to solve problems, not for the sake of experimenting. There’s so many cool new NoSQL or Key-Value data storage projects out now that want to replace your relational database. In the world of developing data-driven web applications, there are a lot of applications of this new technology. However, it doesn’t mean that every project is well suited for this model. Building an in-browser IM client? Key value stores sound like a great way to go for storing the messages for quick retrieval. Building a white-label inventory management console? It could still work ok, but you’re in the gray area. That new CRM for non-profit dog walking businesses? You’re sounding more like you’re in the relational model’s sweet spot. Shiny new things are shiny, and new, but many successful technologies are successful because they’ve been proven to work very well for a wide range of applications.

This is by no means the blueprint to successful data system design, its just the start of some guidelines I intend to use as I build new, bigger systems to replace the ones I’ve already built. We’re working on some awesome new stuff at HubSpot and I can’t wait to post more about what we build. I’m sure we’ll try some new stuff that won’t fit either because we don’t sufficiently understand it or because its ill suited for our data model. I’ll try keep this blog updated as we come across cool new findings (of which expect there will be many).