The foggy season has arrived

instamatic camera in front of foggy background

The fog is getting denser every morning

Posted in Technology | Leave a comment

Technical Austerity

I’ve been reading a lot of economics articles recently and have been thinking about our current debt crisis. From a Keynesian perspective, the government should be heavily investing anywhere it can to offset the lack of private investment. The politically popular austerity movements in Europe and the US argue that the fundamental problem is lack of savings, and that higher deficits will not raise short-term demand, but bring long term ruin to these economies. Macroeconomic policy is way to complex of a topic for me to comment on, but I can say a thing or two about Technical Debt and the investment parallel.

Technical Debt, unlike personal debt, is essentially something your organization owes its future self. Much like government savings bonds purchsed during WWII to finance infrastructure and military buildup, you promise your future self that you’ll do some amount of work in exchange for cutting a corner now. When you look at it not as something the developers are borrowing from someone else, but as something you’re borrowing from yourself, you have to weight it accordingly. There will always be hidden or unexpected costs for having taken the easy way out of some problem, but at least take the time to carefully consider the known problems you’ll likely face. So what happens when you’re faced with insurmountable debt?

The parallel for crushing personal debt is declaring bankruptcy, but is this the right approach? Bankruptcy, in this case, means a rewrite. Maybe not ground up, but it does mean you’ll face a lot of second system problems. Rewrites are so alluring to developers, managers and pretty much everyone. However, they also strand your current product in its current state. If you intend to keep updating the old product, while you build a new replacement, you’ll have to saddle a bunch of your developers, probably some of your best ones, with the enviable task of maintaining a dying code base. One that you’ve decided is so bad that it’s unrecoverable and isn’t worth trying to fix. Truly, these engineers you’ve assigned this task will be grateful for the opportunity to “just keep the lights on” while other teams tackle building the replacement.

The other option is to hunker down and fix bugs. No new features until X bugs have been fixed. No more critical path bugs, no more middle of the night downtime, etc. Usually, management will declare this as the stated goal, and it sounds like such a miserable existence because it is. In the financial parallel, we think of it as the individual paying themselves out of debt month by month. In reality it’s more like a stagnant national economy which isn’t investing in itself. Remember, you owe yourself this time to do things right. Fixing bugs on an individual level isn’t the same as repaying your technical debt. Say you took a shortcut somewhere and built a quick version of a message system to keep two in-house systems in sync. In time, you needed to generalize it, but due to constraints you had to just copy/paste configuration files and code between projects without really abstracting things neatly.

Another aspect of long-term debt is that over time inflation rises can decrease the real burden of debt. If your one-off message bus turned out to be built for a feature users didn’t want in the end, it can die, and your debt is absolved painlessly. You saved the effort, accrued debt and over time, it became trivial that you had accumulated it. Technical debt matters most in high-interest scenarios, e.g. your core products and the libraries that comprise them.

What do you think, have you accumulated meaningful debt? Can you pay down your debt by dropping features that aren’t used or necessary? Or is it time to set aside resources and engineers for reinvestment to offset your debt?

Posted in Technology | Leave a comment

Why I love (and hate) working from home

Since moving to San Francisco a few months ago I’ve been working out of my house. It’s pretty awesome, I have done it once before for an extended period of time so I kind of know my way around the home office thing.  You need to setup routines, get out occaisionally, and generally dedicate some time to stepping away from work to keep yourself sane. This time has been more difficult because it’s coincided with a cross-country move, which brought its own set of challenges. Now that the dust has settled and the boxes are unpacked I’ve fallen into a pretty good routine. Of course, extended periods of WFH time result in little joys and annoyances that may not surface when you spend a day a week in the home office. Here’s just a few of my favorites.

  • You (completely) control your environment
    This is one of the things I love most about working from home. You can throw that new LP you wanted to listen to (I’m a big fan of vinyl, which you can’t lug to an office). The temperature is too cold? Turn up the heat a little. You don’t like the shirt you’re wearing today? Go change. Of course, this also means that you can easily end up being drawn into one after another thing to get yourself settled. I can easily spend an hour making coffee, taking a shower, shaving, getting dressed and milling around the house before I actually sit down and start working. So you have to learn to block out the home distractions same as the office distractions.
  • No one bothers you
    There’s probably not as many people in your house as a typical office during the daytime. This lack of people results in the surprisingly easy ability to work, uninterrupted, for hours at a time. No one bugs you to go get coffee, or to tell you about their new kitten, or  how adorable their kid was last night in the school play.  These distractions are there in the workplace, all the time, and you can’t avoid them entirely unless you’re physically absent. Once you go a few weeks without this forced interaction you’ll be so psyched at your productivity increase, but you may find yourself somewhat unable to cope with these basic distractions when you do go into an office or other busy environment. Force yourself to get out and go to a coffee shop or coworking space to get that base level of social interaction that we all require to function like human beings!
  • Insane productivity
    Getting into the The Zone is difficult in an office, but at home it can be really easy. You’re comfortable, you know your surroundings well, and you can concentrate and really focus like nowhere else.  Then all of a sudden you’re done. Not done in the way you are at the end of the day at work. You’re really done, you’ve burned through two or three days worth of work in an hour and a half. Now you need to get the next thing done. Sure there’s always more work, but I always find myself uninspired to move on to the next thing when my brain is still so fully focused on the last thing.
  • You chose what you pay attention
    Procrastination really is lurking around every corner at your house. Instead of getting to those emails, why not play with the cat for a while? Maybe make an epic sandwich for an early lunch? How about watching that TV show you recorded last week? It’s all waiting for you the moment you stop trying to get work done. Your awesome productivity can nicely offset this danger, but it sucks when you realize you wasted half your day waiting to get started because you just didn’t feel like really paying attention to anything work-related.

In general, I feel like it’s an uphill battle and you have to really pay attention to what you’re doing and how on or off task you are. It doesn’t mean you have to be working every moment of the day. The productivity boost means you can actually relax and take a nap, or actually watch that TV show you recorded last week. But if you aren’t aware of what you’re doing, you’ll quickly see your time slip away and you’ll have just spent the day in your pajamas watching all the episodes of BBC’s The Prisoner back-to-back…not that there’s anything wrong with that.

Posted in General | Leave a comment

Test Driven Refactoring

I was working on refactoring and updating some code that was written by another team of developers today and eventually found myself in the whack-a-mole cycle of “Commit-Ship to QA-Find Bugs-Commit” for a few iterations. The project had unit tests, but they somehow didn’t cover the cases that were coming up after a few shared methods signatures had changed. I realized that through the (correct at the time) use of mock objects to get tests working at all, the tests had managed to diverge from its actual runtime behavior. It’s not just the responsibility of the initial developers to write unit tests to verify their work, it should be the responsibility of everyone who works on a project. You’ll never get perfect coverage, but if you keep it up to date, and enforce that tests must pass for the project to ship, you might have a chance at keep your code maintainable!

Posted in Technology | Leave a comment

Finding an apartment is even harder than they said

Back in June I attended O’Reilly Velocity in San Jose and on the first night amidst all the great Ignite talks, there was one titled “How to find a place to live in San Francisco“.  As with most Ignite talks, it was fast paced, witty and actually offered some meaningful (if slightly sarcastic) advice on the topic. His summary: bring every piece of info about yourself and a deposit check and just pray you get the place, not exactly uplifting, but I figured this was hyperbole for the sake of entertainment on the presenter’s part. By that point both my employer and I knew I was moving out here and my girlfriend and I had started the daily ritual of combing through apartment listings on craigslist to imagine our lives out here.

Fast forward to July, my girlfriend and I have been constantly trolling craigslist and we’re finally ready to pack it up and make the move happen.  We have temporary residence with relatives outside San Francisco proper, but that’s temporary.  So we start putting open houses on the calendar, and we’re both excited at the prospect of finding our dream apartment in our dream city. Oh to be young(er) and naive(also -er)!

Turns out, there’s a LOT of other people also trying to get apartments around here!  We were early for a few different showings and each time, the place was rented within 15minutes.  We started to become a big dejected.  Looking at apartments we didn’t like was becoming a necessity because at least the possibility of a home was there, even if it wasn’t perfect.  I think the lowest moment was after we’d been looking for about 10 days, been to a couple of showings in one day and all of them were in big, uncharming buildings with crappy hallways and they had all been cattle calls.  50+ people showed for a 15min look at an apartment in the Valencia corridor in the Mission.  All had checkbooks and pre-filled applications.  The place was probably rented before we event made it up to the apartment to look.

A few days later we see an *amazing* apartment come up on craigslist.  We email minutes after its posted (as did almost 100 other people we later found out).  But we were the first, and the man leasing the flat turned out to be rather principled and fair!  So we saw the apartment (bringing of course, all our credit history, pay stubs, bank info, mortgage statements, DNA profile, family criminal history, proof of residency, proof of whateveryouwantjustgiveusthisapartmentplease). It was worth it, we were prepared and on the cusp of full-throttle desperation we convinced the owner to allow us, our two cats and our rabbit to all move in starting in September!

So a few weeks and a lot of awful apartments and one glimmer of hope at the end of it all can lead you to your dream apartment in San Francisco, but patience and hard work are probably the only real tips other than be prepared I can offer.  Good luck, it’s a jungle out there.

 

Posted in Technology | Leave a comment

Cross country adventure

My lovely girlfriend and I have made it to California!  We did it Cannonball Run style, three days of driving 18+ hours.  A few stops for food, gas and bathrooms each day.  One less-than enjoyable motel in Illinois, and one rather nice, cheap hotel in Wyoming.  Lots of beautiful scenery and some really cool transitions.

Quick stats:

  • 3210 miles in three days
  • ~140 gallons of gas
  • 23mpg average
  • 63 mph average

Now on to the actual apartment search.  We’re staying outside the city until we actually find a spot, but we’re going to start blitzing open houses with all the piles of required documentation (employment letter, credit history, paystubs, rental app, blood type, recent personal internet usage history, etc).

Posted in Moving | Tagged , , | Leave a comment

Moving across the country

Expect some rather exciting and nerve wracking updates coming in the next few months as my lovely girlfriend, our two cats and of course our ever-dignified rabbit move across the country! I’m from the Cambridge area and have lived here pretty much my entire life and am quite excited to find new favorite coffee shops, new secret spots to explore and most of all, to finally get a chance to experience year-round wonderful weather. We’ll be keeping our jobs at HubSpot while moving and will be looking for coworking spaces once we get out there, but for now, the focus is on renting our condo, finding a place in San Francisco (Noe Valley or Mission ideally) and packing up all of our things.

Posted in General | Tagged , | Leave a comment

Fearlessness is a good thing, sometimes.

In the world of startups the short deadlines and shifting business requirements seem to result in some pretty gnarly code in your organization’s repository. Eventually, you’ll have the unfortunate task of adding a feature or fixing a bug in this code. If you’re really unfortunate, you’ll either have never worked on said code, or only worked on it a long time ago. Every place I’ve worked at has that code. It’s something that turned out to be a critical, but somewhat static piece of the product. There’s a lot of manifestations of this: it’s written in a different language than the current standard, or it’s using old conventions, or it’s just written by a more naive or rushed programmer (e.g. yourself, a few years previous).  When you look at it you just know it’s wrong. Every time someone opens up these files, they sigh quietly to themselves and try to do as little damage as possible and get out of there as quickly and quietly as possible. The fear of making this critical piece of software break overpowers every good programmer’s urges to fix it. Your task, should you chose to accept it, is to stop being afraid and start fixing things.

First, take the opportunity to learn more about the code you’re trying to fix. Maybe it’s got that weird syntax for a reason, or maybe you’re processing some funky legacy data. Ask around for help (or try to remember why you wrote it) and you might suddenly find that it’s not broken at all. Look at old commits to your source control repo (you always write useful check-in messages right?). If the logic behind the weird/bad/ugly code turns out to be missing entirely, you might find that it’s just bad code because everyone is afraid to fix it. That’s the cue to dig in and set things right.

Refactor the existing code by splitting big methods up into smaller ones.  Introduce new classes and files to fix broken patterns, but try to leave the existing logic in place where that important business logic is handled. Add comments explaining the code as you see it and more importantly, find our who wrote it (svn blame ftw) and what they intended it to do. Along the way introduce some sane(r) unit tests to actually verify that your code is a drop-in replacement for the existing one. Once you’ve made the world a saner place, you can confidently add that new feature or fix that bug. Best of all, you’ve left it a little better than when you got there. Being a good programmer is often about being a good citizen, and your source code is your community. If you don’t pick up the trash, then who will?

The biggest victory this brings about is that you’ll better understand the intricacies of your product and you’ll have more maintainable and understandable code. This will pay dividends well past the current project or the next few sprints and help improve velocity for anyone touching this code. Most importantly, it tears down imagined barriers that prevent good developers from fixing problems they know about. So don’t be afraid to do the right thing, especially if you have unit tests to back you up ;)

Posted in Technology | Tagged , | Leave a comment

Building scalable data systems

In my tenure at HubSpot I’ve been on teams that have built and rebuilt various data systems and every time we’ve tried to construct “scalable” solutions. We’ve hit the mark on some projects and been wildly off on others. I’m on another such project and a few things I’ve learned seemed important enough to share, or at least write down to remind myself later.

Make an API. Whether you’re creating a new RESTful HTTP service or using ProtoBuf/Thrift it’s worth it to put a layer between your raw data source and your consumers. By creating a layer between your consumers and your data system, you can insulate client code from all sorts of issues related to data management.  You can add cache layers, shard databases, even switch the entire data storage system behind the API while the client stays blissfully unmodified. All of these things put you on the hook for making a fast and reliable API, but the benefits of being able to swap moving parts behind the scenes are invaluable. Furthermore, by making an API for your data, you also make it easier for other developers (internal and external) to make use of your data. Internal consumers will be able to build tighter integration between systems within your product(s). External developers can start to use your systems data to build plugins and add-ons that add value for your customers in ways you hadn’t thought of yet. Of course with all these other developers and teams accessing your data now…

What you expect to happen is not what will happen. In rebuilding some other parts of the Lead storage system my team at also built a RESTful API to modify and retrieve lead data. We thought it would receive relatively moderate usage internally and very light usage externally. A pair of load-balanced tomcat servers could easily handle the 10,000 requests per day we were expecting. Instead we saw internal usage spike well over 100,000 requests in the first few weeks, forcing us to modify and add capacity to the “scalable” system in a variety of ways. Now over a year later we have over 200 HubSpot customers and dozens of internal customers tallying over 1 million requests and serving up a couple gigs worth of data per day. This isn’t what we had expected for this API, and as such it’s got a lot of rough edges, some which have been sanded down, others which still pose hazards to internal and external users.

Over-engineering vs under-delivering. If your system takes a year to design and another to build a first version and its “infinitely” scalable, your business better be capable of making that investment and continuing to run in the meantime. If you’re like just about every other company smaller than Google or Microsoft, your development team can’t wait that long. Google’s BigTable is an amazing work of computer science and software engineering, but I doubt few established companies and even fewer startups can sustain the rest of their business while waiting on a team to deliver such an amazing work. The other side to the coin is under-delivering.  Designing to your current volume of traffic and requests is a sure way to waste your time.  At the very least plan for double, if not an order of magnitude more, requests/traffic/data than your current system sustains. If you’re building something entirely green field, consider what would happen if you opened up access to this data through an API layer to external developers. Would you see 10 requests per day? 100? 100,000? Think big enough to know that you won’t be up until 3am every night patching your system in a desperate attempt to keep it efficient and high-performing as usage grows.

Experiment to solve problems, not for the sake of experimenting. There’s so many cool new NoSQL or Key-Value data storage projects out now that want to replace your relational database. In the world of developing data-driven web applications, there are a lot of applications of this new technology. However, it doesn’t mean that every project is well suited for this model. Building an in-browser IM client? Key value stores sound like a great way to go for storing the messages for quick retrieval. Building a white-label inventory management console? It could still work ok, but you’re in the gray area. That new CRM for non-profit dog walking businesses? You’re sounding more like you’re in the relational model’s sweet spot. Shiny new things are shiny, and new, but many successful technologies are successful because they’ve been proven to work very well for a wide range of applications.

This is by no means the blueprint to successful data system design, its just the start of some guidelines I intend to use as I build new, bigger systems to replace the ones I’ve already built. We’re working on some awesome new stuff at HubSpot and I can’t wait to post more about what we build. I’m sure we’ll try some new stuff that won’t fit either because we don’t sufficiently understand it or because its ill suited for our data model. I’ll try keep this blog updated as we come across cool new findings (of which expect there will be many).

Posted in Technology | Tagged , , , | 1 Comment

On Traveling And Being A Nerd

I love traveling, the thrill of seeing new and unfamiliar places and people is really unbeatable.  The downside is that I’ve spent a lot of time cultivating an environment in which I thrive as an engineer (for more info on “the cave” and related issues see: http://randsinrepose.com/archives/2007/11/11/the_nerd_handbook.html) and I can’t bring that environment with me when globe-trotting.  I’ll often find myself hating being away from the things I find familiar, annoyed because I can’t control all the variables. When I’m home I’m constantly checking my email, writing code and basically living life through a keyboard and internet connection. When traveling, I often find myself introverting and getting snippy because I’m basically going through withdrawal from my daily routines.  Time changes, different cuisine, tea instead of coffee, coffee that’s too weak, coffee that’s too strong, temperature too hot, too cold, etc. I’m definitely getting better, but I’m come up with a few methods that seem to have helped on recent trips.

*  Bring technology of some sort with you that serves a useful purpose – On our recent trip to Iceland, I brought my new Nook Color. I loaded it with an offline maps program and a pretty good travel book I bought through Amazon Kindle.  It was very useful to have interactive maps we could use in conjunction with our print and digital guidebooks.  It also helped my inner nerd feeling in touch with a familiar piece of technology, and in turn, a little more in control of the world around me.

* Recognize your frustration and squash it – I get snippy when I get hungry.  My partner knows this, she reminds me to eat a snack and I get more annoyed because someone is telling me what to do. I’ve learned to just let it go, eat a Cliff Bar and enjoy the scenery. It’s not easy, and I don’t profess to have stopped entirely, or to expect that I won’t ever get annoyed, but at least I’m making progress and able to get out of my own head long enough to stop acting like a jerk.

* Slow down and relax (at least for a little while) – My partner and I do whirlwind tours when we travel.  This usually means short stays in a ton of different places on a 10 day vacation.  When we went to Costa Rica last year, we never stayed in any place more than 2 nights and we were there for 11 days. But even at that pace, we managed to take naps, enjoy the rain forest, the cloud forests, black sand beaches, and quiet nights in a little beach town.  The only time I ever felt we were rushing was when we were trying to get to something we were required to schedule. Taking time to walk around, enjoy a city or town in the evening or early morning really takes the anxiety out of being away from my normal environment.

I really enjoy traveling, especially with my girlfriend and can’t wait for our next trip. Hopefully I’ll keep myself from succumbing to low blood sugar and enjoy the good decompression time that comes with being away from work and the internet in general.

Posted in Travel | Leave a comment