Today's the first day of year 5 for me in Hortonworks.
I've now spent more than a decade getting paid to do open source work. But until Hortonworks, all work I did was tangential to the business arm of the company. All my customers were internal, which meant mostly I got privileged access to the problems. The bug resolution for complex issues never had to deal with walls of customer solutions obscuring my insights into the system. There was never a day when I couldn't tell someone from Yahoo mail that I was working on another issue with Frontpage.
During my Zynga period, I learnt how to sell your ideas to a product focus team looking for solutions - to explain better, to justify my arguments with numbers, to convince and to persuade. To keep my sanity, I also learnt to pick my battles, drop issues and walk away when the friction gets personal.
I never had to sell to a customer direct - even at my best, on the books of the company I was a cost, an engineering expense. My work at Zynga or Yahoo, was helping the company bring down its bottom line by a fraction without ever touching the top line, except by accident (yay, faster load times).
This is generally an area of work with no real conflicts of interest built into it - use of APC for instance was mandated and in most cases, my code was the only available solution to the problem. Giving up on APC meant switching to HipHop, which also I worked on (never got the CLA signed by Zynga, which was sad though). The pattern followed me through out the period even through ZCloud build-out, through ZPerfmon, ZProf and pretty much everything I touched.
Hortonworks has been a completely different experience. I'm still a developer, but my contribution to the company has turned into a top-line enlarging investment in most scenarios. The work I was doing with Stinger was actively bringing in more customers and indirectly, more revenue in. This brings in some interesting conflicts for a developer, when talking to a customer about your features, because that particular transaction starts off as a zero sum exchange of money with the implicit declaration that the "solution" is a non-zero sum win-win.
This shift in the nature of my job has been interesting. The role now requires a completely different social skill set which I didn't need in any job before this. The goal is nearly the same, but conflicts fall onto my lap instead of being able to just say "Zynga first" to end a discussion.
Honestly, it's been a bit of a culture shock.
I feel like I'm astride an elephant, with my toes pressing suggestions of direction. The benchmark numbers are just numbers, unless someone else decides that those need some action. I cannot actually move the beast, but the beast does move, never to stand still.
As always, I've stayed on top of that beast here for the last four years to learn and cope with that process. The learnings are rather harsh, that while in a single company you get to deal with your users first, the actual users in this mode don't run the systems, those who run the system don't make purchasing decisions and those who write the checks don't really deal with either of those folks directly. And in the end, the check they write is eventually only fractionally related to my paycheck anyway.
In that world, it feels ultra depressing to sit in on a call where the dreaded phrase is said - "what the hell do we pay you for?" . There's the sub-text in your head going - uh, you don't ... I'm here because I am the only one who can help with this and this is not even my job, neither is it my turn on the on-call roster, so get off my back. But you learn not to lash out in knee-jerk fashion and close down the conversation, but continue debugging which is upto your discipline, a thick skin and the knowledge that your self-worth isn't tied up in that guy's opinion. I still don't know how to communicate that Insulting my commitment to your problem isn't helping - making it unpleasant might feel better, but you're burning good will here. Looking back, some of what I did never made any sense in the big picture, to feel the pressure from the sales cycle to deliver something over the christmas weekend and then get back on it through the new year weekend too. However hard those weeks turn out to be, the rewards for such heroics diffuses away as the company gets larger and as the customer base expands outwards - no deal ends up being the end-all of your work.
I still love solving those tough problems and solving them are their own reward, but the process friction is draining for your attention spans and your ability to take criticism for no fault of your own wears thin (I'm rarely the person who wrote the slow bit of code I'm fixing and I'm unlikely to be the one in the next case either, *sigh*). Collective responsibility is hard with open source, when this issue was because of someone back in 2009 who thought this was good enough, way before the YARN and Hive teams sat under the same roof. It's nobody's fault, but remains your problem.
There are some upsides to being thrown about into the rough and tumble of that process, you develop a lot of breadth in your skill sets out of sheer necessity - whether it's about something esoteric like Kerberos or the specifics of file listings in Azure or S3. There are technical savings you build up over the years, which work exactly the opposite of how technical debt works - readymade solutions which you refine through one customer aftet the other, to feed back into the product as much as you can. This is what a front-line engineer does in a small company, to carry a huge memory bank of all the problems cross-referenced to their solutions. A bigger company divides away expertise until no one person has the entire picture and start operating in managable fractions of expertise.
I came into Hortonworks, expecting to fix MapReduce and HDFS, possibly to dig deeper into HBase latencies - it is odd to say that right because instead of polishing old code, I got to work on brand new ideas with Stinger on Hive, ORC and Tez. But unlike my work at Zynga, I've never had to bury any of work (and to have the company salt the earth behind it) - my patents from Zynga are clearly dead-ends which I cannot pursue somewhere else, original ideas which I had and lost posession of. Putting up every little thing I build onto a github repo has been a liberating experience, being able to talk about my work in great detail to everyone including business competitors gives me a sense of autonomy which I have not experienced since college.
There have been so many mishaps along the way too. It was a horrible idea to work as a contractor in India when Hortonworks failed to hire me in time for the H1B cap - every bit of my immigration that followed has been a pain because of the fact that I was an Indian contractor moving to the US. Unfortunately it's too late now that I'm already here and is probably beyond recourse, I would probably have to quit or move out of America to reset that, in the near future. And that's sort of been the only regret of mine in joining Hortonworks the way I did.
But this is the second longest I've worked in my career and this journey has been very different from the Gumball years of the Yahoo. With Hortonworks, I joined a small company in Maude & Matilda four years ago and I now work for a company that is so big it has offices around the planet. When I interviewed at Hortonworks in February of 2012, that was way before the this company was - back before there was a packaged product or an installer, before the "do or die" phase of working for an underdog company.
And an underdog might be down, but always gets to punch up.
--Boredom is the feeling that everything is a waste of time; serenity, that nothing is.
-- Thomas Szasz