Almost exactly to a year ago the merger between Cloudera and Hortonworks went through.
I've had a fairly inside field view of the process and the rollercoaster has been fun or at least never boring. Along the way, I kept having these four questions.
Four questions which are like an organizational rorschach test. But they all look like butterflies to me.
Do you want to release on schedule or by bug count?: There are fundamentally good reasons to ship software only when its bug count drops below a certain margin, but that usually circles around a single development line and not an entire ecosystem distribution. The customers aren't all the same and they don't care about problems in the same way across the components. Shipping a release with bugs is so much worse when you are planning a big bang release with another six months before the next attempt at fixing the problems. However, there are two assumptions hidden in this - new adopters will wait for a release instead of exploring alternatives during the delay and that the internal bug count is a good proxy for the issues that customers will encounter.
There are immediate organizational downsides to this as well. Any team that meets its bug counts for the expected due date through personal heroics of team members will feel disappointed that either they worked too hard or that they took up tech debt for workarounds they didn't need to. And the team that is currently holding up the release will get pressure from all sides and possibly a bad review in the future.
Of course, there's still no good answer because you can't just ship whatever you have because it is the 2nd Tuesday of the month. There's quite a lot of balancing between hitting the dates and closing all the release blockers. In general, most of the quality of code discussions circle around this particular trade-off point.
If there's an argument, is it better to build or discuss?: Decision making is always full of conflict. Technical discussions tend to be easier to tie-break since measurements are possible without involving external entities, unlike say advertising or marketing ones. That said, often specifications which are left on the table without implementations tend to grow outwards either to add more scope to the project or to tackle specific conditions which are imagined during a meeting. A design discussion is not the right place to shift the scope of a project or to add customer scenarios into the mix, but several arguments are discussed out to expand the implementation routinely. These are easily recognizable when applied to cross-cutting features like authentication or authorization which every teams to gets to provide input on.
The main reason why these arguments end up in discussions rather than arriving at an implementation which can be criticized more constructively is because the discussion happens between non-implementors, either being architects or senior engineers. The people who would be responsible for putting together a prototype are usually never in the meeting and even if they were, they are happier to evade being targeted by the engineers in charge of oversight.
Discussions are useful to clarify disagreement. And in this note Chesterton's fence is very much applicable. The implementation is often surprise-heavy and ends up having to bypass several decisions made in dicussions. Empowering the implementor to communicate disagreement with the designer is the most important communication pathway I've observed.
Do you organize teams by skills or involvement?: Before I get into it, let's talk about specialists versus generalists. For a total team conflict reduction, having specialists is better than generalists, since each person has clear responsibility for their own area of expertise with no involvement with others in the team - either to approve or disagree. Naturally, this results in teams getting fragmented into niche skill specialists and leaves the organization to manage staffing by either over-staffing specialists or under-staffing the team when someone is on vacation or quits. From a skip-level up the ladder, that's where the utility of folks like me come in, since I don't mind being thrown into a problem which requires reskilling (at the expense of some conflict with established patterns with questions like "why do we always do it this way?").
Assuming you have a team of specialists and a single manager, then the immediate problem comes up when any of the specialists has a "career" conversation with you. The next level up is management, at which point the specialist will be tackling a team where they are familiar with a fraction of the skills. Having organizations where specialists occupy their own organization structure and float between projects is a way to bypass that issue, since the size of the project is not related to the location of the specialist in the org chart. But that reduces the involvement of the specialist in the team, where the success of the project is only indirectly tied to the future prospects. There is some middle ground here, but that needs to be found for each growth stage of the organization.
Is it better to have big plans or small plans?: Project planning isn't quite war, but it still holds that the map isn't the same as territory. The size and scale of the ideal plan varies as you move between approval of said plan and the execution of it. Big plans tend to motivate leadership, while they tend to overwhelm the foot soldier who can't quite see the map. However, the difference in software engineering is that a big plan can come from the other direction - engineers wanting to do complete rewrites to improve productivity and implement features faster. Because the rates of distruption over a month remains roughly the same, a big plan therefore is likely to fail, since priorities can change over two quarters more than it does over a single one.
It is easy for organizational panic when objectives can be further away than your plans. The comment about "I don't see how your plan is going to get us there" isn't the end of discussion, it is merely an opportunity to admit that from the hole you are currently in, a better plan can come only after you climb out of it. Again, there isn't a good size for a plan - there are plans within plans and all that. And there are ways to meaningfully go ahead with feature flags and fall back mechanisms where the big plan can roll out in stages, where an unexpected event is merely a pause in the process.
And then: These questions are important to me, not because they have right or wrong answers, but because your answers tell me what your experiences and perspective on software development are. And then perhaps, more questions to ask me.
--“I would rather have questions that can't be answered than answers that can't be questioned.”
― Richard Feynman
posted at: 15:11 | path: /observations | permalink |