Push to prod, do it often

Software development might mean different things to different people, but at the end of the day the whole point is to satisfy the needs of its users, whichever they might be.

Think of it in that light, and some gaps will appear in the way those needs are conveyed:

• What the user actually wants != what he thinks he wants.
• What the user thinks he wants != what the business team thinks he wants.
• What the business team requests != what ends up as a Requirement or User Story for the dev team.

This is unfortunate, but apart from trying to get in touch directly with the user, there’s little we can do as devs to close that gap.

What’s maybe more relevant to us is the gap that runs on the other side of the equation, between code being written and the user actually using it (the way the needs are met):

1. Write the code
2. Make a PR
3. Wait for the reviewer and discuss the code
4. Deploy to a staging env
5. Wait for QA to validate it
6. Mark as ready to release
7. Wait for the release cycle/window
8. … Is the user still there?
9. Fuck I pushed a bug to prod -.-U
10. Hotfix? Rollback and goto to step one?

All this with the accompanying mess of branches, merges and possible conflicts with other co-workers or teams.

In an ideal world, the user would tell us what he needs directly and clearly, peek over our shoulders while we code, understand what we are writing, and let us know if we are on the right track.

Sadly, this world is not ideal. But there is a lot we can do to shorten the time between writing code and receiving the users feedback on it.

Trunk Based Development is one approach we might use to achieve this.

What even is TBD?

Trunk Based Development is, in a nutshell, the practice of writing software avoiding branches as much as possible, streamlining the development in one main “Trunk” branch, also used for deploys, QA and the likes.

The fundamental importance of this approach is in recognizing that the main “Trunk” branch is the only source of truth.

There’s no ‘my version vs your version’ here, we all have the same version. No ‘what commit is prod right now?’: the answer is ideally always ‘the last one’.

While using branches is not necessarily blasphemy (it might be more practical to use them), no long-lived branches should exist.

Of course for this to be viable, commits should be small and happen as frequently as possible. Code should be thoroughly tested and always releasable.

Simply put, you should aspire to constantly commit correct, tested code straight to production.

What’s wrong with branches?

Nothing per se, but we often forget the trade-offs they bring. Here’s a refresher.

Merge Hell

In big projects with multiple dev teams working on shared code, Merge Hell is a very real issue.

Nobody wants to spend half a work-day resolving merge conflicts, much less pay for someone’s salary to do so.

The feeling one gets after resolving merge conflicts for 2 hours, only to find out more conflicting code was pushed in the meantime is… not nice.

It’s just not a productive use of time.

Partial Truths

If you are working on a branch created more than a day ago, you know how your code behaves with yesterday’s project. Your ‘Truth’ got stuck in time.

Today’s version of the project might not behave the same, and you might need to re-think what you are doing. Wouldn’t you want to know if that’s the case as soon as possible?

The other side of the coin might be even worse: While your code is not in the main branch, you are hiding information from the rest of the team.

Nobody knows what your code looks like, how it behaves or how to work with it. You are hiding your ‘Truth’ from the rest of the team(s).

Speed (or lack thereof)

Do we really need long-lived branches at all? When a critical hotfix is required, we clearly have no issue pushing directly to the main branch.

Even with a protected main branch, we create a short-lived branch on the fly before deploying it directly to production, no fluff involved.

This approach allows us to quickly update production software, delivering immediate value by squashing a bug or addressing a critical need.

So why not aim for the same speed when rolling out new features or UI updates? Why should we go fast only in emergencies?

TBD? CI? CD?

Some might argue that these problems are avoided by practicing CI/CD, no need for this TBD business. The thing is, you are probably not really doing CI/CD if you aren’t also doing TBD.

On the other hand, if you’re doing TBD, you’re either practicing CI/CD, incredibly smart, or incredibly dumb.

CI/CD is a somewhat ambiguous term: It depends on what one considers “continuous”.

Teams coming from a monthly release cycle might consider weekly integrations to be “continuous”, while some might argue that daily integrations are the bare minimum to be considered “continuous”.

CI/CD is also quite often mistaken with “having a pipeline”, which is indeed a necessary and important part of the deal, but not the whole thing.

One can hide behind these ambiguities when talking about CI/CD, but there’s no hiding from TBD: You either push to prod, or you don’t.

If we want to deliver software continuously (CD), we need to integrate our work continuously (CI). At some point one has to evaluate if the rituals involved in a branch-based workflow really allow any of this, or if we are better off getting rid of the paperwork and focusing on what matters.

The fundamental assumption of CI is that there’s only one interesting version, the current one.

-Wiki

If the only interesting version is the current one, why waste effort, time and resources in other versions?

PRs kinda suck

This does not mean we need to ditch PRs completely: As mentioned before, branches are not the devil, PRs have their place.

TBD is a general way of approaching development, not a strict dogma.

We’ll see that these principles can be followed even in a PR centric workflow.
So long as their use is reasoned and makes sense in context.

Still, it’s worth considering why we use and (supposedly) “need” PRs in the first place, especially given their downsides:

• Reviews are a pain: PR reviews are often treated like chores: rarely does one enjoy doing them and more often than not, they are done as an afterthought in some spare time or hastily before doing the ‘actual work’.

• Context change: They demand frequent context switching, especially if one is expected to prioritize them. This is not productive and should be avoided if possible.

• Slow pace: Reviewing PRs, if done with care, is a time-consuming process (more so with long-lived branches), and even with the best intentions and effort, it still significantly slows down the pace at which we deliver value to users. After all, perfectly good code might be sitting in a PR right now just waiting for someone to review it while it could be adding value to the project.

• Better alternatives: In most situations, live code reviews and pair or mob programming sessions are a much faster way of ensuring code quality (or asking for input/help), with much less chance of overlooking mistakes, introducing bugs or creating unnecessary friction between co-workers.

Gatekeeping

More often than not, PRs are used as a form of gatekeeping. We assign a keeper for the ‘Security’ gate, one for the ‘Testing’ gate, another one for the ‘Efficiency’ gate and so on.

To be clear, this can make sense in some cases:

• Many juniors, few seniors: PRs are a nice way of managing these team layouts ensuring quality standards are met and bugs avoided. Ideally the senior would pair-program his way out of this situation, but it is a useful temporary crutch.

• Open Source Software: In this context there are only a handful of maintainers with a complete picture of the codebase and sometimes hundreds of occasional contributors. It just makes sense for them to inspect the code before merging it and PRs are the best way to do so. The maintainers will be the ones… maintaining the code in the long run after all.

Gatekeeping might be done with good intentions and might be necessary in certain moments and contexts. Usually though, especially in a business setting, there’s a deeper underlying cause.

Trust

You might feel uncomfortable letting “anyone push to prod”.

It might be worth digging deeper here. What would you expect to happen?

Do you expect your teammates to knowingly introduce bugs? Are you worried that they aren’t ‘good enough’? ‘Professional enough’? ‘Smart enough’?

If these doubts sound silly: Congrats, you trust your teammates!

If instead they sound reasonable, you might want to ask yourself if you are comfortable working with people you don’t trust or respect, well before thinking about TBD.

A team can’t work effectively if their members don’t fully trust each other. Each member should feed like the rest of the team has their back, and that everyone is capable of doing at least as good a job as they do.

If this isn’t the case, no, TBD is not for you. I would argue the team itself is not for you either.