Slaying the Development branch – Evolving from git-flow (Part 1)

Continuous Integration and Continuous Delivery (CI/CD) is essential to any modern day, mission critical software development life cycle.

The economic logic is simple. If you’re paying a developer a lot of money to fix bugs or add features to your software, it doesn’t make sense to have those bug fixes and features sitting in a build pipeline for 2 months waiting to be deployed. Its the equivalent of spending money on stock for your grocery store and leaving it on a shelf in your loading bay instead of putting it in the shop window.

But taking legacy software development life cycles and refactoring them so that they can use CICD is a significant challenge. It is much harder to re-factor embedded, relatively stable processes than to design new ones from the ground up.

This was a challenge I was faced with in my most recent employment. This article, and its sequel, describe some of the challenges I encountered and how they were resolved, focusing specifically on how we evolved our source control management strategy from one based on git-flow to one that permitted merging code changes directly from Feature branches to Production.

I’ll begin by describing the environment as I found it.

This was a Multi Tenant Software as a Service (SAAS) provided over the Internet on a business to business basis. The SAAS comprised 16 individual services with a variety of MySQL and PostgreSQL as data stores. The services were built with Java (for processing and ETL operations) and Rails (for Web UI and API operations).

The business profile required parallel development streams, so source control was based on the git-flow model. Each project had a development branch, from which feature branches were taken. Feature branches were merged into concurrent release branches. Builds were created from release branches and deployed in the infrastructure tiers (Dev, QA, UAT, Staging, Prod). There was no historical deployment branch and no tagging. Each release cycle lasted approximately 6 weeks. A loose Agile framework applied, in that stories were part of releases, but Agile processes were not strictly followed.

Infrastructure used in the software development life cycle was shared. There were monolithic central Dev, QA, UAT environments etc. Local developer environments were not homogenous. Java developers couldn’t run local Rails apps and vice versa. All code was tested manually in centralised, shared environments.

The situation described above would be reasonably typical in software development environments which have evolved without a DevOps culture and dedicated Operations resources (ie where development teams build the environments and processes).

While the development/deployment process in this environment was working, it was sub-optimal, resulting in delays, cost overruns and issues with product quality.

A plan was developed to incrementally migrate from the current process to a CI/CD-based process. This involved changes to various functions, but perhaps the most important change was to the source control management strategy, which is what I want to deal with in detail in this article.

A typical development cycle worked as follows.

In every git project, the development branch was the main branch. That is to say, the state of all other branches was relative to the development branch (ahead or behind in terms of commits).

For a scheduled release, a release branch was created from the development branch. For for the purposes of illustration, lets call this release1. Stories scheduled for release1 were developed in feature branches taken from development, which were then merged into release1. These features also had to be merged to development. When all features were merged to release1, release1 was built and deployed to QA.

At the same time, work would start on release2, but a release2 branch would not be created, nor would release2 features be merged to development, as development was still being used as a source for release1 features. Only when development for release1 was frozen could release2 features be merged to development, and only when release1 was built for Production was a release2 branch created.

This system had been inherited from a simpler time when the company was younger and the number of applications comprising the platform was much smaller. Its limitations were obvious to all concerned, but the company didn’t not have a dedicated “DevOps” function until later in its evolution, so no serious attempt had been made to re-shape it.

From talking to developers, it became clear that the primary source of frustration with the system was the requirement to have to merge features to multiple branches. This was particularly painful when a story was pulled from a release, where the commit was reversed in the release branch but not the development branch. It was not infrequent for features to appear in one release when they were scheduled for another.

After talking through the challenge, we decided on a number of requirements:

1. Features would only be merged to one other branch

2. We could have concurrent release branches at any time

3. We would have a single historical “Production” branch, called “deploy”, which was tagged at each release

4. At the end of the process, we would only be one migration away from true CI/CD (merging features directing to deploy)

5. We would no longer have a development branch

From the outset, we knew the requirement that would present the biggest challenge was to be able to maintain concurrent release branches, because when multiple branches are stemmed from the same source branch, you always run the risk of creating merge conflicts when you try to merge those branches back to the source.

At this juncture, its probably wise to recap on what a merge conflict is, as this is necessary to understand how we approached the challenge in the way that we did.

A merge conflict occurs between 2 branches when those branches have a shared history, but an update is made to the same line in the same file after those branches have diverged from their common history. If a conflict exists, only one of the branches can be merged back to the common source.

If you think of a situation in which 2 development teams are working on 2 branches of the same project taken from the same historical branch, and those 2 branches ultimately have to be merged back to that historical branch, you can see how this could present a problem.

When you then extrapolate that problem out over 16 individual development projects, you see how you’re going to need a very clearly defined strategy for dealing with merge conflicts.

Our first step was to define at which points in the development cycle interaction with the source control management system would be required. This was straightforward enough:

1. When a new release branch was created

2. When a new patch branch was created

3. When a release was deployed to Production

We understood that at each of these points, source control would have to be updated to ensure that all release branches were in sync, and that whatever method we used to ensure they were in sync would have to be automated. In this instance, “in sync” means that every release branch should have any commits that are necessary for that release. For instance, if release1 were being deployed to Production, it was important that release2 should have all release1 commits after the point of deployment. Similarly, if we were creating release3, release3 should have all commits from release2 etc etc.

However, we knew that managing multiple branches in multiple projects in this way was bound to produce merge conflicts, but at the same time, we didn’t want a situation in which a company-wide source control management operation was held up by a single merge conflict in a single project.

In light of this, we decided to do something a little bit controversial.

If our aim was to keep branches in sync and up to date, we decided that branching operations should focus on this goal, and that we would use a separate mechanism to expose and resolve merge conflicts. Crucially, this part of the process would occur prior to global branching updates, so that all branches arrived at the point of synchronisation in good order.

So, to return to the 3 points where we interacted with source control, we decided on the following:

1. When a new release branch was created

This branch would be created from the latest tag on the historical production branch ( “deploy”). All commits from all other extant release branches would be merged to this branch, resulting in a new release branch that already contained all forward development. When a merge conflict existed between the new branch and an extant release branch, the change from the extant branch would be automatically accepted (–merge-strategy=theirs)

2. When a new patch branch was created

This branch would be created from the latest tag on the deploy branch . No commits from any other extant release branches would be merged to this branch, because we did not want forward development going into a patch. Because no extant release branches were being merged to the patch, there was no need to deal with merge conflicts at this point.

3. When a release was deployed to Production

At this point, the release branch would be merged to the deploy branch, and the deploy branch would then be merged to any extant release branches. This would ensure that everything that had been deployed to Production was included in forward development. When a merge conflict existed between the deploy branch and an extant branch, the change from the extant branch would be automatically accepted (–merge-strategy=ours). The release branch that had been merged would be deleted, and the the deploy branch tagged with the release version number.

We decided to refer to the automatic acceptance of changes during merge operations as “merge conflict suppression”. In the next part of the article, I’ll explain how we decided to deal with merge conflicts in a stable and predictable way.