Announcing Brigade 1.0: Build a new kind of distributed application

Brendan Burns, one of the creators of Kubernetes (and the head of Azure Container Computing at Microsoft) often quips that, “Distributed computing is the new CS 101.” Instead of being considered an advanced topic in computer science, distributed computing is now a core requirement.

It is with this in mind that we built Brigade, which we believe is a new class of tools for developing a special type of distributed applications.

For a few decades, the client-server model dominated our industry. But the microservices pattern, the rise of containers, and the serverless movement have changed things. We, as software developers, have begun breaking down our once-monolithic systems into discrete units (containers, functions, and microservices) woven together to build macro-scale applications. Brigade is part of this movement, but it differs in a crucial way: it makes this process of stitching together serverless parts a programmable process.

Brigade is now 1.0

Now we’re thrilled to announce that Brigade has reached 1.0, its first major completed version. In addition, it has just become a Cloud Native Computing Foundation (CNCF) sandbox project alongside many other great tools in this space.

Brigade scripting: Automation as a container-based application

On a recent Brigade developer call, one of the major contributors to Brigade made an interesting observation– it is hard to explain Brigade by comparison to other projects. This is because Brigade is the first in what we hope is an emerging class of tools. It provides an event-based JavaScript programming environment for constructing pipelines of serverless processes.

Perhaps our industry’s most common example of stitching together pipelines is in the domain of CI/CD. We are familiar with the process of automating the testing and delivery of software by running a set of discrete steps:

1. Compile the software.
2. Run the tests to validate the software.
3. Tag the software.
4. Upload it to its destination.
5. Alert the team that the process has completed.

Brigade is focused on the class of applications that has this property of chain-like sequences. But it goes further– in Brigade, each step is handled within a separate container (of which the contents are solely in the developer’s control). Brigade provides a way to explicitly script the workflow. Here’s a simplified representation of the first three steps of the pipeline above, intended just to give you the sense of what a Brigade pipeline looks like:

events.on("push", async () => {
  var compileStep = new Job("compile", "example/compiler:latest")
  var testStep = new Job("test", "example/tester:latest")
  var tagStep = new Job("tag:, "example/releasetagger:latest")
  // We could continue on creating the remaining steps

  await compileStep.run()
  await testStep.run()
  await tagStep.run()
  // We could continue running the remaining steps
});

The example above says, “When a push event happens (e.g., code is pushed to GitHub), create three jobs: compileStep, testStep, and tagStep. Then run the jobs in order, waiting for each to finish before starting the next.

That last part – the part about waiting – raises an interesting feature of Brigade. Brigade is all about controlling workflows. Instead of waiting for each step before proceeding, you may instead choose to run some steps in parallel. Brigade provides tools to let you do more sophisticated grouping, as well. Group several steps, run that entire group in parallel, wait for all of them to finish, and then proceed on to the next job or group. Since you have all the power of JavaScript at your disposal, you can react to the results of a job, and then decide what the pipeline should do next (“if this returns an even number, go run Job A; but if it returns an odd number, run Job B”). You can use loops, repeating a job until it achieves a desired goal. And, of course, you have full control over error handling. Catch an exception and handle it in the script rather than having to give up on the entire pipeline run.

The example above is simplified down to its bare structural properties and does not show how Brigade manages sending data between steps or sending specific instructions to each Job, but those things are among Brigade’s broad feature set.

What can Brigade do for you?

When we first imagined Brigade, we had a few scenarios in mind, including the CI/CD story above, as well as some of the big data and ETL work we had worked on in the past. But we have been pleasantly surprised to see some of the things people have done with Brigade. Here’s a small sampling:

• Run a nightly quality assessment of all of the different code repositories in an organization.
• Containerize legacy data processing systems, and then control the flow of data in and out of this legacy system using Brigade.
• Use CloudEvents to trigger Brigade pipelines to perform time consuming data analysis.
• Build pipelines that coalesce large bodies of data on a nightly or monthly cadence, and then analyze the data and produce reports.
• Integrate with GitHub (BitBucket or GitLab, too), and send Slack notifications to the team responsible for watching the repository. (In fact, a new project built upon Brigade implements a full CI/CD UI!).
• Create internal “review only” instances of new content by watching for changes on the content repository and provisioning an internal review instance for content managers to preview.

In addition to building the core pipeline system and scripting environment, we built Brigade to be extensible. There are many new tools and gateways you can take advantage of and we have tried to make it easy for you to write your own integrations.

Brigade is not a general-purpose tool. For example, it’s not a website building framework. But when it comes to assembling and executing sequences of tasks, Brigade provides a powerful scripting environment that still feels approachable. Harness the power of distributed computing… with just a few lines of JavaScript.

Brigade: Built on Kubernetes

Finally, it’s worth mentioning the technology upon which Brigade is built. Brigade runs in a Kubernetes cluster, which provides all of the tooling necessary for managing the execution of myriad containers. Kubernetes gives us the primitives we need not just for running containers, but also for shuttling data between containers, accessing logs and debugging information, and providing caching of frequently shared data. At its core, Brigade comes with a few pieces:

• One or more gateways that listen for events, and trigger new Brigade runs.
• A controller that responds to a gateway by creating a new Brigade run.
• A worker that runs the JavaScript. Each Brigade run gets its own worker.

Because the components are kept minimal, Brigade can run on even small Kubernetes clusters without consuming too many system resources.

If you are interested in building distributed pipeline-based applications, we’d love to have you take Brigade for a test drive. It’s entirely open source software, and runs on any Kubernetes cluster from minikube to Azure Kubernetes Service (AKS). Head to https://brigade.sh to get started. And if you find yourself interested, join us in the #brigade channel on the Kubernetes slack, or drop in to our weekly public Brigade developer’s call.

Questions or feedback? Let us know in the comments below.