Serverless Camunda Terraform Recipe using Cloud Run and Cloud SQL

14 Mar 2020|Technology

Camunda is an excellent piece of infrastructure to host business logic. Business logic is dynamic, so it makes sense to express it as configuration and provide tooling for business users to visualize what is going on. Camunda ticks these boxes, but its deployment is a bit old fashioned, requiring dedicated machines and the headaches that implies. I thought, wouldn't it be cool if we could run Camunda as a serverless process?

Here is a recipe to get the Camunda business workflow engine up-and-running fast using Terraform and serverless hosting technologies. Full code on Github

Persistence with Cloud SQL

Camunda requires a relational database. I used Google's managed Cloud SQL. Hardly any ops required! I put Camunda in its own database namespace, so we can reuse the SQL service for other applications. I also provisioned a unique user, so we can track Camunda's access uniquely.

We’d like to help you make a fully informed decision about cookies. Accessing some of our embedded content that could be of interest and use to you requires you to enable cookies.The choice is always yours.

Customizing the Camunda Docker Image With Cloud Build

Camanda very helpfully supplies a basic docker image, but we need to customize it. We will typically need to do this for two reasons. First, we need to supply a custom database driver to auto handle Google Cloud's certificate rotations. Second, we need to load our Camanda runtime customization. For now, I just show you how to load the custom driver, but the approach extends to the second case that you will also probably need to do.

First, we mirror the public docker image into our projects private image repository. We don't want our assets on the general internet! Plus, it's much faster for Cloud Build to work with images hosted on Google Container Registry.

Next, in a Dockerfile, we start with the mirrored images, install wget, delete the old postgres driver and download Google's driver into the right place. There is also some bullshit with linux USER accounts to get apk working properly.

How did I know where to install the driver? I shelled into a container to explore the filesystem with the command docker run -it camunda/camunda-bpm-platform:7.12.0 /bin/sh

To build the container I use Cloud Build invoked at the command line through gcloud. We can express this with a null_resource.

That's a bit ugly, but for a real deployment I would build cloud automation in a CD/CI pipeline. So this step would not normally be in Terraform, so customize this step to suit your needs. The important thing is that we have offloaded the image building to a cloud hosted process, and that process will place a custom image in our private image repository.

Hosting Camunda Workers on Cloud Run

We use Cloud Run to host the Camunda service. This will spin up docker processes on demand, based on the customized docker image we built earlier. We need the service exposed on the public internet to access it, and, even though Camunda has authentication, I feel there is risk associated. So we provision a locked down service account, so the Camunda process only has access to Cloud SQL.

By default, Cloud Run does not allow public access to its end points, so we first create a public cloud run invoker policy and bind that to our Camunda service to expose it. Then we create a custom service account with just Cloud SQL client permissions.

Finally we run the service. We have to pass the database credentials in through env parameters. We have to customize the database URL and threadpool, to prevent too many instances of Camunda starting so we stay within the connection limits of our tiny Cloud SQL test instance. After running terraform apply, it should be up! Check the cloud console to find the domain.

Stop Writing Business Logic, Start Hosting Business Workflows

Camunda ships with an REST API. You can use these workers as a service in a larger deployment. I myself have big plans to make delivery of custom business automation simpler. Stay tuned! The full code is on Github