This is the fourth and last entry in our article series about Caravan, Bitrock’s Cloud-Native Platform based on the HashiCorp stack. Read the first, second and third part on our blog.

The communication layer between application components running on top of Caravan leverages HashiCorp Consul to expose advanced functionalities. Service discovery, health checks, and service mesh are the key features that Consul enables in Caravan.

Service Discovery & Health Checks

While Consul makes it easy to lodge services in its registry, it offers a painless discovery process thanks to the different ways of inspection, such as API, CLO or DNS SRV queries. 

The service registry would not be complete without the health checking capabilities. It is possible to set up different kinds of health checks, to inspect whether a service is healthy and thus can be shown as available in the registry. When a health check fails, the registry no longer returns the failed instance in the client queries. In this way the consumer services stop making requests to the faulty instance.

Consul Connect with Envoy

Consul Connect provides Authorization and Encryption of the communication between services using mutual TLS. Applications are not aware of Consul Connect thanks to sidecar proxies deployed next to them to compose a Service Mesh. These proxies "see" all service-to-service traffic and can collect data about it. 

Consul Connect uses Envoy proxies and can be configured to collect layer 7 metrics and export them to tools such as Prometheus. Connect uses the registered service identity (rather than IP addresses) to enforce access control with intentions. Intentions declare the source and the destination flow where the connection is allowed - by default all connections are denied following the Zero Trust principles.

Within the Service Mesh, incoming and outgoing communication traffic is handled with a dedicated component called Gateway. The Gateway is secure by default, it encrypts all the traffic and requires explicit intentions to allow the requests to pass through. 

Service Mesh in Nomad

Nomad thoroughly integrates with Consul, allowing the specification of Consul configurations inside the Nomad job description. This way operators can define in a single place all the configurations needed to run a Nomad task and to register it in Consul, making it available to other components running in the platform.  In detail, Nomad agent automatically registers the service in Consul, sets up its health check, requests dynamic short-lived TLS certificates for a safe in-mesh communication enabled by the Envoy sidecar proxy, whose lifecycle is managed directly by Nomad without any manual intervention required.

Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock

This is the third entry in our article series about Caravan, Bitrock’s Cloud-Native Platform based on the HashiCorp stack. Check the first and second part.

Caravan heavily relies on the features offered by HashiCorp Vault. Vault is at the foundation of the high dynamicity and automation of Caravan. We may even say that Caravan would have not been the same without Vault, given its deep integration with all the components in use.

In this article, we show some of the Vault features that Caravan relies on.

PKI Secrets Engine

The PKI secrets engine generates dynamic X.509 certificates. It is possible to upload an existing certification authority or let Vault generate a new one, and in this way Vault will fully manage its lifecycle. This engine replaces the manual process of generating private keys and CSRs, submitting them to the CA, and waiting for the verification and signing process to complete. By using short TTLs it is even less likely that one needs to revoke a certificate, thus CRLs are short and the entire system easily scales to large workloads.

In Caravan we use Vault’s PKI to sign both Consul Connect mTLS certificates and server-side (eg. Consul and Nomad) certificates for TLS communications. 

Consul & Nomad Dynamic Secrets

Dynamic Secrets are a key feature of Vault. Their peculiarity is the fact that the secrets do not exist until they are read, so there is no risk of someone stealing them or another client using the same secrets. Vault has built-in revocation mechanisms: this way dynamic secrets are periodically revoked and regenerated, minimizing the risk exposure.

Vault integrates dynamic secrets with different components:

• Cloud providers (e.g.  AWS, Azure, GCP, …)
• Databases (e.g. PostgreSQL, Elasticsearch, MongoDB, …)
• Consul
• Nomad
• and many more…

In Caravan, we use the dynamic secrets engine for the generation of access tokens for both Consul and Nomad agents. First of all, we define in Vault the needed Consul and Nomad roles with the needed permissions, and then we map them to Vault roles. This way, we allow authenticated Vault entities to request Consul and Nomad tokens with the permissions defined in the associated role. For example, we set up Nomad Server role and Nomad Client role, with different authorization scopes.

Cloud Auth Backends

Distributing access credentials to Vault clients might be a difficult and sensitive task, especially in dynamic environments with ephemeral instances. Luckily for us, Vault addressed this operation and simplified it a lot in the cloud scenario. Vault implements different auth methods that rely on the cloud provider for the authentication of Vault entities.

For example, when running Vault with AWS instances, it is possible to authenticate the entities according to their associated AWS IAM role. Vault leverages AWS APIs to validate the identity of the clients, using the cloud offered primitives. This way, a Vault client running in an AWS instance does not need to know any Vault-related access credentials to access Vault, instead, AWS directly validates the identity of the client. The same logic applies also to other cloud providers such as Azure, GCP, and many more.

In Caravan, we rely on cloud auth backends to authenticate both the server-side and client-side components of the platform. This way, we no longer need to distribute credentials to the spinned instances, which would be a difficult and tedious task. 

Vault Agent

Vault Agent is a client daemon that provides useful functionality to clients who need to integrate and communicate with Vault without changing the client application code. Vault Agent allows for easy authentication to Vault in a wide variety of environments. Vault Agent allows client-side caching of responses containing newly created tokens and responses containing leased secrets generated off of these newly created tokens. Vault Agent allows rendering of user-supplied templates by Vault Agent, using the token generated by the Auto-Auth.

In particular, Caravan relies on Vault Agent templates to render configuration files for a variety of components. For example, the configuration file of Nomad agents is a template rendered by Vault Agent, since it contains dynamic secrets like the Consul token and the TLS certificates used for communication with the server components.

Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock

This is the second entry in our article series about Caravan, Bitrock’s Cloud-Native Platform based on the HashiCorp stack. Click here for the first part.

What is GitOps

GitOps is "a paradigm or a set of practices that empowers developers to perform tasks that typically fall under the purview of IT operations. GitOps requires us to describe and observe systems with declarative specifications that eventually form the basis of continuous everything" (source: Cloudbees).

GitOps upholds the principle that Git is the only source of truth. GitOps requires the system’s desired state to be stored in version control such that anyone can view the entire audit trail of changes. All changes to the desired state are fully traceable commits, associated with committer information, commit IDs, and time stamps.

Together with Terraform, GitOps allows the creation of Immutable Infrastructure as Code. When we need to add or perform an update, we have to modify our code and create a Merge/Pull Request to let our colleagues review our changes. After validating our changes we merge to our main branch and let our CI/CD pipelines apply the changes to our infrastructure environments.

Another approach in GitOps avoids triggering a CI/CD pipeline after a new change is merged. Instead, the system automatically pulls the new changes from the source code, and executes the needed actions to align the current state of the system to the new desired state declared in the source code.

How GitOps helped us build Caravan

GitOps provides us with the ability and framework to automate Caravan provisioning. In practice, GitOps is achieved by combining IAC, Git repositories, MRs/PRs, and CI/CD pipelines.

First of all we define our infra resources as code. Each layer of the Caravan stack is built following GitOps principles, and the first one is of course the Infrastructure layer that allows declaring the required building block for the major cloud provider. Networking, Compute resources and Security rules are all tracked in the Git repository.

Then, the following layer is the Platform one where we bring online the needed components with the required configuration. Finally, we declare the Application Support components deployed on top of the Platform.

Currently, the applications are deployed using a simpler approach leveraging standard Terraform files that we called “Carts”. Nomad itself can pull configuration files from git repository but lacks a solution like ArgoCD for automatically pulling all the nomad job descriptors from git.

Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock