Caravan Series PT4

This is the fourth and last entry in our article series about Caravan, Bitrock’s Cloud-Native Platform based on the HashiCorp stack. Read the first, second and third part on our blog.

The communication layer between application components running on top of Caravan leverages HashiCorp Consul to expose advanced functionalities. Service discovery, health checks, and service mesh are the key features that Consul enables in Caravan.

Service Discovery & Health Checks

While Consul makes it easy to lodge services in its registry, it offers a painless discovery process thanks to the different ways of inspection, such as API, CLO or DNS SRV queries. 

The service registry would not be complete without the health checking capabilities. It is possible to set up different kinds of health checks, to inspect whether a service is healthy and thus can be shown as available in the registry. When a health check fails, the registry no longer returns the failed instance in the client queries. In this way the consumer services stop making requests to the faulty instance.

Caravan Logo

Consul Connect with Envoy

Consul Connect provides Authorization and Encryption of the communication between services using mutual TLS. Applications are not aware of Consul Connect thanks to sidecar proxies deployed next to them to compose a Service Mesh. These proxies "see" all service-to-service traffic and can collect data about it. 

Consul Connect uses Envoy proxies and can be configured to collect layer 7 metrics and export them to tools such as Prometheus. Connect uses the registered service identity (rather than IP addresses) to enforce access control with intentions. Intentions declare the source and the destination flow where the connection is allowed - by default all connections are denied following the Zero Trust principles.

Within the Service Mesh, incoming and outgoing communication traffic is handled with a dedicated component called Gateway. The Gateway is secure by default, it encrypts all the traffic and requires explicit intentions to allow the requests to pass through. 

Service Mesh in Nomad

Nomad thoroughly integrates with Consul, allowing the specification of Consul configurations inside the Nomad job description. This way operators can define in a single place all the configurations needed to run a Nomad task and to register it in Consul, making it available to other components running in the platform.  In detail, Nomad agent automatically registers the service in Consul, sets up its health check, requests dynamic short-lived TLS certificates for a safe in-mesh communication enabled by the Envoy sidecar proxy, whose lifecycle is managed directly by Nomad without any manual intervention required.


Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock

Read More
PNRR Bitrock

Non c’è alternativa”, recitava un vecchio slogan politico che portò alla creazione del governo più duraturo del Novecento. Oggi, il medesimo mantra si può applicare alle molteplici necessità di innovazione e digitalizzazione del tessuto produttivo italiano, che si (ri)affaccia sul mercato globale al termine, si spera, della crisi pandemica già affardellato da decennali cali di produttività e competitività. Per chi vuole prosperare nuovo scenario, il cambiamento tecnologico rappresenta un principio cogente.

In questo senso, il Piano Nazionale di Ripresa e Resilienza (PNRR) costituisce una opportunità rilevante. Elaborato in risposta alla grave crisi economica e sociale innescata dal Covid19, prevede l’allocazione di 191,5 miliardi di euro in una serie di interventi atti a rilanciare la fragile economia italiana e stimolare l’occupazione. Gli ambiti spaziano dallo sviluppo della mobilità sostenibile, alla transizione ecologica e all’inclusione di gruppi sociali ulteriormente marginalizzati dalla precarietà lavorativa.

Transizione digitale 4.0 per il sistema Italia

La prima missione del PNRR mette al centro “Digitalizzazione, Innovazione, Competitività, Cultura e Turismo”, valorizzando i concetti chiave che fungono da leitmotiv per l’intero Recovery Plan. Prevede lo stanziamento di 40,32 miliardi di euro per un programma di transizione digitale che interessa sia il settore pubblico sia quello privato. 

L’obiettivo è quello di sostenere lo sviluppo e la capacità competitiva di un sistema paese che, al momento, si posizione al 25mo posto (su 28) nel Digital Economy and Society Index (DESI). Come ricorda il PNRR (pag. 83), tale arretratezza fa il paio con il calo di produttività che ha caratterizzato l’economia italiana nell’ultimo ventennio, a fronte di una tendenza positiva nel resto del continente europeo. Questa contrazione è sovente legata alla ridotta innovazione digitale delle piccole e medie imprese, che rappresentano il 92% delle aziende e impiegano l’82% dei lavoratori in Italia (Il Sole 24 Ore).

La missione si articola in tre componenti:

  1. Digitalizzazione, Innovazione e Sicurezza nella PA (9,75 Mrd)
  2. Digitalizzazione, Innovazione e Competitività del Sistema Produttivo (23,89 Mrd)
  3. Turismo e Cultura (6,68 Mrd)

Vediamo nel dettaglio il secondo punto, cui è dedicato uno dei maggiori investimenti del PNRR.

Digitalizzazione, Innovazione e Competitività del Sistema Produttivo: come funziona

Il programma per il settore privato si prefigge, nelle parole del documento, di rafforzare “la politica di incentivazione fiscale già in corso (studiata per colmare il gap di “digital intensity” del nostro sistema produttivo verso il resto d’Europa – minori investimenti valutabili in due punti di Pil – specie nella manifattura e nelle PMI), che ha avuto effetti positivi sia sulla digitalizzazione delle imprese che sull’occupazione, soprattutto giovanile e nelle nuove professioni” (pag. 98).

Prevede una serie di investimenti e riforme che hanno l’obbiettivo di potenziare la digitalizzazione, innovazione tecnologica e internazionalizzazione del tessuto produttivo e imprenditoriale, con un occhio specifico alle PMI che maggiormente risentono del clima di volatilità contemporanea. 

All'interno del PNRR, il piano di investimento “Transizione 4.0” costituisce un’evoluzione del già noto programma Industria 4.0 del 2017, di cui viene allargato il novero delle aziende potenzialmente beneficiarie. Prevede tra le altre cose l’erogazione di un credito di imposta per società che decidono di investire in

  1. Beni capitali, materiali e immateriali
  2. Ricerca, sviluppo e innovazione
  3. Attività di formazione alla digitalizzazione e di sviluppo delle relative competenze

La prima voce riguarda l’investimento per strumenti “direttamente connessi alla trasformazione digitale dei processi produttivi” – i cosiddetti Beni 4.0 già indicati negli allegati A e B alla legge 232 del 2016 –, e “beni immateriali di natura diversa, ma strumentali all’attività dell’impresa (pag. 99)

Se il primo allegato dettaglia una serie di componenti hardware, tra cui macchinari, utensili e sistemi di monitoraggio, il secondo si concentra su soluzioni software ad alto tasso tecnologico che possono sostenere le aziende in un percorso di crescita scalabile e sostenibile.

Le applicazioni possibili

Integrati all’interno di una visione strategica, le soluzioni hardware e software menzionate nel PNRR possono trovare applicazione in una serie di ambiti, tra cui:

  • La transizione verso il paradigma Cloud Native, un approccio che sfrutta le tecnologie del Cloud Computing per progettare e implementare applicazioni sulla base dei principi di flessibilità, adattabilità, efficienza e resilienza. Grazie a strumenti metodologici e tecnologici come DevOps, container e microservizi, il Cloud Native permette di ridurre il time to market e sostenere l’evoluzione agile dell’intero ecosistema aziendale.
  • La valorizzazione del patrimonio informativo aziendale attraverso l’implementazione di sistemi di Data Analysis in tempo reale, IIoT (Industrial Internet of Things) e Data Streaming che, combinati con Machine Learning e Intelligenza Artificiale, possono essere sfruttati per la manutenzione predittiva, con un evidente ottimizzazione dei costi. Rientrano in questo ambito anche i Digital Twin, le copie virtuali di risorse o processi industriali che permettono di sperimentare in vitro nuove soluzioni e prevenire malfunzionamenti.
  • La cybersecurity, sempre più centrale in un contesto di crescente digitalizzazione di processi e servizi, e di crescente interdipendenza di attori nazionali e stranieri, pubblici e privati all’interno della catena del valore digitale.

Questi percorsi di maturazione digitale possono essere rilevanti sia per le grandi realtà, sia per le PMI che faticano maggiormente a tenere il passo con l’evoluzione tecnologica e la competizione internazionale. Lo sforzo è premiato: come riporta l’Osservatorio innovazione digitale PMI del Politecnico di Milano, le aziende medie e piccole digitalizzate riportano in media un incremento del 28% nell’utile netto, con il margine di profitto più alto del 18% (La Repubblica).

Perché quindi le aziende non digitalizzano? Il problema, spesso, è nella mancanza di personale qualificato. La carenza di staff qualificato affligge il 42% delle PMI italiane (La Repubblica), e la cifra sale al 70% se prendiamo in esame l’intero tessuto produttivo europeo (Commissione Europea). Un altro possibile fattore bloccante concerne la renitenza all’abbandono o evoluzione di sistemi legacy già consolidati all’interno dei processi aziendali.

Questi sono solo alcuni dei motivi per cui è fondamentale affiancarsi a un partner qualificato, che possa accompagnare l’azienda nella pianificazione degli investimenti tecnologici e digitali resi possibili dal PNRR (e non solo).

Bitrock ha competenze certificate ed esperienza internazionale per offrire soluzioni su misura che innovano l’ecosistema tecnologico e digitale, mantenendo gli investimenti legacy del cliente. Il know-how specializzato in ambito DevOps, Software Engineering, UX&Front-End e Data&Analytics è la chiave per affrontare il percorso di evoluzione digitale, con al centro i valori di semplificazione e automazione che generano valore duraturo.

Per conoscere nel dettaglio come possiamo supportare la tua azienda, contattaci subito!

Read More

This is the third entry in our article series about Caravan, Bitrock’s Cloud-Native Platform based on the HashiCorp stack. Check the first and second part.

Caravan heavily relies on the features offered by HashiCorp Vault. Vault is at the foundation of the high dynamicity and automation of Caravan. We may even say that Caravan would have not been the same without Vault, given its deep integration with all the components in use.

In this article, we show some of the Vault features that Caravan relies on.

PKI Secrets Engine

The PKI secrets engine generates dynamic X.509 certificates. It is possible to upload an existing certification authority or let Vault generate a new one, and in this way Vault will fully manage its lifecycle. This engine replaces the manual process of generating private keys and CSRs, submitting them to the CA, and waiting for the verification and signing process to complete. By using short TTLs it is even less likely that one needs to revoke a certificate, thus CRLs are short and the entire system easily scales to large workloads.

In Caravan we use Vault’s PKI to sign both Consul Connect mTLS certificates and server-side (eg. Consul and Nomad) certificates for TLS communications. 

Consul & Nomad Dynamic Secrets

Dynamic Secrets are a key feature of Vault. Their peculiarity is the fact that the secrets do not exist until they are read, so there is no risk of someone stealing them or another client using the same secrets. Vault has built-in revocation mechanisms: this way dynamic secrets are periodically revoked and regenerated, minimizing the risk exposure.

Vault integrates dynamic secrets with different components:

  • Cloud providers (e.g.  AWS, Azure, GCP, …)
  • Databases (e.g. PostgreSQL, Elasticsearch, MongoDB, …)
  • Consul
  • Nomad
  • and many more…

In Caravan, we use the dynamic secrets engine for the generation of access tokens for both Consul and Nomad agents. First of all, we define in Vault the needed Consul and Nomad roles with the needed permissions, and then we map them to Vault roles. This way, we allow authenticated Vault entities to request Consul and Nomad tokens with the permissions defined in the associated role. For example, we set up Nomad Server role and Nomad Client role, with different authorization scopes.

Caravan Logo

Cloud Auth Backends

Distributing access credentials to Vault clients might be a difficult and sensitive task, especially in dynamic environments with ephemeral instances. Luckily for us, Vault addressed this operation and simplified it a lot in the cloud scenario. Vault implements different auth methods that rely on the cloud provider for the authentication of Vault entities.

For example, when running Vault with AWS instances, it is possible to authenticate the entities according to their associated AWS IAM role. Vault leverages AWS APIs to validate the identity of the clients, using the cloud offered primitives. This way, a Vault client running in an AWS instance does not need to know any Vault-related access credentials to access Vault, instead, AWS directly validates the identity of the client. The same logic applies also to other cloud providers such as Azure, GCP, and many more.

In Caravan, we rely on cloud auth backends to authenticate both the server-side and client-side components of the platform. This way, we no longer need to distribute credentials to the spinned instances, which would be a difficult and tedious task. 

Vault Agent

Vault Agent is a client daemon that provides useful functionality to clients who need to integrate and communicate with Vault without changing the client application code. Vault Agent allows for easy authentication to Vault in a wide variety of environments. Vault Agent allows client-side caching of responses containing newly created tokens and responses containing leased secrets generated off of these newly created tokens. Vault Agent allows rendering of user-supplied templates by Vault Agent, using the token generated by the Auto-Auth.

In particular, Caravan relies on Vault Agent templates to render configuration files for a variety of components. For example, the configuration file of Nomad agents is a template rendered by Vault Agent, since it contains dynamic secrets like the Consul token and the TLS certificates used for communication with the server components.


Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock

Read More
Bitrock & CNCF

We are thrilled to announce that Bitrock has recently joined the Cloud Native Computing Foundation (CNCF), vendor-neutral home for many of the fastest-growing projects on GitHub (including Kubernetes and Prometheus) that fosters the collaboration between the industry’s top developers, end users, and vendors.

Kubernetes and other CNCF projects have quickly gained adoption and diverse community support in recent years, becoming some of the most rapidly-moving projects in the open-source history.

The CNCF, part of the non-profit Linux Foundation, creates long-term ecosystems and fosters a community around a collection of high-quality projects that orchestrate containers as part of microservice architectures.

Cloud Native

Organizations can use cloud native technologies to build and run scalable applications in modern, dynamic environments including public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable resilient, manageable, and observable loosely-coupled systems. They allow developers to make high-impact changes frequently and predictably with minimal effort when combined with robust automation.

The CNCF aims to accelerate adoption of this paradigm by cultivating and maintaining an ecosystem of open source, vendor-neutral initiatives that democratize cutting-edge patterns and make them available to everybody.

Bitrock’s New Membership

Bitrock has recently joined the CNCF community with a Silver Membership: a community that includes the world’s major public cloud and enterprise software firms as well as innovative startups. 

The choice to join this community was based on Bitrock Team's intention to build and influence the cloud native ecosystem alongside industry peers.

By providing governance, thought leadership, and engineering resources to shape and influence the development of the cloud-native ecosystem, Bitrock helps CNCF achieve its mission of making cloud-native computing ubiquitous.

Bitrock also upholds the community's key values as a member - find out more by accessing the CNCF charter:

  • Fast is better than slow 
  • Open 
  • Fair
  • Strong technical identity
  • Clear boundaries 
  • Scalable 
  • Platform agnostic

As a result of this new membership, Bitrock will be able to demonstrate thought leadership in the cloud native space, provide a first-class experience with Kubernetes and other CNCF projects in collaboration with the CNCF and its community, and design and maintain the technologies that are at the forefront of the industry.

Read More
Caravan Series - GitOps

This is the second entry in our article series about Caravan, Bitrock’s Cloud-Native Platform based on the HashiCorp stack. Click here for the first part.

What is GitOps

GitOps is "a paradigm or a set of practices that empowers developers to perform tasks that typically fall under the purview of IT operations. GitOps requires us to describe and observe systems with declarative specifications that eventually form the basis of continuous everything" (source: Cloudbees).

GitOps upholds the principle that Git is the only source of truth. GitOps requires the system’s desired state to be stored in version control such that anyone can view the entire audit trail of changes. All changes to the desired state are fully traceable commits, associated with committer information, commit IDs, and time stamps.

Together with Terraform, GitOps allows the creation of Immutable Infrastructure as Code. When we need to add or perform an update, we have to modify our code and create a Merge/Pull Request to let our colleagues review our changes. After validating our changes we merge to our main branch and let our CI/CD pipelines apply the changes to our infrastructure environments.

Another approach in GitOps avoids triggering a CI/CD pipeline after a new change is merged. Instead, the system automatically pulls the new changes from the source code, and executes the needed actions to align the current state of the system to the new desired state declared in the source code.

Caravan Logo

How GitOps helped us build Caravan

GitOps provides us with the ability and framework to automate Caravan provisioning. In practice, GitOps is achieved by combining IAC, Git repositories, MRs/PRs, and CI/CD pipelines.

First of all we define our infra resources as code. Each layer of the Caravan stack is built following GitOps principles, and the first one is of course the Infrastructure layer that allows declaring the required building block for the major cloud provider. Networking, Compute resources and Security rules are all tracked in the Git repository.

Then, the following layer is the Platform one where we bring online the needed components with the required configuration. Finally, we declare the Application Support components deployed on top of the Platform.

Currently, the applications are deployed using a simpler approach leveraging standard Terraform files that we called “Carts”. Nomad itself can pull configuration files from git repository but lacks a solution like ArgoCD for automatically pulling all the nomad job descriptors from git.


Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock

Read More
Caravan Series Part 1

Introduction

The current IT industry is characterised by multiple needs, often addressed by an heterogeneous number of products and services. To help professionals adopt the best performing solutions for sustainable development, the Cloud Native Computing Foundation was created in 2015 with the aim of advancing container technology and aligning the IT industry around its evolution.

We conceived Bitrock's Caravan project following the Cloud Native principles defined by the CNCF:

  • leverage the Cloud
  • be designed to tolerate Failure and be Observable
  • be built using modern SW engineering practices
  • base the Architecture on containers and service meshes

The HashiCorp stack fulfills these needs, enabling developers to build and run applications faster and more efficiently.

The Caravan Project

Caravan is your open-source platform builder based on the HashiCorp stack. Terraform and Packer are used to build and deploy a cloud-native and ready-to-use platform composed of Vault, Consul and Nomad.

Vault allows you to keep secrets, credentials and certificates safe across the Company. Consul allows the Service Discovery and, with Consul Connect, a Service Mesh to get the power of a truly dynamic communication among your next gen and legacy applications. Nomad allows powerful placing, scaling and balancing of your workloads that may be containerized or legacy, services or batches.

Thanks to Terraform and Ansible, the Infrastructure and Configuration as Code lie at the core of Caravan.

The rationale behind Caravan is to provide a one-click experience to deploy an entire infrastructure and the configuration needed to run the full HashiCorp stack in your preferred cloud environment.

Caravan’s codebase is modular and layered to achieve maximum flexibility and cover the most common use cases. Multiple cloud providers and optional components can be mixed to achieve specific goals.

Caravan supports both Open Source and Enterprise versions of HashiCorp products.

Caravan Project Functioning

Caravan in a nutshell

Caravan is the perfect modern platform for your containerized and legacy applications:

  • Security by default
  • Service mesh out of the box
  • Scheduling & Orchestration
  • Observability
  • Fully automated

Want to know more about Caravan? Visit the dedicated website, check our GitHub repository and explore our documentation.

Authors: Matteo Gazzetta, DevOps Engineer @ Bitrock - Simone Ripamonti, DevOps Engineer @ Bitrock

Read More