Monitoring Kafka Connector with Kubernetes

Monitoring Kafka Connector with Kubernetes

Monitoring Kafka Connector with Kubernetes


The Problem

The popularity of microservice architecture has enormously increased recently; but this comes with new challenges.

One of these is monitoring. In one of our projects, we used a Kafka connector to intercept changes in our database and write data to a topic. This was a very important component of the system, so we needed to consider its health status carefully.


Solution

In our first version, we created a Kubernetes’ CronnJob with a simple shell script that checks the status of the connector and, eventually, deletes the failed and restarts it.

This worked quite well; however, this is different from how the other services are health checked with the Kubernetes.

The connector was deployed with Kubernetes; the most natural thing to do is thus using k8s for monitoring pods and eventually restarting it.

The Kafka Connect framework comes with Rest API, and one of these gives you the state of the connectors:

i.e : https://bitrock.it/blog/monitoring-kafka-connector-with-kubernetes/

This seems to resolve our problem... But is it really the case?

Kubernetes health check controls the HTTP status code; the problem is that the Kafka connector API returns 200 HTTP status.

For instance, if the task is failed, the API will return:

HTTP/1.1 200 OK

{"state":"FAILED","id":1,"worker_id":"192.168.86.101:8083"}

In this case, from the Kubernetes point of view, everything is ok.

The solution that worked well for us consisted in adding a sidecar container that takes responsibility for exposing the state of the connector task.

The sidecar pattern allows you to extract some functionalities of your application in a different component. For example, we can separate the authentication layer from our “main” component that contains the business logic or - as in our case - extracts the monitoring part.

Our goal is to obtain something like this:

First of all, we created a simple application that takes care of calling the connector API and exposes an API for Kubernetes (we used a simple Python application using Flask - but you can use whatever you want). Something like this:

As you can see, the code is very simple.

The application does two different things: first of all, it exposes an endpoint at “/health” paths that will be called periodically by Kubernetes; secondly, it checks the status of a task and eventually returns an Internal Server Error, in case the HTTP status of the connector was not 200 or if the status was not “RUNNING”.

Now, this application needs to be deployed in the same pods of the connector. This can be done by adding to our deployment.yaml file the container that contains our Python application:


Conclusions

The logical result?

Both containers expose the health check of the sidecar, since Kubernetes does not restart the entire pods if one container is up; exposing the same API, the destiny of both containers would be the same.

Once the connector is in FAILED state, Kubernetes will restart the pod.

Some cloud providers may provide a built-in solution for problems like this; but if you can’t use it - for whatever reason - this can be a possible solution.


Author: Marco Tosini, Principal Engineer @Bitrock

Read More
getting Started with React Push-based Architecture

Getting Started with React Push-based Architecture

Getting Started with React Push-based Architecture

When approaching the React world, using Redux or MobX as state management is almost automatic. Or, in any case, the libraries change, but the basic architecture doesn’t: it is always something similar to the Redux Pattern with reducers, actions, selector, middleware, etc.

But is there the possibility of using a different architecture? Something with RxJs as with Angular? By doing some research, it seems so. Let's see more in detail what we are talking about.

First of all, we need to think outside the classic pull-based pattern and move to something new for those coming from the React world: a push-based architecture.

With data-push architectures, view components simply react to asynchronous data change notifications and render the current data values.

The library that allows us to manage the store in this way is Akita:

“Akita is a state management pattern, built on top of RxJS, which takes the idea of multiple data stores from Flux and the immutable updates from Redux, along with the concept of streaming data, to create the Observable Data Stores model.”

So basically, Akita enables us to easily build reactive, asynchronous, data-push solutions for our state management needs.

Another important concept to add is the one related to the Facades. Facades are a programming pattern in which a simpler public interface is provided to mask a composition of internal, more-complex, component usages.

In order to build our application, we rely on RxJS and React Hooks; nothing else is needed.

Let's now consider a very simple example built on the ideas found in some articles.

In our case we need to have a list of users and to be able to interface through the classic CRUD functions.

Starting from the well-known create-react-app with the addition of TypeScript, we create a folder that will contain our entities; in this case, it will only have a "user" folder as a child.

Inside, we define a simple interface of our "user" entity in the model.ts file:

Let's now start by initializing the store of our entity, creating a "UsersState" interface and then creating a "UsersStore" store by extending the Akita store, and finally exporting it:

At this point, we can create services to manipulate the store, also relying on the methods that an Akita store provides.

This is where we can use all our knowledge of RxJS in order to be able to create more complex flows to act on the store.

Finally, through the "QueryEntity", we can take the whole store - or just a filtered part - and channel it into an observable stream of RxJS.

Last but not least, the creation of a custom Hooks that will internally manage all issues regarding RxJS, Facades, and Akita.

First, we map and expose the services of our "userService", in this case all.Then, we create the internal state of our custom hook. Finally, we need to build the selectors for \users\ and \active\ state changes and manage subscriptions with auto-cleanup.

Now our user entity should have everything needed. We import our custom Hooks, and that's it.

To play a little bit, let’s divide the application into several components in order to test it. The result? Well, it works!

And here’s the child component:

Here’s how the application works in the browser:

Conclusions

Although this example is quite simple, the outcomes are pretty surprising. It was really easy - and also quite logical - to connect all the pieces to compose the state management and, as we have seen, no configurations (of any kind) were needed.

For those approaching an architecture like this for the first time, the greatest difficulty is certainly represented by RxJS. To write simple services or queries, it may be enough to know the basics of RxJS; however, in case of large applications with complex services, a good knowledge of technology makes a huge (positive difference), really giving an edge. Furthermore, you need to be very careful where and how you use all the various facades in your application. Being in a push pattern, any change of state triggers the React lifecycle in every component that uses our hooks; watching and controlling performance is thus very important.

Obviously, this is just the beginning: there is a world of things to say about Akita, RxJS, push-patterns etc, and it would take much more than one simple article to explore all of them.

The aim of this contribution was to give you just a little idea of this "new" architecture for state management with React. I hope I’ve hit the target.


Author: Mattia Ripamonti, UX/UI Engineer @Bitrock


Useful Resources:

1 - React Facade Best Practices

2 - React Hooks RxJs Facades

3 - Push Based Architectures with RxJs

4 - Managing State in React with Akita

Read More
From Layered to Hexagonal Architecture

From layered to Hexagonal Architecture (Hands On)

From layered to Hexagonal Architecture (Hands On)


Introduction

The hexagonal architecture (also called “ports and adapters”) is an architectural pattern used in software design designed in 2005 by Alistair Cockburn.

The hexagonal architecture is allegedly at the origin of the microservices architecture.


What it Brings to the Table

The most used service architecture is layered. Often, this type of architecture leads to dependencies of business logic from external contract (e.g., database, external service, and so on). This brings stiffness and coupling to the system, forcing us to recompile classes that contain the business logic whenever an API changes.


Loose coupling

In the hexagonal architecture, components communicate with each other using a number of exposed ports, which are simple interfaces. This is an application of the Dependency Inversion Principle (the “D” in SOLID).


Exchangeable components

An adapter is a software component that allows a technology to interact with a port of the hexagon. Adapters make it easy to exchange a certain layer of the application without impacting business logic. This is a core concept of evolutionary architectures.


Maximum isolation

Components can be tested in isolation from the outside environment or you can use dependency injection and other techniques (e.g., mocks, stubs) to enable easier testing.

Contract testing supersedes integration testing for a faster and easier development flow.


The domain at the center

Domain objects can contain both state and behavior. The closer the behavior is to the state, the easier the code will be to understand, reason about, and maintain.

Since domain objects have no dependencies on other layers of the application, changes in other layers don’t affect them. This is a prime example of the Single Responsibility Principle (the “S” in “SOLID”).


How to Implement it

Let's now have a look on what it means to build a project following the hexagonal architecture to better understand the difference and its benefit in comparison with a more common plain layered architecture.


Project layout

In a layered architecture project, the package structure usually looks like the following:

Here we can find a package for each application layer:

  • the one responsible for exposing the service for external communication (e.g., REST APIs);
  • the one where the core business logic is defined;
  • the one with all the database integration code;
  • the one responsible for communicating with other external services;
  • and more...


Layers Coupling

At first glance, this could look like a nice and clean solution to keep the different pieces of the application separated and well organized, but, if we dive a bit deeper into the code, we can find some code smells that should alert us. In fact, after a quick inspection of the core business logic of the application, we immediately find something definitely in contrast with our idea of clean and well defined separation of the various components. The business logic that we'd like to keep isolated from all the external layers clearly references some dependencies from the database and the external service package.

These dependencies imply that in case of changes in the database code or in the external service communication, we'll need to recompile the main logic and probably change and adapt it, in order to make it compatible with the new database and external service versions. This means that we need to spend time on this new integration, test it properly and, during this process, we expose ourselves to the introduction of some bugs.


Interfaces to the Rescue

This is where the hexagonal architecture really shines and helps us avoid all of this. First we need to decouple the business logic from its database dependencies: this can be easily achieved with the introduction of a simple interface (also called “port”) that will define the behavior that a certain database class needs to implement to be compatible with our main logic.

Then we can use this contract in the actual database implementation to be sure that it's compliant with the defined behavior.

Now we can come back to our main logic class and, thanks to the changes described above, we can finally get rid of the database dependency and have the business logic completely decoupled from the persistence details.

It's important to note that the new interface we introduced is defined inside the business logic package and, therefore, it’s part of it and not of the database layer. This trick allows us to apply the Dependency Inversion Principle and keep our application core pure and isolated from all the external details.

We can then apply the same approach to the external service dependency and finally clear the whole logic class of all its dependencies from the other layer of the application.


DTO for model abstraction

This already give us a nice level of separation, but there is still room for improvement. In fact if you look at the definition of the Database class you will notice that we are using the same model from our main logic to operate on the persistence layer. While this is not a problem for the isolation of our core logic, it could be a good idea to create a separate model for the persistence layer, so that if we need to make some changes in the structure of the table, for example, we are not forced to propagate the changes also to the business logic layer. This can be achieved with the introduction of a DTO (Data transfer object).

A DTO is nothing more that a new external model with pair of mapping function that allow us to transform our internal business model to the external one and the other way around. First of all, we need to define the new private model for our database and external service layers.

Then we need to create a proper function to transform this new database model into the internal business logic model (and vice versa based on the application needs).

Now we can finally change the Database class to work with the newly introduced model and transform it into the logic one when it communicates with the business logic layer.

This approach works very well to protect our logic from external interference, but it has some consequence. The main one is an explosion of the number of the models, when most of the time the models are the same; the other one is that the logic about transforming models can be tedious and always need to be properly tested to avoid errors. One compromise that we can take is starting only with the business models (defining them in the correct package) and introduce the external models only when the two models diverge.

When to embrace it

Hexagonal architecture is no silver bullet. If you’re building an application with rich business rules that can be expressed in a rich domain model that combines state with behavior, then this architecture really shines because it puts the domain model in the center.

Combine it with microservices architecture and you’ll get a future-proof evolutionary architecture.

Read More