Service mesh implemented via iptables

So, what’s Conduit again?

As I said, Conduit is a service mesh for Kubernetes. Instead of letting application services to talk directly to each other, it forces them to send all traffic through the mesh. In return this allows us to collect all sorts of statistics and logs, adds more control over traffic routing and provides all other goodies that self respecting mesh should provide.

Installation

It’s also fairly easy to install. Basically, downloading the binaries via provided shell script and adding it to PATH is more than a half of a deal:

The second part is to add its control layer to Kubernetes cluster (in my case – minikube), which is simply another shell command:

Dissecting install script

Before we go further, it looks like conduit install simply generates YAML configuration for Kubernetes, which then gets applied by kubectl apply. I actually wonder, what’s inside.

Cool! So among the whole bunch of accounts, services and deployments they also use Prometheus and Grafana. I think I know how they are collecting and displaying the metrics now.

Checking the dashboard

Sure, there will be no data, but it still interesting to see how it looks. conduit dashboard does the trick:

That’s nice. All is up and running, all’s green. The most interesting part is in the bottom. It says that cluster has 4 k8s namespaces, but only one of them – conduit – is fully connected to the service mesh.

So they are monitoring conduit with conduit? Nice. And it also means that there will be some network activity to see after all. Clicking at conduit namespace and…

They do have metrics! Per deployment and per pods. OK, going further – clicking at grafana deployment:

And we see Grafana, displaying traffic data about Grafana. Almost a recursion.

Wiring up more pods

Let’s head back to the main page, to the list of wired/unwired namespaces.

conduit is completely wired, default and kube-public are empty, but kube-system is both unwired and has 9 pods in it. I don’t think that would be a good idea to do that in production, but for minikube – what if we try to connect some system pods to the mesh?

Connection to the mesh is done via command line (I think they also have API for that): conduit inject. It takes YAML configuration for Kubernetes object (e.g. pod or deployment) as an input and returns a new one, slightly modified, with its traffic configured to go through the mesh. Before I actually connect anything to it, let’s check what exactly gets injected into, let’s say, deployment.

Comparing before and after YAMLs

kube-system namespace, which is the only one in default minikube setting, that has any pods in it, has two deployments:

I stored existing kube-dns YAML configuration into before.yml file:

Modified one – into after.yml:

And then compared them with vim diff:

Look at that beauty. Apparently, they are adding two containers: one that seems to accept the traffic and resend it to controller pods (gcr.io/runconduit/proxy:v0.4.2, line 189) and the other one – Init Container, gcr.io/runconduit/proxy-init:v0.4.2, line 211 – to run before anything else does and to setup the whole thing. I wonder what’s inside.

Almost the rabbit hole

Literally the first google result gives me a Dockerfile for proxy-init image.

So it’s written in Go. The source files are also right there and the great reveal is that unlike Linkerd with it routing a traffic via http_proxy env variable, Conduit does this via iptables. At least that’s what my understanding of Go and word iptables tells me. As a side note, code for the second Docker image – proxy – is written in Rust. Right language for right job, cool.

Attaching kube-system deployments to the mesh

This satisfies my curiosity for now, so it’s time to try to connect the only other namespace with pods to the mesh: kube-system.

A-a-and it seems to work! Well, partially. kube-dashboard  deployment seems to realize that it was tampered and quickly reverted itself back to original configuration. But kube-dns stayed wired:

Bonus round: CLI

Conduit also support querying few sorts of statistics directly from a command line. For instance, the same data we saw for kube-dns deployment at previous screenshot, we can get in text form:

What’s even cooler, we can see live feed of traffic logs from a command line as well. For instance, for viewing logs for kube-dns deployment from kube-system namespace, this is the command to run:

Beautiful.

Conclusion

So this is Conduit. Like with Linkerd and a service mesh in general, I can’t say that playing with it was an eye opener for me. But I do like nice concepts and interesting implementation and a service mesh with Conduit is definitely the one. I really love how much one could learn just from looking at their source code: mixing languages works not only in theory, but in real products; Rust is alive; Init Containers are cool; iptables are too. I wonder, however, has anyone did a benchmarking of how using a service mesh affects performance? That naturally is going to be the first question anyone will ask me if I try to ‘sell’ the idea.