GitLab CI for ESP32 and Arduino

It’s been a while since my last post. But fear not – the blog is not dead! No sir. In fact, I’ve gotten myself into a new project which urges to be written about. The project is called “Jerry Annihilator” and it’s a robot that eventually will bring me a fame, a fortune, and a world domination. Here he is, my ticket to the bright future:

OK, he doesn’t look intimidating now, but first – it’s not the latest revision. And the second – the guy’s just a few months old, give him a break.

So anyway, despite the age, the project is backed by a quite a bit of code, which, though being maintained by GitLab server (basement edition), still has zero unit tests and equal amount of continuous integration. Of cause, I could justify it by me being an ex web-developer, so any CI/CD and unit testing activities in Arduino IDE / ESP32 C++ project environment would be disturbingly unnatural. However, being a consultant now, especially one who also specializes in CI/CD, makes that “ex web-developer” excuse invalid.

So today I try to make the things right and for starters fix the CI part – if code hits the server, it should be built. That should be a good start.

Main ingredient

The thing that prevented me from adding CI in the past was Arduino IDE. Yup, that window hungry and toolchain concealing tool, which I had no idea how to cross breed with windowless containerized build server. However, Arduino IDE’s got its younger brother now – arduino-cli, which sounds like a silver bullet for my problem. If it does what it says, I’ll simply need to draft a good old .gitlab-ci.yml, point it to GitLab CI’s docker executor, install and configure arduino-cli, download the same libraries that I used with Arduino IDE and voila!

I won’t get into details about how GitLab CI itself works. I also won’t describe its runners and especially the ones that use docker. That’s just too much info, which already was written about. Instead, will start right at the point where GitLab CI already exists and it has properly configured docker runner tagged with the word docker. So the obvious first step is to create .gitlab-ci.yml file.

.gitlab-ci.yml

Here’s what our .gitlab-ci.yml should do:

1. Request a build in docker container
2. When in there, install arduino-cli and all imaginable dependencies
3. After that, build project

Why in docker? Well,  I don’t want to pollute the environment. Go green and stuff. Plus, building a project and configuring all of its dependencies would be the ultimate proof of us knowing both how to build, and how to configure the environment for that.

1. Request a build in docker executor

.gitlab-ci.yml can target specific GitLab CI runner by specifying its tag. As I mentioned before, our docker runner is tagged by docker, so the same tag should be mentioned CI’s build step. Another thing is that I don’t remember what docker image that runner uses by default. In fact, that’s a good thing, as being explicit about ones expectations is a virtue in many contexts. The project’s ‘home’ OS is Ubuntu, so let’s request ubuntu:18.04 image for a build as well. Having said that, here’s what .gitlab-ci.yml starts to look like:

2. Installing arduino-cli and its dependencies

2.1 Setting up arduino-cli

arduino-cli is just a downloadable archive, so a few lines of bash script can be called ‘an installation’. However, after these few bash lines many other will follow, so let’s create a new shell file and put everything in there. I’ll call it setup-build-env.sh:

The file has zero sudo‘s, because 1) most likely it’s already going to run as root in Docker container and 2) even if it’s not, it’s easier to add one sudo before the file name than one before every other line in the file itself.

2.2 Setting up ESP32 support

Now the fun part begins. I remember that in order to add ESP32 support to Arduino IDE I had to open some board manager window and copy-paste mysterious URL. Apparently, the same stuff needs to be done here, but this time through cli.

That’s very easy. First, we create .arduino-cli.yaml file. The name has no special meaning, it’s just the first one that came into my mind. I believe arduino-cli does have some default file name, but I don’t trust defaults in a tool that’s still released in ‘preview’ edition. The last line installs magical esp32:esp32, which I learned about by executing arduino-cli core search esp32 --config-file .arduino-cli.yaml. Having ESP32 core installed, we can, for instance, run another command – arduino-cli board listall – and find out the identifier of the board for which we’ll compile the project. I’m using a ESP32 Dev Board equivalent, whose fully qualified board name appears to be  esp32:esp32:esp32.

2.3 Installing standard packages

It took me some time, but finally I came up with the list of Arduino libraries that I installed over the past couple of months. Now we’ll need that list to reinstall those libraries for arduino-cli.

2.4 Installing non-standard packages

Along with standard Arduino packages I also used a number of Github projects which I cloned into Arduino libraries folder. If we find where arduino-cli stores its stuff, we could git clone them again.

Fortunately, arduino-cli config dump can tell us exactly where:

So:

2.5 Setting up python and pyserial

If I’m not mistaken, these two are required by ESP32 IDF – the native ESP32 toolchain that Arduino runs under the hood. Whether or not that is true, without python and pyserial arduino-cli refuses to compile ESP32 boards, so we’ll need them anyway:

2.6 …and putting it all together

Putting all these pieces into setup-build-env.sh file, here’s what we get in the end:

The remaining part is to add it to  .gitlab-ci.yml and call just before the build.

Ideally, instead of installing the dependencies before every build we should’ve baked them into underlying Docker image and focus only at the build part. However, for that we’d also need the docker registry to store that image, configure gitlab runner to know where the registry is, come up with the image lifecycle, etc. Given the frequency of my commits, the time overhead caused by fully configuring the environment is bearable.

3. Finally, the build step

So the only missing step is the compilation itself. I already know fully qualified board name, I know how to spell the word compile, so putting it all together and pointing it to a project file called robot.ino, here’s what build line ends up being:

PROFIT!

This is the final .gitlab-ci.yml:

And this is how happy GitLab CI server is when it sees it!

It’s alive!

Conclusion

You see, setting up CI for ESP32 and Arduino project wasn’t that hard after all. My result is not entirely an example of a perfect design, but it works, it does its job, and it’s definitely better than not having CI at all. After all, now I at least know what dependencies my project has and how to reconfigure the build environment on other laptops. Without any doubt, that’s a total win.