Easy continuous integration and deployment with GitLab CI

Last month we finally finished migration from our previous CI/CD system to GitLab CE and that’s something that makes me extremely happy. It’s just so much easier to maintain our CI/CD monster, when repository, build configurations, build results, test results and even that “Approve” button that publishes the build to release repository – when they all are in the same place.

And what I particularly love about GitLab is how simple it is to configure all of that. So simple, that today I’ll show you how to setup fully functional CI/CD for demo project starting from installing GitLab and finishing with successful commit landing at “production” server. So, without further ado, let’s begin.

Step 0. Demo project

I have silly TypeScript web project that simply changes the message on a web page as soon as it’s loaded. As this project requires some sort of compilation, has room for testing, and can be deployed after first two steps succeeded, we can setup CI/CD pipeline for it with “build”, “test” and “deploy” stages. But we’ll get to that later. For now, this is our guinea pig:

Apart from the web content, the project also needs .gitignore file so we don’t commit generated JS files, and README.md, so the project looks cooler than it is.

And that’s it. Assuming there’s TypeScript compiler installed locally (e.g. via sudo npm install -g typescript), tsc index.ts will do the compilation and any web server, sourced from current directory, will be able to display remarkably unremarkable page:

Now, let’s run few more commands to create local repo and we’re good to go to the next step: installing GitLab.

Step 1: Installing GitLab

Cool thing about gitlab is that it can be deployed as Docker container and there’s even image for that. In fact, dockerized gitlab has been running at my home server for months with absolute zero of problems.

As I’m going to deal with more than one container, instead of typing docker run ... I’ll use docker-compose instead, so we can add more containers as we go.

Initial version of docker-compose.yml file is trivial:

docker-compose up -d gitlab begins and concludes installation and after a minute or so we can configure our root account.

OK, we’re in. The next thing is adding our demo project into it. “Add” button is right in the top:

When it’s done, we can use git remote add and git push -u origin master commands nicely suggested by GitLab to import the demo project and then move to the first stage in our CI/CD GitLab pipeline: the build stage.

Step 2: Configuring “Build” stage

Telling GitLab to do things automatically when code gets pushed in it is quite simple. It requires .gitlab-ci.yml instructions file in project repository and some sort of build server (or container, or VM) called “runner” to perform those instructions. Let’s start with the first one.

.gitlab-ci.yml

Our build stage simply means invoking TypeScript compiler against index.ts file on any build runner that has TypeScript installed. Here’s how I’d put that into .gitlab-ci.yml:

Pretty straightforward, right? stages describes what stages we have, stage: build tells what stage current build job (“Compile”) belongs to, and tags will look for build runners that have these tags assigned to, so we can send build jobs to specific machines. It’s up to us to come up with the list of tags and assign them to both the runner and the build job. Personally, I tag runners with the features that they support (compilers, versions, tools) and then mention those tags in individual build jobs as a requirements list.

If we commit this file and push it to origin, something slightly cool will happen: we can go to “Pipelines” page and see something like this:

Apparently, GitLab noticed new commit, recognized CI configuration and tried to launch a build job for it. But it’s stuck, as there’s no capable runner to perform that job. Don’t worry, we’ll create that in a second.

Configuring gitlab-runner service

What converts regular host to build runner? Installed gitlab-runner service and nothing more. Basically, we need to find a container or host, apt-get or download gitlab-runner to it, configure it to work with GitLab server, assign a tag or two to describe what it’s capable of, and we’re done.

For simplicity I’ll install a runner in a Docker container and add it to docker-compose.yml, so the runner and GitLab will be in the same network. This is how a Dockerfile for such runner could look like:

I used nodejs base image, so it later can be used for installing TypeScript. Most of the code I copy-pasted from the official guide. RUN section installs all the dependencies we need and CMD section actually registers and starts the runner. Ugly, but it will work.

There’re four particularly interesting parameters in gitlab-runner register:

• -u – specifies where GitLab server is. As container is going to run in the same Docker network as gitlab container does, I can use container name as a host name.
• -r – secret token that establishes the trust between gitlab and its runner. You can find yours at “GitLab -> Settings -> CI/CD settings” page. 
• --executor specifies how to interpret commands from .gitlab-ci.yml file. In our case it’s shell – regular shell commands. It also could’ve been powershell for Windows machine, or even docker.
• --tags – what features current gitlab-runner is going to support. For now, it only supports typescript, which was installed few lines above.

When I add this Dockerfile as runner-ts service to docker-compose.yml and start it with docker-compose up -d runner-ts, that pending pipeline will continue and succeed.

Breaking the build

Out of curiosity, what will happen if I try to commit invalid index.ts? It should fail, right? Let’s insert unwanted whitespace inside of the document.

Without much of a surprise, the pipeline for broken commit fails as well:

I can click at any of red icons and see more details about the failure.

Obviously, fixing the commit will fix the build as well:

That’s the power of CI!

Step 3. Configuring “Test” stage

I have no intention to install and configure testing framework, especially when the only thing to test I can think of is testing rendered content, but we can install tslint for testing the code style. It’s just one more section in .gitlab-ci.yml and one more runner to add. We also could’ve installed tslint on existing runner, but I prefer to keep existing stuff immutable.

First, let’s add new test job and its stage into .gitlab-ci.yml:

And then copy-paste the most of the Dockerfile for runner-ts into runner-tslint (if anyone asks, I never heard about code reuse):

Because copy-pasted typescript installation is also still there, our runner supports both typescript and tslint build jobs and we can clearly indicate that with proper tags.

Finally, let’s add new runner to docker-compose.yml, create it with docker-compose up -d runner-tslint and after commit with updated .gitlab-ci.yml finds its way to remote origin, we’ll have ourselves two consequent CI stages: Build and Test:

Apparently, tslint didn’t like my index.ts style and the build failed.

However, one more commit with the fix will make it happy again.

Step 4. Adding “Deploy” stage

CI/CD has “D” in it for a reason. Successfully built and tested code should be deployed somewhere and this is exactly what we’re going to do now.

Quite often new builds are deployed at two environments: staging and production. The first one is safer to deploy to as that supposed to be testing environment anyway. Installing the new code in production on the other hand requires more thinking and prayers. Having said that, let’s add two more build jobs – deployment jobs. One will automatically deploy successful build into staging environment as soon as the build is ready. Installing the build in production, however, will remain manual job.

Deployments runner

I was puzzled at first about how to emulate two different servers, but eventually decided to start two nginx containers with their html folders being connected to third container – another gitlab runner. One more copy-pasting experience, and we’ve got ourself one more Dockerfile for deployment runner:

This time it’s based on ubuntu image and unlike the first two times gitlab-runner will use root account to run. Otherwise it won’t have enough permissions to write to shared volume (and ubuntu:16.04 doesn’t have sudo in it).

Sharing the volumes between containers is quite straightforward:

Launching new containers works without a glitch, so we can move to .gitlab-ci.yml.

Deployment jobs in .gitlab-ci.yml

First of all, as we’re going to deploy compiled JS files, and those were produced at “Build” stage, we need to make sure that “Deploy” stage, which runs at separate host, can access them. Making compilation result an “artifact” will make that happen:

Now to deployment jobs. I’ll put the whole and final version of .gitlab-ci.yml file right here and explain what’s changed:

Obviously, we’ve got ourself a new stage (deploy) in the top and two build jobs in the bottom (Deploy-Staging and Deploy-Production). The only thing that differs regular build job from deployment job is environment section. Everything else is identical.

Environment name is absolutely arbitrary (and can be dynamically generated) and you can choose it in the way that suits the most for your project.

The last line, when: manual really makes the job manual, no magic here.

After I commit the changes in .gitlab-ci.yml file and push them to remote origin (and fix few errors along the way), here’s what happens:

We’ve got three green “check” marks, three successful stages. Clicking on pipeline details, however, shows that only one of two deployment jobs has actually run:

As configured, deploying in production is manual job and we can start it by clicking “Play” button.

Finally, we can see our deployments history at “Environments” page. Link buttons at left hand side are pointing to staging and production  containers (127.0.0.1:8081 and 127.0.0.1:8082) with newest content deployed in them:

Conclusion

Just think about what we’ve just done. We’ve configured fully functional CI/CD system, in which any push to repository will get compiled, tested and potentially deployed at testing server. And it’s all 100% automatic. If new code has a mistake, you’ll get notified about it right away. If the code is correct, your shiny new feature will get to the testing server as soon as possible. That’s huge.

CI/CD configuration looks trivial for hello-world app, but I assure you, for large projects it’s not much harder. More build jobs, more runners, but the same principles.