GitlabCI can help you debug many problems that happen in cloud based continuous integration by allowing you to run the build locally. If you're leveraging docker to run your builds, it can also eliminate the "It works on my machine -- but not in CI" class of problems as well.

I'm going to walk you through setting up gitlab-runner locally, and show you how to start using it to speed up your debugging process. I'll explain how the architecture of GitlabCI allows us to do build locally, and some gotchas you may encounter.

Assumptions

The main assumption is that you're interested in or already using Gitlab. Not convinced you should switch? Read why I use Gitlab.

You Already Have GitlabCI Setup for Your Project

This post isn't very helpful unless you've already setup GitlabCI using a .gitlab-ci.yml file. If you haven't already done that, checkout the getting started guide.

You Should Install Docker

If your builds can run on Linux, you really should be using the Docker executor. Even if your machine runs on Windows, MacOS, or something else that supports Docker, you should install Docker.

The main benefits that accrue to using Docker with GitlabCI are:

  • Docker basically guarrantees that testing on your machine will produce identical results with a remote CI server (which is kinda the whole point of doing this). Without docker, you're much less likely to discover why something worked locally but broke in CI.
  • Docker can ensure that the testing environment configured to our exact specifications. The $PATH can be configured ahead of time, various binaries and utilities installed, DNS configured, security restrictions imposed, etc, etc. The tradeoff is that docker will use more memory and be slower running on bare metal. And docker pull... why can't it be faster???

The main reason you might not care about installing Docker is if you want to build and test on a platform that isn't Linux. Docker can entirely capture all aspects of a local Linux system, but it can't capture Windows, Mac, Solaris, or BSD systems (to name a few). If you specifically want to reproduce issues occurring on those platforms, Docker won't help you. You'll need to find some other way to make sure your local machine is an exact replica of the remote CI server so that you can locally debug those issues.

So, if you haven't already installed Docker, here's how to do it in Linux:

$ curl -sSL https://get.docker.com/ | sh

For MacOS or Windows, go to the Docker download page, download your installer, and run it.

How Does Gitlab Runner Relate to GitlabCI?

GitlabCI is composed of a "dashboard" and many "runners".

The GitlabCI Dashboard

The dashboard is part of Gitlab and can be controlled by:

  • doing a git push to your gitlab instance
  • starting, cancelling, or retrying a pipeline in the web interface
  • making API requests to the gitlab instance

The dashboard is the command and control center. It doesn't do the work; it just delegates and supervises a pool of runners. When the dashboard receives instructions to start a pipeline, it adds that pipeline to a queue, and then feeds work from the queue to available runners.

The Gitlab Runners

The runners are individual computers that are running the Gitlab Runner binary. Typically, runners register themselves with the dashboard using a secure token generated by the dashboard. Runners can have names and tags; they receive instructions from the dashboard, execute the command and continuously report back results of running pipelines and jobs. It's very important that registered runners have as close to 100% network connectivity as possible to ensure that jobs can be run on demand. Typically, the runners are installed as a system service or daemonized.

Runners can also operate without connecting to a Gitlab instance. In this offline mode, they have limited functionality, but they can read a local git repository and directly execute jobs listed in a .gitlab-ci.yml. Offline runners must be invoked directly on the command line to execute; they don't run daemonized in this mode.

In this guide, we'll be installing Gitlab Runner on your local machine and running in this offline mode to debug CI problems locally.

Install Gitlab Runner on MacOS

At time of writing these are the official steps:

$ sudo curl --output /usr/local/bin/gitlab-ci-multi-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-darwin-amd64
$ sudo chmod +x /usr/local/bin/gitlab-ci-multi-runner

NOTE: For the most current instructions, check the official documentation.

Unfortunately, Gitlab Runner has different names on MacOS and Linux. I tend to prefer the shorter name, so I'd recommend you add a symlink in your path to the Linux name (gitlab-runner). I'll be referring to the command as gitlab-runner in the rest of this post, so if you don't symlink as recommended below, use gitlab-ci-multi-runner in your terminal instead of gitlab-runner.

$ sudo ln -s /usr/local/bin/gitlab-ci-multi-runner /usr/local/bin/gitlab-runner
$ which gitlab-runner
/usr/local/bin/gitlab-runner

Install Gitlab Runner on Ubuntu

For Ubuntu (and presumably Debian as well), you'll want to use Gitlab's PPA so that you're always on a current version of GitLab Runner. At the time of writing, you can add the Gitlab PPA like so:

$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash

(Note that this may change over time. Here's the current installation instructions for Gitlab Runner.)

Let's install gitlab-runner itself now:

$ sudo apt-get install gitlab-ci-multi-runner

Double Check that Gitlab Runner is working

$ gitlab-runner

You should see output something like this:

NAME:
   gitlab-runner - a GitLab Runner

USAGE:
   gitlab-runner [global options] command [command options] [arguments...]

VERSION:
   1.11.0 (33af656)

AUTHOR(S):
   GitLab Inc. <support@gitlab.com> 

COMMANDS:
   exec			execute a build locally
...

I've shortened the output considerably. Basically, you should see lots of help text explaining how this command works.

If you don't see help text and see an error message instead, reach out to me on Twitter or in the comments, or consider reaching out to Gitlab for support.

Run a Job with gitlab-runner

Let's assume you have .gitlab-ci.yml file already committed that looks like this:

image: node:latest

my_special_tests:
  script:
    - npm install
    - npm test

We can run the job my_special_tests like so:

$ cd path/to/project
$ ls .gitlab-ci.yml
.gitlab-ci.yml
$ gitlab-runner exec docker my_special_tests

The output of that command will look nearly identical to what you normally see on gitlab.com in your job page.

Let's break down the above syntax a little bit:

  • cd ... - just make sure you're in the root directory of your project
  • ls .gitlab-ci.yml - there should already be a .gitlab-ci.yml file in this directory
  • gitlab-runner exec docker tests
    • gitlab-runner - this is main command that allows you to control the gitlab-runner. This is same code used in a remote environment to run your tests on gitlab.com and on your own instances of gitlab.
    • exec - this subcommand allows us to work directly with our local git repo and execute builds without doing lots of fancy setup. It's quick and dirty -- exactly what we want for quick debugging
    • docker - this means "use the docker executor to run this build". This is the preferred way to run builds. However if you need to run builds on MacOS or Windows, you'll probably want to replace with shell instead. You just won't get many guarrantees about the environment your shell runs in.
    • my_special_tests - this is the name of the test to run taken from the .gitlab-ci.yml file. If you had a job named 'foobar' you'd repalce my_special_tests with foobar instead.

Gitlab Runner Exec Gotchas

  • Even your local changes must be committed to git or they won't be available to gitlab-runner
  • The cache and artifacts in your .gitlab-ci.yml won't work with the gitlab-runner exec docker command

Protips

  • Run gitlab-runner locally any time you're initially setting up a new .gitlab-ci.yml file. You can catch typos in your shell scripting or find problems with your $PATH in a few seconds or minutes instead of hours.
  • Leverage Docker - you'll be able to debug problems from a machine using the big 3 operating systems. You'll also be guarranteed an identical build in your remote CI.
  • Fallback to deploying from local machines if your cloud based CI goes down; just make sure to use a password manager like LastPass or 1Password to store deployment secrets, and document the procedure. This can be a really nice way to reduce a single point of failure.

Conclusion

While CircleCI, TravisCI, and Jenkins have many fantastic features, its very difficult to get into a "flow" state when debugging them. This can be especially true if many other jobs are running at the same time. GitlabCI lets you do end-run around contention in CI and get results much faster. As a side benefit, when GitlabCI goes down, you can continue to run your deploy scripts locally by invoking Gitlab Runner directly. Setup Gitlab Runner right now and speed up your own debugging speed.