GitlabCI can help you debug many problems that happen in cloud based continuous integration by allowing you to run the build locally. If you're leveraging docker to run your builds, it can also eliminate the "It works on my machine -- but not in CI" class of problems as well.
I'm going to walk you through setting up
gitlab-runner locally, and show you how to start using it to speed up your debugging process. I'll explain how the architecture of GitlabCI allows us to do build locally, and some gotchas you may encounter.
The main assumption is that you're interested in or already using Gitlab. Not convinced you should switch? Read why I use Gitlab.
You Already Have GitlabCI Setup for Your Project
This post isn't very helpful unless you've already setup GitlabCI using a .gitlab-ci.yml file. If you haven't already done that, checkout the getting started guide.
You Should Install Docker
If your builds can run on Linux, you really should be using the Docker executor. Even if your machine runs on Windows, MacOS, or something else that supports Docker, you should install Docker.
The main benefits that accrue to using Docker with GitlabCI are:
- Docker basically guarrantees that testing on your machine will produce identical results with a remote CI server (which is kinda the whole point of doing this). Without docker, you're much less likely to discover why something worked locally but broke in CI.
- Docker can ensure that the testing environment configured to our exact specifications. The
$PATHcan be configured ahead of time, various binaries and utilities installed, DNS configured, security restrictions imposed, etc, etc. The tradeoff is that docker will use more memory and be slower running on bare metal. And docker pull... why can't it be faster???
The main reason you might not care about installing Docker is if you want to build and test on a platform that isn't Linux. Docker can entirely capture all aspects of a local Linux system, but it can't capture Windows, Mac, Solaris, or BSD systems (to name a few). If you specifically want to reproduce issues occurring on those platforms, Docker won't help you. You'll need to find some other way to make sure your local machine is an exact replica of the remote CI server so that you can locally debug those issues.
So, if you haven't already installed Docker, here's how to do it in Linux:
$ curl -sSL https://get.docker.com/ | sh
For MacOS or Windows, go to the Docker download page, download your installer, and run it.
How Does Gitlab Runner Relate to GitlabCI?
GitlabCI is composed of a "dashboard" and many "runners".
The GitlabCI Dashboard
The dashboard is part of Gitlab and can be controlled by:
- doing a git push to your gitlab instance
- starting, cancelling, or retrying a pipeline in the web interface
- making API requests to the gitlab instance
The dashboard is the command and control center. It doesn't do the work; it just delegates and supervises a pool of runners. When the dashboard receives instructions to start a pipeline, it adds that pipeline to a queue, and then feeds work from the queue to available runners.
The Gitlab Runners
The runners are individual computers that are running the Gitlab Runner binary. Typically, runners register themselves with the dashboard using a secure token generated by the dashboard. Runners can have names and tags; they receive instructions from the dashboard, execute the command and continuously report back results of running pipelines and jobs. It's very important that registered runners have as close to 100% network connectivity as possible to ensure that jobs can be run on demand. Typically, the runners are installed as a system service or daemonized.
Runners can also operate without connecting to a Gitlab instance. In this offline mode, they have limited functionality, but they can read a local git repository and directly execute jobs listed in a .gitlab-ci.yml. Offline runners must be invoked directly on the command line to execute; they don't run daemonized in this mode.
In this guide, we'll be installing Gitlab Runner on your local machine and running in this offline mode to debug CI problems locally.
Install Gitlab Runner on MacOS
At time of writing these are the official steps:
$ sudo curl --output /usr/local/bin/gitlab-ci-multi-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-darwin-amd64 $ sudo chmod +x /usr/local/bin/gitlab-ci-multi-runner
NOTE: For the most current instructions, check the official documentation.
Unfortunately, Gitlab Runner has different names on MacOS and Linux. I tend to prefer the shorter name, so I'd recommend you add a symlink in your path to the Linux name (
gitlab-runner). I'll be referring to the command as
gitlab-runner in the rest of this post, so if you don't symlink as recommended below, use
gitlab-ci-multi-runner in your terminal instead of
$ sudo ln -s /usr/local/bin/gitlab-ci-multi-runner /usr/local/bin/gitlab-runner $ which gitlab-runner /usr/local/bin/gitlab-runner
Install Gitlab Runner on Ubuntu
For Ubuntu (and presumably Debian as well), you'll want to use Gitlab's PPA so that you're always on a current version of GitLab Runner. At the time of writing, you can add the Gitlab PPA like so:
$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash
(Note that this may change over time. Here's the current installation instructions for Gitlab Runner.)
gitlab-runner itself now:
$ sudo apt-get install gitlab-ci-multi-runner
Double Check that Gitlab Runner is working
You should see output something like this:
NAME: gitlab-runner - a GitLab Runner USAGE: gitlab-runner [global options] command [command options] [arguments...] VERSION: 1.11.0 (33af656) AUTHOR(S): GitLab Inc. <email@example.com> COMMANDS: exec execute a build locally ...
I've shortened the output considerably. Basically, you should see lots of help text explaining how this command works.
If you don't see help text and see an error message instead, reach out to me on Twitter or in the comments, or consider reaching out to Gitlab for support.
Run a Job with gitlab-runner
Let's assume you have .gitlab-ci.yml file already committed that looks like this:
image: node:latest my_special_tests: script: - npm install - npm test
We can run the job
my_special_tests like so:
$ cd path/to/project $ ls .gitlab-ci.yml .gitlab-ci.yml $ gitlab-runner exec docker my_special_tests
The output of that command will look nearly identical to what you normally see on gitlab.com in your job page.
Let's break down the above syntax a little bit:
cd ...- just make sure you're in the root directory of your project
ls .gitlab-ci.yml- there should already be a .gitlab-ci.yml file in this directory
gitlab-runner exec docker tests
gitlab-runner- this is main command that allows you to control the gitlab-runner. This is same code used in a remote environment to run your tests on gitlab.com and on your own instances of gitlab.
exec- this subcommand allows us to work directly with our local git repo and execute builds without doing lots of fancy setup. It's quick and dirty -- exactly what we want for quick debugging
docker- this means "use the docker executor to run this build". This is the preferred way to run builds. However if you need to run builds on MacOS or Windows, you'll probably want to replace with
shellinstead. You just won't get many guarrantees about the environment your shell runs in.
my_special_tests- this is the name of the test to run taken from the .gitlab-ci.yml file. If you had a job named 'foobar' you'd repalce
Gitlab Runner Exec Gotchas
- Even your local changes must be committed to git or they won't be available to gitlab-runner
- The cache and artifacts in your .gitlab-ci.yml won't work with the
gitlab-runner exec dockercommand
- Run gitlab-runner locally any time you're initially setting up a new .gitlab-ci.yml file. You can catch typos in your shell scripting or find problems with your
$PATHin a few seconds or minutes instead of hours.
- Leverage Docker - you'll be able to debug problems from a machine using the big 3 operating systems. You'll also be guarranteed an identical build in your remote CI.
- Fallback to deploying from local machines if your cloud based CI goes down; just make sure to use a password manager like LastPass or 1Password to store deployment secrets, and document the procedure. This can be a really nice way to reduce a single point of failure.
While CircleCI, TravisCI, and Jenkins have many fantastic features, its very difficult to get into a "flow" state when debugging them. This can be especially true if many other jobs are running at the same time. GitlabCI lets you do end-run around contention in CI and get results much faster. As a side benefit, when GitlabCI goes down, you can continue to run your deploy scripts locally by invoking Gitlab Runner directly. Setup Gitlab Runner right now and speed up your own debugging speed.