Setup Gitlab Runner in AWS Ec2

This article showcase all the required steps to get the GITLAB CI/CD pipelines run in a self-hosted Gitlab runner. In addition , will see how to run docker commands in the custom runners.

Gitlab Runners: They are open-source tool developed by GitLab that works with GitLab CI/CD to run jobs and execute workflows. It is responsible for coordinating the execution of jobs defined in your GitLab CI/CD configuration file (.gitlab-ci.yml) on specific runners, whether they are shared runners provided by GitLab or your own custom runners.

Essential things that you must know about a GitLab runner:

Executor: It is a runtime that is responsible for running jobs. GitLab Runner supports various executors, such as Docker, Kubernetes, Shell, and more.

You can use the shared GitLab, but if you think your project requires:

1. Specific software configurations and dependencies or libraries to be handled that are not available on shared runners.
2. An isolated execution environment for the CI/CD jobs.
3. security and compliance requirements to ensure everything runs on the organization’s infrastructure.
4. Optimized performance in case of heavy execution load required to run extensive test suites or resource draining software builds
5. any integrations with api or external systems that could not be accessed from the external runner.

Pre-requisite :

1. Create a Virtual Machine in AWS , Here I have created an Ubuntu based VM. Install docker in the Ubuntu machine by following the official doc.
2. Access to Gitlab project/group to create the runner and also to create PAT to configure access for the runners.

Workflow:

In the Gitlab, create a runner.

In Project-> Settings -> CI/CD -> Runners, Select a new runner.

Select the platform and tags based on your requirements.

These tags would be used in the .gitlab-ci.yml to ensure the jobs are run in the respective GitLab runners only.

Once you create the runner, you would get the details of the token that you need to register the runner in the virtual machine.

In your VM,

Install the Gitlab Runner if it is Linux.

# Download the binary for your system
sudo curl -L - output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
# Give it permission to execute
sudo chmod +x /usr/local/bin/gitlab-runner
# Create a GitLab Runner user
sudo useradd - comment 'GitLab Runner' - create-home gitlab-runner - shell /bin/bash
# Install and run as a service
sudo gitlab-runner install - user=gitlab-runner - working-directory=/home/gitlab-runner
sudo gitlab-runner start

Select the installation scripts based on the host (linux/macos/windows).

Run the commands to check if the runner is installed properly and also to check the status.

gitlab-runner --version
gitlab-runner status

Sample screenshots for reference:

Register the runner by copying the commands from the gitlab ‘Register Runner’ screen to the VM

gitlab-runner register - url <GITLAB URL> - token <TOKEN>

It would ask for :

Gitlab URL, Runner name , Executor and default Docker Image if it is a docker executor.

If successfully registered, you would get an output like,

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Run the gitlab-runner using the command ,

gitlab-runner run

Go to the Runner page in Gitlab, you must see a runner online.

Change the behavior of Gitlab Runner

The configuration of the runner is stored in the host VM at ./etc/gitlab-runner/config.toml . You can change the add additional configurations to this file based on your project requirements.

Follow the official document for guidance.

If the PAT is not configured in the Gitlab, then you may face a similar issue as shown below while you run the pipelines.

Create a PAT for the project with access to registry and using the token login to the registr.gitlab.com with username and PAT

docker login registry.gitlab.com

Once the host has the appropriate credentials in the /root/.docker/config.json, then the Docker executor would be able to pull the default images that you mentioned for the docker runner, when it is initialised to run the CI/CD job.

If the images are pulled from internal registry, the respective registry credentials must also be added to the docker config and restart the docker service.

default:
  tags: ['test-linux']

stages:
  - test

test-job:
  image: python:3.10-slim-buster
  stage: test
  script:
    - echo "this is test"
    - apt-get update
  rules:
    - if: $CI_COMMIT_BRANCH == 'runner-test'

Above is a sample code to check the runner connectivity and executors. If the runner was properly configured, then you would see your pipeline execute the stages successfully.

Docker commands inside Gitlab Runner — Docker in Docker

If you want to execute docker commands in the GitLab, you need to use ‘Docker in Docker’ concept. Quick glance to a sample code

default:
  tags: ['docker-run']

stages:
  - test

test-job:
  image: docker:20.10.16
  stage: test
  services:
    - docker:20.10.16-dind
  script:
    - echo $CI_REGISTRY
    - echo $CI_REGISTRY_USER
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - echo "this is test for docker"
    - docker images
  rules:
    - if: $CI_COMMIT_BRANCH == 'runner-test'

To run this pipeline,

1. I created a gitlab runner with a docker base image.
2. In the pipeline, using a docker image then we use services as docker:20.10.16-dind

I added few configs :

Changed privileged to true; added to volumes — /var/run/docker.sock mounted to the host path.

[[runners]]
  name = "docker-run"
  url = "https://gitlab.com/"
  id = 31561533
  token = "<token>"
  token_obtained_at = 2024-01-14T17:08:19Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "<any docker image>"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
    network_mtu = 0

Ensure in the host the docker daemon is up and running .

Job got succeeded

Without mounting the /var/run/docker.sock , it was unable to identify the docker daemon, though I added DOCKER_HOST as variable. I would be trying a similar docker-in-docker with the GitLab runner configured in a Kubernetes cluster where dockershim is deprecated and see how to execute such commands . “docker-in-any-compatible-CRI-runtime executed from the hosted Gitlab runner”.

Thanks for reading.