Setting up your own fully functional GitLab: HTTPS, Registry, CI + Runners
When you’re tired of GitLab.com’s errors and lags, the obvious solution will be to set up a personal server with GitLab and stop suffering. Although installation of GitLab is straightforward, the trick is that if you want fully functional GitLab, you had to go through a lot of configuration. This article should help you to go through that process without catching all the problems I caught.
What does “fully functional” mean
First, let’s define what “fully functional” means to us. I’ll be talking about GitLab CE for personal usage. I.e. I have no special requirements in terms of performance, I just want to have all the major functionality I have on GitLab.com out of the box:
- Web UI
- Git push/pull
- Import from GitLab.com
- Domain
- HTTPS
- Registry
- CI
- Runners
- Pages
- Autoscaled runners
- Email notifications
With a fresh instance of GitLab, you will have only web UI and ability to do git push and git pull. GitLab.com import configuration is described in the separate article.
I still can’t get GitLab Pages to work, and I am not satisfied with how autoscaled runners behave, so this article will be about HTTPS, built-in Docker container registry, and basic CI configuration.
This is not a guide by any means. I described a lot of mistakes I made here. Maybe this will help you to avoid them. It is a description of a path I went through while I was setting up certain pieces of GitLab. I think it was worth writing since GitLab is reading this blog and hopefully will make some improvements to make the flow less painful
The reason
I was experimenting with Docker images for deployment. My internet connection has low upload speed, so I decided to do that in the cloud.
I followed this instruction from Digital Ocean, but when I tried to push an image to GitLab.com registry I have got “remote error: tls: bad record MAC”.
I found five months old issue, giving no clues on how to fix that. This was the last straw for me, so I decided that I should move development process to my GitLab completely.
Container Registry
On a freshly installed GitLab you will not find Registry tab in projects:
My next intention was to find in the admin area, but there was nothing. Project settings also give no clue that this feature even exists.
Ok, it’s googling time. Found this: https://docs.gitlab.com/ce/administration/container_registry.html
It was pretty hard to find the most important thing on a 12-screen documentation page:
It looks like my GitLab should have a domain name at least.
Domain name configuration
Previously, my GitLab had no domain name, and I was fine with it. I manage DNS records with Amazon Route 53, so I added A record with key “gitlab.domain.name” and IP address as value, and added the following line to config.rb:
external_url ‘https://gitlab.domain.name'Reconfiguring and restarting:
$ gitlab-ctl reconfigure && gitlab-ctl restartDone!
Domain for container registry
There’re two options in the documentation: use the existing GitLab domain or use a completely separate domain.
Also there’s a hint: “Since the container Registry requires a TLS certificate, in the end it all boils down to how easy or pricey is to get a new one”
But no hints which way is better if you don’t care about the price, but do care about the easiness of configuration.
Anyhow, I concluded that I have to make my GitLab work through SSL, so I started to look for a way to achieve that.
SSL configuration
I was always trying to avoid this topic as long as possible. At this point it was no longer unavoidable. The first thing I found was AWS certificate manager. The good thing is that Amazon provides certificates for free. The bad thing is that they work only within Amazon infrastructure. I realized that too late, so I went through the whole cycle of domain ownership confirmation, which included Zoho mail configuration since I had no configured email system for my domain.
After I realized the limitation of Amazon, the options were, to get paid certificate somewhere or try Letsencrypt. I knew it existed from GitLab’s blog posts but never tried it.
I found another 15-step tutorial from Digital Ocean:
And of course, I ran into troubles there as well :\
Letsencrypt with GitLab
After I added the following line to GitLab config, GitLab stopped to work:
nginx[‘custom_gitlab_server_config’] = “location ^~ /.well-known { root /var/www/letsencrypt; }”…and the following instructions didn’t work as well:
$ gitlab-ctl status
down: nginx: 1s, normally up, want up; run: log: (pid 13187) 2082974sFound this: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/214
Yay, I learned new command:
$ gitlab-ctl tail nginx==> /var/log/gitlab/nginx/error.log <==
2017/03/03 17:11:00 [emerg] 17138#0: BIO_new_file(“/etc/gitlab/ssl/registry.gitlab.domain.name.crt”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/gitlab/ssl/registry.gitlab.domain.name.crt’,’r’) error:2006D080:BIO routines:BIO_new_file:no such file)
It looks it already expects certificate, but it is not there yet. Realized that I uncommented some ssl related lines in config.rb earlier than needed. Commented it out.
Also found another link:
Trying to launch it again:
Again, realized, that I put another line in config too early:
registry_external_url ‘https://registry.gitlab.domain.name:4567'Removed, but kept this:
nginx[‘custom_gitlab_server_config’] = “location ^~ /.well-known { root /var/www/letsencrypt; }”Ugh, finally it worked:
Next:
external_url ‘https://gitlab.domain.name'nginx[‘redirect_http_to_https’] = true. . .nginx[‘ssl_certificate’] = “/etc/letsencrypt/live/gitlab.domain.name/fullchain.pem”nginx[‘ssl_certificate_key’] = “/etc/letsencrypt/live/gitlab.domain.name/privkey.pem”
Reconfigure & restart. Doesn’t work
$ gitlab-ctl status==> /var/log/gitlab/nginx/error.log <==2017/03/03 17:32:33 [emerg] 21173#0: BIO_new_file(“/etc/gitlab/ssl/registry.gitlab.domain.name.crt”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/gitlab/ssl/registry.gitlab.domain.name.crt’,’r’) error:2006D080:BIO routines:BIO_new_file:no such file)
Then, I realized that I have to have two sets of certificates for two domain names. So I needed certificates for the second domain as well:
Ah, of course, I haven’t set up DNS record for the second domain name. The good news is that GitLab started to work through HTTPS.
So, three things left:
- DNS for registry.gitlab.domain.name
- registry configuration
- certificates auto renewal
Ok, I configured second DNS record and modified config.rb accordingly. Then, I remembered that second domain name is not necessary.
Changing registry_external_url:
registry_external_url ‘https://gitlab.domain.name:4567'Reconfigure & restart. Doesn’t work at all:
$ gitlab-ctl status==> /var/log/gitlab/nginx/error.log <==22017/03/03 17:53:20 [emerg] 25738#0: BIO_new_file(“/etc/gitlab/ssl/gitlab.domain.name.crt”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/gitlab/ssl/gitlab.domain.name.crt’,’r’) error:2006D080:BIO routines:BIO_new_file:no such file)
Ugh, felt tired at this point. Decided to switch to auto renewal configuration.
Certificates auto renewal
crontab -eNo errors. Looks like a quick win.
Certificates for container registry
Ok, back to our main monster.
When I comment out registry_external_url, it works again. And I have this for ssl certificates in config.rb:
nginx[‘ssl_certificate’] = “/etc/letsencrypt/live/gitlab.domain.name/fullchain.pem”nginx[‘ssl_certificate_key’] = “/etc/letsencrypt/live/gitlab.domain.name/privkey.pem”
But the error message said that it looked for certificates at /etc/gitlab/ssl/gitlab.domain.name.crt
It looks like registry can’t take certificates from the same place as GitLab itself, so we need to put them there.
We can simply copy them, but let’s create symlinks instead, so that new certificates will be picked automatically after auto-renewal:
root@scw-e3ab26:/etc/letsencrypt/live/gitlab.domain.name# ln -s /etc/letsencrypt/live/gitlab.domain.name/fullchain.pem /etc/gitlab/ssl/gitlab.domain.name.crtroot@scw-e3ab26:/etc/letsencrypt/live/gitlab.domain.name# ln -s /etc/letsencrypt/live/gitlab.domain.name/privkey.pem /etc/gitlab/ssl/gitlab.domain.name.key
Enabled registry_external_url ‘https://gitlab.domain.name:4567’
Reconfigure & restart. It works! I have my registry tab:
I can push docker images to the built-in registry! Yay!
The funny thing is that it is useless without CI :)
I have a separate story about how I tried to get autoscaled workers to work fine for me, but it will go as a separate article.
The most stupid way to get CI running is to set up a runner on your own laptop.
Setting up local GitLab CI Runner
Yep, it is the most straightforward way to get CI running.
I’ve followed the docs, and everything worked as expected:
Created docker image, pushed it to my GitLab, created test project with following .gitlab-cy.yml:
image: gitlab.domain.name:4567/user/repo.at
test:
script: ls -lEverything worked like a charm! Finally!
Summary
GitLab has a goal to become “opinionated and integrated set of tools based on convention over configuration that offers superior user experience”. I think GitLab is good at being “opinionated and integrated”, but for GitLab administrator it is still a lot “configuration instead of conventions”. Of course, there should be a certain flexibility for those who have existing infrastructure to integrate GitLab with, but I don’t see how it contradicts with the idea that newcomer should be able to get working GitLab without too much suffering.
GitLab assumes that you deal with server configuration on a daily basis, and understand what are you doing during the configuration process. In my situation it was not the case, so I’ve got a collection of all kinds of errors, and had not a very pleasant experience.
So I guess it doesn’t count as developers-friendly. Administrators-friendly at max. The separate thing is the documentation.
Documentation
After this challenge I was finally able to formulate my complaint to GitLab’s documentation: Yes, it is usually very comprehensive, it describes a lot of different scenarios, but when you try to solve your specific problem, it is not very helpful — there’s too much noise around, and you spend too much time trying to figure out what is relevant to your situation and what is not:
Compare this to guides at GoRails. All the options are just drop-downs, and irrelevant things don’t pollute the page:
