Cgit, Nginx & Gitolite: A Personal Git Server

Admin User

For whatever reason, be it for security or protecting the server from my stupidity, one of the first things I do when creating a new server is add a new user for my general admin tasks.

Adding a new user is remarkably easy to do on a Ubuntu system:

$ adduser limb

You'll be prompted to answer a few questions, including creating a new UNIX password. This will be the password you'll need to sudo -i and gain root permissions, so make it a good one, or use tools like Pass, or BitWarden to help you remember.

Then give our new limb user sudo permissions:

$ usermod -G sudo limb

I've also largely eliminated all password based authentication when signing into servers, relying on open source smart cards like NitroKey for authentication. If interested, this requires we setup .ssh/authorized_keys for our limb user:

Just replace key with your public ssh key:

$ mkdir /home/limb/.ssh
$ echo "key" > /home/limb/.ssh/authorized_keys

Next, set the .ssh directory's file permissions so the ssh daemon can read the files:

$ chown -R limb:limb /home/limb/.ssh
$ chmod 700 /home/limb/.ssh
$ chmod 644 /home/limb/.ssh/authorized_keys

If everything worked, after you restart the ssh daemon (service sshd restart) you will now be able to login as the administrator user:

$ ssh limb@host

OpenSSH

Git and Gitolite (installed in the next sections) will need us to keep port 22 open, allowing us to git push from anywhere on the internet. This open port will eventually attract "a lot" of attention from bots who endlessly scour the internet looking for vulnerable servers, mindlessly stuffing passwords, hoping one password will eventually let them in.

We can eliminate all worry about weak or compromised passwords by disabling all password based authentication, relying solely on asymmetric cryptography, or "ssh keys". Just use your favorite text editor to open /etc/ssh/sshd_config and ensure these lines exist somewhere in it:

PubkeyAuthentication yes
PasswordAuthentication no

While we're here, a large majority [1] of these bots are interested in logging in as the root user. If you created a new admin account in the previous section and ensured you can login using your public key, you can also disable root logins entirely with this line in the config:

PermitRootLogin no

If you uploaded your public key to your VPS provider, most of these changes should have already been configured for you. But in the off chance you had to make some changes, restart the ssh service to load the new config changes in:

$ service ssh restart

Uncomplicated FireWall

Depending on your VPS provider, they may also have a firewall system built into their admin panel allowing you to apply rules simply by adding tags to a server. However, I enjoy keeping all my firewall rules inside each box, if only for the same reason I keep all my socks on the left hand drawer, so everything stays organized and in the same place.

You can install ufw using the Advanced Packaging Tool:

$ apt install ufw

Right now, the only thing we have enabled is ssh which uses port 22. To allow port 22 through ufw just use the following command:

$ ufw allow ssh

and then turn the firewall on:

$ ufw enable

and viola! You have a firewall.

fail2ban

Even though we've turned off password based authentication in a previous section, we will still receive a significant amount of bots wasting our compute cycles trying to login. And while the likelihood of this being successful is zero when rounded to any order of magnitude, the bots will nevertheless continue to pilfer a non-zero amount of CPU if given the opportunity.

To stop the most brazen of these bots, tools like Fail2Ban, which creates temporary firewall rules to block IP address who repeatedly fail to authenticate with ssh, are a great compromise between usefulness and annoyance.

The Advanced Packaging Tool can again help us install fail2ban.

$ apt install fail2ban

Once installed, the ssh "jail" will come pre-enabled for you. If you wish to make any changes, you will need to make a copy of the fail2ban config file:

$ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Then add your changes to jail.local so they will persist after an upgrade.

fail2ban does a great job documenting what each option does in the config file. Some of the changes I made are:

• because I use ufw to manage my firewall, I changed banaction = ufw.
• enabled bantime.increment to increase the duration of a ban based on how many times the IP address has been banned previously.
• enabled bantime.rndtime to "randomize" the length of a ban, preventing bots from knowing exactly when they can resume their assault.
• enabled bantime.maxtime so I won't need to unban IP addresses (if you're unfortunate enough to share an IP with a bot).
• lowered bantime, findtime and maxretry allowing me to issue small bans that increase in severity as the IP address continues to antagonize.

Once you're satisfied with your changes, start fail2ban using systemd.

$ systemctl enable fail2ban
$ systemctl start fail2ban

And Done!

Depending on how "popular" you are on the internet, you should start to see NOTICE lines in /var/log/fail2ban.log of misbehaving bots and the equivalent firewall rules in ufw.

$ cat /var/log/fail2ban.log | grep 'NOTICE' | tail -1
...  [125553]: NOTICE  [sshd] Ban 156.155.159.161

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   REJECT      156.155.159.161
22/tcp                     ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)

Step: 1 - Create The Git User

Before we install Gitolite, we'll need to create a new user for everyone to log into and to run Gitolite's Perl scripts. I typically use the username git for this, feel free to replace git with the username that you feel is more appropriate.

$ adduser --system --group --disabled-password --home /var/lib/git git

This creates a new system-user on the server called git. Because this is not a "normal" user, there will be no aging information in /etc/shadow, which is convenient when nobody will be monitoring this account.

We also used the --home option to set the $HOME variable to /var/lib/git. This is where we will eventually put Gitolite's configuration files and our Git repositories. Feel free to adjust this to where you prefer, I've seen many use /home/git.

I included the --disabled-password to disable any password based access into our new user. In the previous sections, we've disabled all password based authentication into the server and Gitolite requires ssh keys for authentication, so disabling passwords for our user is a smart move.

Step: 2 - Install Gitolite

Because Gitolite is just a bunch of Perl scripts, I prefer to install Gitolite from the source. As we will see, installing Gitolite from source also has the benefit of making upgrades and adding custom patches in the future extremely easy.

This also means we'll need to install Gitolite's dependencies ourselves:

$ apt install perl git

When we (or a colleague) signs into the server, using the git user, we will automatically run Gitolite's Perl scripts, which means these scripts must be executable by our git user. So, to make managing file permissions easier, we'll use our git user for the rest of the installation process.

Log into our git user with the "substitute user" command: (assuming you're the root user)

$ su - git

Then clone Gitolite's source code into the $HOME directory: (this should be /var/lib/git unless you changed it when we setup the git user above)

$ git clone https://github.com/sitaramc/gitolite

Step: 3 - Setup Gitolite

To setup Gitolite on the server, we'll need to assign Gitolite an admin that will have full control over editing Gitolite's configuration repository. This will most likely be you.

Still as the git user, use your favorite text editor to create a new file with your desired username in the $HOME directory and copy your public ssh key into it. For example, my file would be called bryanbrattlof.pub and look like this:

$ cat $HOME/bryanbrattlof.pub
ssh-ed25519 AAAAC3NzaC1l ...

Then run Gitolite's sanity checks and setup script to finish the installation:

$ $HOME/gitolite setup -pk bryanbrattlof.pub

Färdig!

If everything went well, you should now be able to clone Gitolite's configuration repository:

$ git clone git@host:gitolite-admin

I recommend consulting Gitolite's incredible documentation to understand how to properly configure access and add hooks to all of your projects.

Step: 4 - Add ~/.profile

While not technically needed for Gitolite to function properly, I find adding gitolite to our git user's $PATH is a great quality of life improvement on the rare days I need to play system administrator.

First, as our git user, create a new $HOME/bin directory:

$ mkdir -p $HOME/bin

Next, create the $HOME/.profile file and add the $HOME/bin directory to our $PATH.

PATH=$HOME/bin:$PATH

The born again shell will automatically run $HOME/.profile when someone starts a new session.

Now, we can use Gitolite's install script to add a symbolic link inside our $HOME/bin folder:

$ gitolite/install -ln $HOME/bin

Done!

Logout and back in to the git user (or use source $HOME/.profile) to pick up the changes. If everything was done correctly, you won't need to type the full path to Gitolite anymore.

$ whereis gitolite
gitolite: /var/lib/git/bin/gitolite

Step: 1 - Install Cgit

Just like with Gitolite, I prefer to install Cgit from source so I can add personal patches and quickly change what version is running on the server. This also means we'll need to install the dependencies ourselves:

$ apt install libc6 liblua5.1-0 zlib1g \
  python3-docutils python3-markdown python3-pygments

We'll also need the build-essential packages to install the gcc and make tools needed to compile Cgit after we've cloned the project:

$ apt install build-essential

Then, as the root user in the /root directory clone the Cgit project:

$ git clone https://git.zx2c4.com/cgit

Because Cgit uses parts of Git's source code, (included as a submodule) we'll need to use git submodule to download the remaining code from the Git project.

After you cd into cgit

$ git submodule init    # register the git submodule in .git/config
$ git submodule update  # clone/fetch and checkout correct git version

Step: 2 - Build Cgit

With a full copy of Cgit on the server, we can now create some patches to customize it for our use-case. We'll start with creating cgit.conf inside the cgit project we just cloned, to tell make where we want to install the Cgit binaries.

CGIT_SCRIPT_PATH = /var/www/html/cgit/cgi
CGIT_CONFIG = /var/www/html/cgit/cgitrc
CACHE_ROOT = /var/www/html/cgit/cache
prefix = /var/www/html/cgit
libdir = $(prefix)
filterdir = $(libdir)/filters

Because this is a version controlled project, we can commit our changes to save our work:

$ git add -f cgit.conf
$ git commit -m "installation path changes"

Some additional changes I made:

• Updated the cgit.png and favicon.ico icons
• Changed the pygments highlighting style to "algol_nu"
• Removed the Data URI icons from the tab menu
• Limited the max-width of readme pages to 95ch
• Added padding: 1em; to code-blocks

When you're satisfied with your changes, use make to compile and install Cgit:

$ make && make install

If everything went well, when you execute Cgit from the terminal, a web-page should print out:

$ ./cgit
Content-Type: text/html; charset=UTF-8
Last-Modified: Tue, 12 Jan 2021 22:35:43 GMT
Expires: Tue, 12 Jan 2021 22:40:43 GMT

<!DOCTYPE html>

And Done!

We can further customize Cgit's behavior using the cgitrc file located at /var/www/html/cgit/cgitrc. Feel free to check out the man page for a complete description of what every option does.

Some of the options I used:

• set scan-path to the location of our repositories /var/lib/git/repositories
• set project-list to the location of the projects.list file Gitolite creates, adding descriptions and categories to the list of repositories on Cgit's index page