Go-Gettable Code

I've been using Go for a number of years now and have found it to be a real treat. It doesn't have the multi-threading and maintenance issues of Python. But it has vectors and associative arrays built in with a rich standard library. I hope to write more in the future about the benifits of Go.

Something that impressed me as soon as I started writing in Go was the powerful simplicity of the import mechanism. It is trivial to put code up on a site like GitLab as a library and then import it without any fussy package management. Go takes care of the details for you and makes vendoring easy. But I was not satisfied with this. I wanted to be able to use my domain as the namespace for my Go packages - because you own your own domain to flex after all. It turns out there is a little work on the back end to make all the magic happen under your own name.

You can skip all the way to the last section if you are going to host your code somewhere like GitLab. The next two sections are about self-hosting git, for style points.

Lets say you have a great idea for a Go package and want to publish it:

// Package hello is a dumb example package.
package hello

// Hello returns the string "Hello".
func Hello() string {
    return "Hello"
}
        

Once we publish it we can share the package with the world and compile our disruptive application:

package main

import (
    "fmt"

    "necheff.net/hello"
)

func main() {
    fmt.Println(hello.Hello() + ", World!")
}
        

The go utility will automatically fetch the necheff.net/hello package for compilation if it isn't already cached. But how do we make the hello package available in the necheff.net namespace for Go to automatically import?

Hosting git Yourself (Optional)

First, we need to create a git repository. On my server I ran the following commands to set up a repo. Do note that this is an optional step, if you want to host on a service like GitHub or GitLab that will work too. If you do that, skip ahead to the nginx setup.

~$ su root
~$ adduser --shell /usr/bin/git-shell --home /srv/git --disabled-password git
        

The adduser command is a Debian script wrapped around the useradd command, it takes care of a lot of things for us like making a git group as the git user's primary group. Since we've disabled the git user's password as a security measure and used the git-shell as the login shell (a highly restricted shell that rejects interactive logins) we actually can't su to git and will need to run some setup commands as root and then chown files later.

~$ mkdir /srv/git/hello.git && cd /srv/git
~$ ln -s hello.git hello
~$ cd hello && git init --bare
~$ chown -R git:git /srv/git/hello.git /srv/git/hello
        

Ok, but now we need a way for users to access the git repo. Since we are already running SSH, it will be easy and relatively secure to tunnel git over SSH. Make sure to lock down the directory permissions on the .ssh/ directory, we probably won't be generating any SSH keys as the git user but it is a good habit to protect the directory contents from other users with shell access.

~$ mkdir /srv/git/.ssh
~$ chmod 0700 /srv/git/.ssh
~$ touch /srv/git/.ssh/authorized_keys
~$ chown -R git:git /srv/git/.ssh
        

Now all you need to do is append the public SSH key for users you want to allow read-write access to your git repos to the /srv/git/.ssh/authorized_keys file.

On our client laptop we can clone the repo and add hello.go:

~$ git clone git@necheff.net:/srv/git/hello.git
~$ cd hello
~$ vim hello.go
~$ git add hello.go
~$ git commit -m "initial commit of the super disruptive Go package!"
        

Now, we need to let Go know that this is a module.

~$ go mod init necheff.net/hello
~$ go mod tidy
        

This created a go.mod file for us which should be tracked in version control too and updated periodically.

~$ git add go.mod
~$ git commit -m "make the hello package part of the necheff.net module namespace"
        

Now, before we push, lets tag a release. This will come in handy later for when we want to vendor our package with Go.

~$ git tag -a v1.0.0 -m "tagging the initial release of the super disruptive Go package!"
~$ git push
~$ git push origin v1.0.0
        

Using nginx To Self-Host git Over HTTP (Optional)

Ok great. We have a working git repo with the code for the hello package and we even tagged a release. Now we can make our package go-gettable and share it with the world! We need to configure nginx, or some other HTTP server, to host some information that the go utility will use to fetch packages in our module namespace. First, a full on nginx configuration is probably a whole post itself. I already have nginx setup to host a few virtual sites off my server and I have certbot configured to keep my LetsEncrypt TLS certificates renewed. It is 2020, there is no excuse not to TLS your stuff - just do it. This guide will cover only what I did to add a new virtual site specifically for managing Go packages hosted under the necheff.net module namespace.

If you are familiar with Debian, you know they have their own little system for managing the Apache configuration. And as a kindness to Apache/Debian admins, they have gone and done something similar with nginx. So keep that in mind, if you are on another distro, the location of config files might look a little different.

First things first, if you hosted your own git repos we are going to run some CGI like it is 1995 - install fastcgi and we'll point nginx at it later.

~$ apt-get install fcgiwrap
        

We need to set up a virtual site for the "necheff.net", notice it isn't "www.necheff.net" nor "git.necheff.net" but just our domain and the TLD. That is because I want my Go module namespace to be "necheff.net". Don't forget to update your certbot config to include "necheff.net" in the valid domain list! Something we need to consider is how we are going to approach security. We don't just want any pleb to have write access to our git repo! If we are hosting our own git repos, we can use nginx filters to split the git protocol into "writes" and "reads", require authentication for "writes" and allow unauthenticated, anonymous "reads". If you chose to host at GitHub or GitLab, they already require authentication so ignore those bits.

First, add the www-data user to the git group. This is the user that nginx runs as, it'll need group membership to access our repos under /srv/git/

~$ gpasswd -a www-data git
        

Now, git comes with a CGI script called git-http-backend that is going to do most of the heavy lifting for us. We need to make some fastcgi configurations for it that we'll include in the nginx config file. Edit the /etc/nginx/git-http-backend.conf file and add the following:

fastcgi_pass unix:/var/run/fcgiwrap.socket;
include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
fastcgi_param GIT_HTTP_EXPORT_ALL "";
fastcgi_param GIT_PROJECT_ROOT /srv/git;
fastcgi_param PATH_INFO $1;
fastcgi_param REMOTE_USER $remote_user;
        

Edit /etc/nginx/sites-available/git-tls and add the following:

server {
    # tighten this down!!!
    # no SSLv3 or TLSv1.1 crap
    # in 5 years or so come back and get rid of TLSv1.2 also.
    ssl_protocols TLSv1.2 TLSv1.3;

    listen [::]:443;
    ssl on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/www.necheff.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/www.necheff.net/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    # make SURE you regenerate your Diffie-Hellman params with good entropy levels.
    # otherwise, the unscrupulous folks can crack your TLS session. See weakdh.org.
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    # notice the rootdir is /srv/ not /srv/git/ !!!
    # if I was smarter, I would have put the git home directory down a level so my web root
    # wasn't just /srv/ - I guess in 5 years when I need to put something else under /srv/ on this machine
    # I'll...figure something out
    root /srv/;
    access_log /var/log/nginx/git-access.log;
    error_log /var/log/nginx/git-error.log;

    index index.html;

    # note the server name is identical to the desired Go module namespace
    server_name necheff.net;

    location / {
        try_files $uri $uri/ =404;
    }

    # grab URLs that look like necheff.net/git/foo
    # i.e. have "git" in them
    # then rewrite them for later filters
    location ~ /git(/.*) {
        if ($arg_service = git-receive-pack) {
            rewrite /git(/.*) /git_write$1 last;
        }
        if ($uri ~ ^/git/.*/git-receive-pack$) {
            rewrite /git(/.*) /git_write$1 last;
        }

        if ($arg_service = git-upload-pack) {
            rewrite /git(/.*) /git_read$1 last;
        }
        if ($uri ~ ^/git/.*/git-upload-pack$) {
            rewrite /git(/.*) /git_read$1 last;
        }

        # stop plebs from browsing /srv/git/ at random
        return 404;
    }

    # just pass through git read requests
    location ~ /git_read(/.*) {
        include /etc/nginx/git-http-backend.conf;
    }

    # require authentication for git write requests...
    # we'll never add any accounts to .htpasswd!
    location ~ /git_write(/.*) {
        auth_basic "git push requires authentication...";
        auth_basic_user_file /etc/nginx/.htpasswd;
        include /etc/nginx/git-http-backend.conf;
    }
}

server {
    # it is 2020 - if a pleb tries to initiate an unencrypted connection on port 80,
    # redirect them to 443 like a boss - aka STARTTLS
    if ($host = necheff.net) {
        return 301 https://$host$request_uri;
    }

    listen 80 ;
    listen [::]:80 ;
    server_name  necheff.net;
    return 404; # RIP anyone that can't do TLS

}
        

Now, enable the new virtual site and if the nginx parser doesn't return any errors, restart the daemon.

~$ ln -s /etc/nginx/sites-available/git-tls /etc/nginx/sites-enabled/git-tls
~$ nginx -t
~$ systemctl restart nginx
        

Using nginx to Route Go Imports (Manditory)

Whew! Now, whether or not you are hosting your own git repo or using a third-party host, we need to add some HTML to help the go tool out. The go utility will first try to grab a file at https://necheff.net/hello and check for a specific meta tag. See how that is the module namespace slash-separated from the package name? Then, to sorta-kinda protect against malicious redirects it will look for an identical meta tag at https://necheff.net/

Ok, in the webroot, /src/ create a hello file with the following contents:

<!DOCTYPE html>
<html>
    <head>
        <meta name="go-import" content="necheff.net/hello git https://necheff.net/git/hello">
    </head>
</html>
        

The name attribute is always "go-import" but the content attribute is always "fully qualified import path" "mechanism" "repo location" in that order.

Now, copy the hello file to index.html. Over time, as new packages are added to the module namespace, each package will have its own named file in the webroot. But index.html will have a list of meta tags. For example, say we later host a "foobar3000" package off GitLab, there would be a foobar3000 HTML file but then index.html would look like:

<!DOCTYPE html>
<html>
    <head>
        <meta name="go-import" content="necheff.net/hello git https://necheff.net/git/hello">
        <meta name="go-import" content="necheff.net/foobar3000 git https://gitlab.com/yourusername/foobar3000">
    </head>
</html>
        

And that is it! Now, the hello is go-gettable, all you and your friends need to do is add import "necheff.net/hello" to their source files.