Combell shared hosting: Autogit

Combell is kind enough to offer me this A++ hosting for free and was my employer for many years, so the least I could do in return is spread the word about their hosting solutions. Today we zoom in on this little less known feature of their Linux hosting offering. It enables me to manage this little blog with minimal effort.

I should also note that I am the main implementor of the first versions of this nifty feature. From the looks of it, it didn’t change much since I left.

Combell is mainly targeted at the hugely popular PHP community but it is obviously also possible to host plain static content. And moreover, the Autogit automation allows you to use a static file generator such as Hugo to generate HTML, CSS, javascript, merely based on some Markdown formatted text files, the only thing your really need to manage. And static assets such as images of course.

Combell has some nice examples hosted on Github that illustrate the features of Autogit and how you can manage your favourite PHP project using Autogit: https://github.com/combell

SSH jail

At the base of Autogit is a so called ‘SSH jail’. SSH really should be the way how you interact with your hosting, at any hosting company for that matter. It allows you to securely transfer your files using scp or sftp to your website storage. FTP is to be avoided, as should be obvious. The jail aspect means that you are shielded from other hosting accounts using container technologies. A long time ago, I was the main implementor of the first SSH jail versions. It has hugely evolved since, mainly thanks to the uprise of container technologies since Docker became a thing.

It also enables a lot of extra cool features mainly targeted to the PHP community. Autogit is the one we leverage here. There is a so called empty bare git repository provisioned (when enabled in the control panel) that you can securely access using the SSH protocol. I should also note here that the Combell control panel allows you to setup SSH Keys enabling secure access to your jail.

Using a simple YAML config file you can automate stuff when you push new content to the Autogit git repository using so called git hooks.

Hugo

Hugo (https://gohugo.io/ ) is a popular way to create very fast modern looking static websites. When I want to create a project I usually google something like ‘best themes hugo’ and start from the sample website the great theme creators usually provide. At the time of writing I picked https://github.com/adityatelange/hugo-PaperMod/tree/exampleSite

You usually only need to run the hugo binary without extra parameters to generate your static content, after adding/updating some Markdown (.md) formatted text files.

Before you publish, also open the config file and customize it to your liking. Look for config.toml or config.yml in the project root.

Setup

Extra binaries

Combell does not provide the hugo binary (yet). Also there is no Go support as far as I know. I noticed Hugo needs (or needed?) Go for some features, so I also install it. And also to show you that it is possible, even if the host did not provide it for you.

Luckily it is pretty easy to install dependencies.

Just download the latest ‘Linux-64bit’ of Hugo from https://github.com/gohugoio/hugo/releases and drop it in the home of your jail.

I installed version 0.98.0 using this command:

$ wget https://github.com/gohugoio/hugo/releases/download/v0.98.0/hugo_extended_0.98.0_Linux-64bit.tar.gz

Note that we chose the ’extended’ version. Some themes require functionality not available in the non-extended version.

The tar needs to be extracted:

$ tar xvzf hugo_extended_0.98.0_Linux-64bit.tar.gz
$ rm hugo_extended_0.98.0_Linux-64bit.tar.gz

For Go, just head to https://go.dev/dl/ and download the latest linux tar and extract it, again in your home. You can remove folders we don’t need such as ‘src’ and ’test’.

A fair warning here, although Combell goes to great lengths to accomodate your needs, custom binaries are a bit in the grey zone of what is supported in a shared hosting product. I don’t know their exact policy nowadays, but if you can’t seem to get your binary working, it might have triggered some “defense” mechanism. In that case just contact their very responsive support and you probably will be back on your way in no time.

Autogit config

Let’s study the Autogit config file, .autogit.yml:

shared_folders: [ maarten.gent/themes ]

hooks:
# SETUP: Create folder structure for newest release
  setup_before: |
    # Download themes when not done yet
    if [ -z "$(ls -A /site/checkout/master/shared/maarten.gent/themes)" ]; then 
      git clone /autogit/
      cd autogit/
      git submodule update --init --recursive

      mv maarten.gent/themes/* /site/checkout/master/shared/maarten.gent/themes/
      rm -rf /site/checkout/master/shared/maarten.gent/themes/*/.git

      cd ..
      rm -rf autogit/
    fi    
  setup_after: |
        exit 0

# INSTALL: Put code in release folder
  install_before: |
        exit 0
  install_after: |
        exit 0


# SHAREDSYMLINK: Create symlink to shared files and folders
#                present at every release (config, logs, ...)
  sharedsymlink_before: |
        rm -rf maarten.gent/themes/
  sharedsymlink_after: | 
    export GOPATH="~/go"
    export PATH="~:$GOPATH/bin:~:$PATH"
    cd maarten.gent/
    hugo

In the config you can specify script snippets specifying what you want to do before or after different phases of the deploy process. We hook in on the setup_before, sharedsymlink_before and sharedsymlink_after opportunities.

We also specified some ‘shared folders’.

shared_folders feature

This allows us to specify some folders that are common to all deployments and that we typically not want to store in git. Autogit will symlink this folders automatically at every deploy.

Hugo themes are not managed in this git project. We use git submodules to refer to the author’s repository, as explained in next section.

Shared folders usually are set up manually once. As alternative we choose to use the setup_before hook to check if the initialization must be done (empty folder) and download the theme(s) only if needed. This way part of this initialization is cleanly stored in git and automatically applied when needed.

If at one point in time we want to update our theme, i.e. we updated the submodule with another commit id, we can delete the contents of the folder /site/checkout/master/shared/maarten.gent/themes and our hook implementation will download the updated theme at the next git push.

For more information about this feature I refer you again to the excellent Autogit documentation on Github: https://github.com/combell/autogit-reference#how-to-manage-shared-resources

setup_before hook

This is the first possible hook in the process and we leverage it do initialize stuff, if needed.

The bash snippet is a little necessary ‘hack’ because we didn’t commit the Hugo themes to git but make use of so called git submodules. This means that we keep a reference to the git repository and a specific commit (usually HEAD in master at setup time). Since Autogit (feature request) does not process git submodules, we need to do it ourself.

This is a time and bandwidth consuming action, so we make use of the ‘shared folders’ feature, as explained, and we check if the shared folder is empty. If not, we assume the theme is already downloaded and give control immediately back to Autogit.

But when we need to initialize our theme, we perform a new checkout of the Autogit repository, after which we can checkout the submodules (git submodule update). You should know (or learn here) that the bare git repository is located at the folder /autogit in the SSH jail.

Finally we move the downloaded theme(s) to the shared_folder location so Autogit can correctly link it to our release later. We don’t need the .git folder so we also get rid of it.

sharedsymlink_before hook

In this phase we make sure the shared_folderssymlinks can be setup correctly: the (mostly empty) themes/ folder is removed.

sharedsymlink_after hook

This is where the real magic happens. In this phase we have all the files and themes where Hugo expects them.

We setup the environment variables $GOPATH and $PATH so Go can be executed as if it is installed system wide. The home folder (~) is also added to $PATH, also enabling us to execute hugo without specifying its path.

Next we navigate to our project folder maarten.gent/ and simply execute hugo. This will generate all the static assets needed to render this little blog. Hugo puts these in the public/ folder of the Hugo project.

Git structure

The only missing piece of the puzzle to do this yourself is the git project structure.

Usually you start with checking out the empty repository after enabling the Autogit feature in your control panel. Autogit needs at least two things to function, committed to git: the already mentioned .autogit.yml config file and a www/ folder or symlink.

The www folder (or symlink) should contain your index.html. In our case this should point to the automatically generated public folder, that we have triggered in our sharedsymlink_after hook by running hugo.

We generate the www symlink as follows in the root of our git repository:

$ ln -sf /site/checkout/master/current/maarten.gent/public www

Next to this your git project should obviously contain your Hugo project. We opted to store this in the folder maarten.gent/ but this could have been the git root just as well.

Autogit magic at work

Autogit generates some output that allows you to follow what happens when you push something to your hosting. As you can see in the output, Hugo generates content very fast and together with the efficient Autogit deploy, this leads to near instant deployments.

Please note that it looks a bit more complicated than explained above because I have an extra little Hugo project to host my CV. The setup_before hook generates no output and is only an if test statement always resulting to false because the Hugo themes are already initialized before.

**************************************************************
*                                                            *
*                                                            *
*               Unauthorized access to this                  *
*               system/network is prohibited.                *
*                                                            *
*                                                            *
**************************************************************
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 1.60 KiB | 1.60 MiB/s, done.
Total 7 (delta 3), reused 0 (delta 0), pack-reused 0
remote: production branch 'master' ref received.
remote: Parsing config '.autogit/config':
remote: domain: maarten.gent
remote: production: master
remote: Updated 1 path from adc5bdb
------> Executing BEFORE SETUP hook..
        Checking out in /data/sites/web/maartengent1807/checkout/master/48ec229ce5b16d4b381d357ad39d3f6b752c222b...
------> Executing AFTER SETUP hook..
------> Executing BEFORE INSTALL hook..
remote: HEAD is now at 48ec229 autogit v2
------> Executing AFTER INSTALL hook..
------> Executing BEFORE SHAREDSYMLINK hook..
        Creating shared folder link maarten.gent/themes...
        Creating shared folder link cv/themes...
------> Executing AFTER SHAREDSYMLINK hook..
remote: Start building sites …
remote: hugo v0.98.0-165d299cde259c8b801abadc6d3405a229e449f6+extended linux/amd64 BuildDate=2022-04-28T10:23:30Z VendorInfo=gohugoio
remote:
remote:                    | EN
remote: -------------------+-----
remote:   Pages            | 35
remote:   Paginator pages  |  0
remote:   Non-page files   |  0
remote:   Static files     |  0
remote:   Processed images |  0
remote:   Aliases          | 11
remote:   Sitemaps         |  1
remote:   Cleaned          |  0
remote:
remote: Total in 322 ms
remote: Start building sites …
remote: hugo v0.98.0-165d299cde259c8b801abadc6d3405a229e449f6+extended linux/amd64 BuildDate=2022-04-28T10:23:30Z VendorInfo=gohugoio
remote:
remote:                    | EN
remote: -------------------+-----
remote:   Pages            |  4
remote:   Paginator pages  |  0
remote:   Non-page files   |  0
remote:   Static files     | 98
remote:   Processed images |  0
remote:   Aliases          |  0
remote:   Sitemaps         |  1
remote:   Cleaned          |  0
remote:
remote: Total in 335 ms
------> Executing BEFORE SYMLINK hook..
        Reloading PHP... "OK"
------> Executing AFTER SYMLINK hook..
------> All done. Please visit your fresh deploy: http://maarten.gent
------> Executing BEFORE CLEANUP hook..
        Cleaning up release fdff1feb74b689bfe4126b2fc70f2a4e993d4db6...
------> Executing AFTER CLEANUP hook..
To ssh057.webhosting.be:auto.git
   9a1d08f..48ec229  master -> master

SSH jail#

Hugo#

Setup#

Extra binaries#

Autogit config#

shared_folders feature#

setup_before hook#

sharedsymlink_before hook#

sharedsymlink_after hook#

Git structure#

Autogit magic at work#