Over-engineering with t2.nano’s

I switched job recently and in my new position it’s unlikely that I can use AWS. To prevent a cold turkey from the AWS environment, I moved my personal website and blog (with < 2 visitors a month, myself included) to AWS. This allows me to further play around with BOTO3 and the tools AWS provides for somewhere around 20 euro’s a month. Hooray for the t2.nano instances. I am completely over-engineering the setup with docker, haproxy, nginx workers, jekyll and ansible. Getting it all to work smoothly is fun and it ensures me that I am not loosing my AWS-Mojo.

Encrypt all the things!

As a long time user and supporter of PGP, I strongly encourage everyone to use encryption by default. Thanks to the Let’s Encrypt project, everyone now has the opportunity to (and in my humble opinion: should) enable HTTPS on webservers without any additional costs for buying a valid certificate or the downsides of self-signed certificates.

Request the certificate

Using the docker container provided by Let’s encrypt and having port 443 available this is literally a two second job:

$  docker run -it --rm -p 443:443 --name letsencrypt \
  -v "/etc/letsencrypt:/etc/letsencrypt" \
  -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
  quay.io/letsencrypt/letsencrypt:latest auth \
  -d www.gargleblaster.org \
  -d blog.gargleblaster.org \
  -m [email protected]

  IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/www.gargleblaster.org/fullchain.pem. Your
   cert will expire on 2016-07-09. To obtain a new version of the
   certificate in the future, simply run Let's Encrypt again.
 - If you like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

HAProxy setup

The certificates are stored in /etc/letsencrypt, so you have to load this directory into your docker container. But before you can use the certificate, you will have to combine the key and certificate in a single .pem file.

The following script does this:

#!/bin/sh

set -e

[ -d "/etc/letsencrypt/live" ] && find /etc/letsencrypt/live -name 'privkey.pem' |
while read line; do
  KEY_FILE="$line"
  CERT_FILE="$(dirname $line)/fullchain.pem"
  HAPROXY_FILE="$(dirname $line)/haproxy.pem"
  DOMAIN_NAME="$(basename "$(dirname $line)")"
  cat "$CERT_FILE" "$KEY_FILE" > $HAPROXY_FILE
  mkdir -p /etc/letsencrypt/haproxy
  cp "$HAPROXY_FILE" "/etc/letsencrypt/haproxy/$DOMAIN_NAME.pem"
done

I use ansible to start docker containers, so the corresponding docker container stanza looks like:

    - name: start new Docker image
      docker:
        image: ":"
        name: ""
        hostname: "-container"
        volumes:
          - "/persistent_storage//config:/haproxy-override"
          - "/etc/letsencrypt/:/etc/letsencrypt/"
          - "/etc/letsencrypt/haproxy:/etc/haproxy/certs/"
        ports:
          - "80:80"
          - "443:443"
          - "8890:8890"
        state: running

The Haproxy configuration looks like:

frontend http-in
    bind    :443 ssl crt /etc/haproxy/certs/www.gargleblaster.org.pem
    bind    :80

    # letsencrypt certification requests/renewals
    acl url_acme_http01 path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET url_acme_http01

    # prepend domain name only
    acl starts_with_www hdr_beg(host) -i www
    acl starts_with_blog hdr_beg(host) -i blog
    http-request redirect code 301 location https://www.%[hdr(host)]%[capture.req.uri] unless starts_with_www OR starts_with_blog

    # redirect http to https
    redirect scheme https code 301 if !{ ssl_fc }

    # HSTS (15768000 seconds = 6 months)
    rspadd  Strict-Transport-Security:\ max-age=15768000

    use_backend www_container if { ssl_fc_sni www.gargleblaster.org }
    use_backend blog_container if { ssl_fc_sni blog.gargleblaster.org }

On port 80 all incoming requests which don’t use https are redirected to port 443. And requests without www. are redirected to www.gargleblaster.org.

Finally, you also need to instruct HAProxy to use the correct ciphers (for this moment, speaking about fast moving targets) by setting global options to:

    ssl-default-bind-options no-sslv3 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

    ssl-default-server-options no-sslv3 no-tls-tickets
    ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

Renewing Certificates

I use a custom rolled HAProxy container, where I have included the lua plugin which allows an easy method of handling the incoming acme requests needed for requesting and/or renewing certificates.

The command for renewing a certificate is:

docker exec -it haproxy certbot certonly -\-text \
-\-webroot -\-webroot-path /var/lib/haproxy \
-d www.gargleblaster.org \
-d blog.gargleblaster.org \
-\-renew-by-default
-\-agree-tos \
-\-email [email protected]

rock’n’roll