I needed to set-up a new website with HTTPS and so I took Let’s Encrypt procedure from my past instructions. But to my surprise, Certbot is installed via Snap now, which is just retarded. That discovery triggered me to remember that I read about other ways of getting Let’s Encrypt certificate, such as acme.sh.

On top of that, last month Electronic Frontier Foundation (creators of Certbot) announced that they have joined the hounding of Richard Stallman, so now they can go fuck themselves for sure.

Your domain and DNS settings

Obviously, you need to have a domain (as an example, let’s take domain.dev):

You register it at some domain registrar
DNS records for that domain need to point to the IP address of your host: at the very least there should be an A record pointing your domain.dev to the public IP of your host
- either you add DNS records right at your registrar
- or you point your registrar to nameservers of your web-hoster, and then you manage DNS records (create DNS zone) on web-hoster side

I usually was going with nameservers option and was managing DNS records on web-hoster side, but this time my host was a virtual machine created in Oracle Cloud under Free Tier subscription (absolutely amazing deal, by the way), and Free Tier users cannot have DNS zones, so my only option was to set DNS records at registrar (which actually turned out to be a more convenient option).

Below I’ll describe how I was getting a new domain, but that’s almost completely unrelated to the topic of getting a certificate, so you can just skip to the next section.

Looking for the cheapest domain registration

So, I needed to register yet another domain, but this time for myself. And at first I wanted .io, but it costs 40-50 USD per year. So I started looking at other domains, and .dev seemed like a good alternative: it is also a cool domain (maybe even cooler) and it costs 15-20 USD per year. What I also liked about this domain is that it requires your websites to run via HTTPS. What I didn’t like about this domain is that apparently it belongs to Google.

It also seemed like a good time to get an overview of the current situation on the domains market. First I’ve checked registrars that I’ve been using so far, but neither of them had prices for .dev lower than 15 USD. Then, thanks to Google, I got this list:

Having compared prices from most of those, I didn’t find anything significantly better, but then I got to Porkbun, and they offer .dev domains for as low as 12 USD per year.

Out of curiosity, I also checked their prices for my other domains that I already have registered at other registrars, and those turned out to be cheaper as well, some even almost twice as cheap! So I am now considering moving all my domains to Porkbun.

These guys are hilarious: from their website design to domain features like “Dejigamaflipper”. And if you don’t like their name, there is a link in the footer just for you.

Jokes aside, it looks like a great registrar. Not only their seem to have the lowest prices on the market, they also do not charge anything on top for the WHOIS privacy protection (quite often this option costs around 5 USD per year), which is awww. Their website looks alright (though a bit too “Bootstraped”), they provide API, there is 2FA and even WebAuthn.

Sadly, they haven’t payed me anything for this promotion (sorrowful oink).

acme.sh

Basically, acme.sh is an ACME protocol client written in shell script. As the bare minimum, it supports issuing a new certificate and automatically renewing it with a cron job.

Installation

The following command downloads and executes an “installer” script, which in turn will download and “install” the acme.sh itself and its assets:

$ cd ~
$ curl https://get.acme.sh | sh -s email=YOUR@EMAIL.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   937    0   937    0     0  12662      0 --:--:-- --:--:-- --:--:-- 12662
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  204k  100  204k    0     0  1169k      0 --:--:-- --:--:-- --:--:-- 1169k
[Sat Apr  3 20:44:45 UTC 2021] Installing from online archive.
[Sat Apr  3 20:44:45 UTC 2021] Downloading https://github.com/acmesh-official/acme.sh/archive/master.tar.gz
[Sat Apr  3 20:44:45 UTC 2021] Extracting master.tar.gz
[Sat Apr  3 20:44:45 UTC 2021] It is recommended to install socat first.
[Sat Apr  3 20:44:45 UTC 2021] We use socat for standalone server if you use standalone mode.
[Sat Apr  3 20:44:45 UTC 2021] If you don't use standalone mode, just ignore this warning.
[Sat Apr  3 20:44:45 UTC 2021] Installing to /home/USERNAME/.acme.sh
[Sat Apr  3 20:44:45 UTC 2021] Installed to /home/USERNAME/.acme.sh/acme.sh
[Sat Apr  3 20:44:45 UTC 2021] Installing alias to '/home/USERNAME/.bashrc'
[Sat Apr  3 20:44:45 UTC 2021] OK, Close and reopen your terminal to start using acme.sh
[Sat Apr  3 20:44:45 UTC 2021] Installing cron job
no crontab for USERNAME
no crontab for USERNAME
[Sat Apr  3 20:44:45 UTC 2021] Good, bash is found, so change the shebang to use bash as preferred.
[Sat Apr  3 20:44:46 UTC 2021] OK
[Sat Apr  3 20:44:46 UTC 2021] Install success!

E-mail is needed to register at Let’s Encrypt, so they could send you renewal notice.

Let’s see what we got:

$ ls -lah ~/.acme.sh/
total 236K
drwx------ 5 USERNAME USERNAME 4.0K Apr  3 20:44 .
drwxr-xr-x 6 USERNAME USERNAME 4.0K Apr  3 20:44 ..
-rw-rw-r-- 1 USERNAME USERNAME  193 Apr  3 20:44 account.conf
-rwxrwxr-x 1 USERNAME USERNAME 205K Apr  3 20:44 acme.sh
-rw-rw-r-- 1 USERNAME USERNAME   90 Apr  3 20:44 acme.sh.env
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 deploy
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 dnsapi
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 notify

It also says that it did something with crontab. Let’s check what’s new there:

$ crontab -e
45 0 * * * "/home/USERNAME/.acme.sh"/acme.sh --cron --home "/home/USERNAME/.acme.sh" > /dev/null

So it wants to run the script every day at 00:45. Documentation says that this job will be checking if certificate needs renewal, so it will trigger renewal only when certificate is about to expire. For instance, here’s the log I got from running this cron job on schedule:

[Mon Apr  5 00:45:01 UTC 2021] ===Starting cron===
[Mon Apr  5 00:45:01 UTC 2021] Renew: 'domain.dev'
[Mon Apr  5 00:45:01 UTC 2021] Skip, Next renewal time is: Wed Jun  2 20:35:54 UTC 2021
[Mon Apr  5 00:45:01 UTC 2021] Add '--force' to force to renew.
[Mon Apr  5 00:45:01 UTC 2021] Skipped domain.dev
[Mon Apr  5 00:45:01 UTC 2021] ===End cron===

Getting Let’s Encrypt certificate

The acme.sh script supports different certificate authorities, but I’m interested in exactly Let’s Encrypt.

In order for Let’s Encrypt to verify that you do indeed own the domain.dev, your host will need to pass the ACME verification challenge. There are several types of that challenge, but the easiest (I think) is the HTTP-01:

It will generate a verification token, put it to .well-known/acme-challenge/ of your website root (suppose it’s /var/www/domain.dev/) and initialize the challenge
Let’s Encrypt will then try to reach http://domain.dev/.well-known/acme-challenge/TOKEN. If it succeeds, then you’ve proven that the domain is yours and you’ll get a certificate. It’s not an issue that .dev enforces HTTPS - this concerns only browsers, so Let’s Encrypt will be able to reach the HTTP 80 port on your host just fine

Since the script can be run from a non-root user without sudo, I would recommend to do exactly that. Perhaps you should even create a new user for this purpose. Make sure that user has write access to the website root folder. As I am using NGINX, it is enough to add user to www-data group:

$ sudo usermod -a -G www-data USERNAME

If user doesn’t get www-data in his groups right away:

$ id USERNAME
uid=107(USERNAME) gid=121(USERNAME) groups=121(USERNAME)

log-off, log-in back and check again:

$ id USERNAME
uid=107(USERNAME) gid=121(USERNAME) groups=121(USERNAME),33(www-data)

Then grant access to the website folder:

$ sudo mkdir -p /var/www/domain.dev/.well-known/acme-challenge
$ sudo chown -R www-data:www-data /var/www/domain.dev
$ sudo chmod -R g+rw /var/www/domain.dev

Finally, NGINX settings:

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    server_name domain.dev;

    root /var/www/$server_name;

    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

And you are ready to request a certificate and to try pass the challenge:

$ acme.sh --issue -d domain.dev -w /var/www/domain.dev
[Sat Apr  3 20:52:13 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat Apr  3 20:52:14 UTC 2021] Create account key ok.
[Sat Apr  3 20:52:14 UTC 2021] Registering account: https://acme-v02.api.letsencrypt.org/directory
[Sat Apr  3 20:52:15 UTC 2021] Registered
[Sat Apr  3 20:52:15 UTC 2021] ACCOUNT_THUMBPRINT='07D34pqZi498dFtJwZs8tO-PoF8mM5MKFXaWgx2TkjY'
[Sat Apr  3 20:52:15 UTC 2021] Creating domain key
[Sat Apr  3 20:52:15 UTC 2021] The domain key is here: /home/USERNAME/.acme.sh/domain.dev/domain.dev.key
[Sat Apr  3 20:52:15 UTC 2021] Single domain='domain.dev'
[Sat Apr  3 20:52:15 UTC 2021] Getting domain auth token for each domain
[Sat Apr  3 20:52:16 UTC 2021] Getting webroot for domain='domain.dev'
[Sat Apr  3 20:52:16 UTC 2021] Verifying: domain.dev
[Sat Apr  3 20:52:20 UTC 2021] Pending
[Sat Apr  3 20:52:22 UTC 2021] Pending
[Sat Apr  3 20:52:25 UTC 2021] Pending
[Sat Apr  3 20:52:28 UTC 2021] domain.dev:Verify error:Fetching http://domain.dev/.well-known/acme-challenge/mfRdXgmXwos0wYgxiplwIB45qJJWxMl_B3ZRqWrgswA: Timeout during connect (likely firewall problem)
[Sat Apr  3 20:52:28 UTC 2021] Please add '--debug' or '--log' to check more details.
[Sat Apr  3 20:52:28 UTC 2021] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

As you can see, for me it failed at first. As I realized, that was because I messed up my DNS records a bit, so they were not pointing my new domain to the right host. After I fixed that, the challenge went well:

$ acme.sh --issue -d domain.dev -w /var/www/domain.dev
[Sat Apr  3 21:05:27 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat Apr  3 21:05:27 UTC 2021] Single domain='domain.dev'
[Sat Apr  3 21:05:27 UTC 2021] Getting domain auth token for each domain
[Sat Apr  3 21:05:29 UTC 2021] Getting webroot for domain='domain.dev'
[Sat Apr  3 21:05:29 UTC 2021] Verifying: domain.dev
[Sat Apr  3 21:05:32 UTC 2021] Pending
[Sat Apr  3 21:05:35 UTC 2021] Pending
[Sat Apr  3 21:05:38 UTC 2021] Pending
[Sat Apr  3 21:05:40 UTC 2021] Success
[Sat Apr  3 21:05:40 UTC 2021] Verify finished, start to sign.
[Sat Apr  3 21:05:40 UTC 2021] Lets finalize the order.
[Sat Apr  3 21:05:40 UTC 2021] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/117340399/8841120922'
[Sat Apr  3 21:05:41 UTC 2021] Downloading cert.
[Sat Apr  3 21:05:41 UTC 2021] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/04o8df0db3041a3vf897b6632a163ddlla74'
[Sat Apr  3 21:05:42 UTC 2021] Cert success.
-----BEGIN CERTIFICATE-----
HERE-GOES-CERTIFICATE
-----END CERTIFICATE-----
[Sat Apr  3 21:05:42 UTC 2021] Your cert is in  /home/USERNAME/.acme.sh/domain.dev/domain.dev.cer
[Sat Apr  3 21:05:42 UTC 2021] Your cert key is in  /home/USERNAME/.acme.sh/domain.dev/domain.dev.key
[Sat Apr  3 21:05:42 UTC 2021] The intermediate CA cert is in  /home/USERNAME/.acme.sh/domain.dev/ca.cer
[Sat Apr  3 21:05:42 UTC 2021] And the full chain certs is there:  /home/USERNAME/.acme.sh/domain.dev/fullchain.cer

You might also get these errors in the output:

[Sat Apr  3 21:05:27 UTC 2021] writing token:9OMrOenJSJl3nYnNelPcXpl3ugQzZ8hbwokAsAcDD0M to /var/www/domain.dev/.well-known/acme-challenge/9OMrOenJSJl3nYnNelPcXpl3ugQzZ8hbwokAsAcDD0M
[Sat Apr  3 21:05:27 UTC 2021] Changing owner/group of .well-known to www-data:www-data
[Sat Apr  3 21:05:27 UTC 2021] chown: changing ownership of '/var/www/domain.dev/.well-known/acme-challenge/9OMrOenJSJl3nYnNelPcXpl3ugQzZ8hbwokAsAcDD0M': Operation not permitted
chown: changing ownership of '/var/www/domain.dev/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/domain.dev/.well-known': Operation not permitted

But it will likely succeed anyway. If not, then you might need to add www-data user to the group of the user that executes acme.sh (as challenge tokens get his ownership).

Now let’s see what’s new in the folder:

$ ls -lah ~/.acme.sh/
total 248K
drwx------ 7 USERNAME USERNAME 4.0K Apr  3 20:52 .
drwxr-xr-x 7 USERNAME USERNAME 4.0K Apr  3 20:46 ..
-rw-rw-r-- 1 USERNAME USERNAME  304 Apr  3 21:05 account.conf
-rwxrwxr-x 1 USERNAME USERNAME 205K Apr  3 20:44 acme.sh
-rw-rw-r-- 1 USERNAME USERNAME   90 Apr  3 20:44 acme.sh.env
drwxrwxr-x 3 USERNAME USERNAME 4.0K Apr  3 20:52 ca
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 21:05 domain.dev
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 deploy
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 dnsapi
-rw-rw-r-- 1 USERNAME USERNAME  490 Apr  3 21:05 http.header
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 notify

And here’s what’s inside domain.dev:

$ ls -lah ~/.acme.sh/domain.dev/
total 36K
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 21:05 .
drwx------ 7 USERNAME USERNAME 4.0K Apr  3 20:52 ..
-rw-rw-r-- 1 USERNAME USERNAME 1.6K Apr  3 21:05 ca.cer
-rw-rw-r-- 1 USERNAME USERNAME 1.8K Apr  3 21:05 domain.dev.cer
-rw-rw-r-- 1 USERNAME USERNAME  624 Apr  3 21:05 domain.dev.conf
-rw-rw-r-- 1 USERNAME USERNAME  968 Apr  3 21:05 domain.dev.csr
-rw-rw-r-- 1 USERNAME USERNAME  206 Apr  3 21:05 domain.dev.csr.conf
-rw-rw-r-- 1 USERNAME USERNAME 1.7K Apr  3 20:52 domain.dev.key
-rw-rw-r-- 1 USERNAME USERNAME 3.4K Apr  3 21:05 fullchain.cer

How to use certificate with NGINX

You could simply use files from ~/.acme.sh/domain.dev/, but documentation explicitly says not to do so. Instead you should “install” the certificate into some other folder (accessible by NGINX’s user), for example:

$ sudo mkdir -p /home/www-data/certs/domain.dev
$ sudo chown -R www-data:www-data /home/www-data
$ sudo chmod -R g+rw /home/www-data/certs

$ acme.sh --install-cert -d domain.dev \
--key-file /home/www-data/certs/domain.dev/key.pem  \
--fullchain-file /home/www-data/certs/domain.dev/fullchain.pem \
--reloadcmd "sudo systemctl restart nginx.service"

$ sudo chown -R ubuntu:www-data /home/www-data/certs
$ sudo chmod -R g+rw /home/www-data/certs

$ ls -l /home/www-data/certs/domain.dev/
total 12
-rw-rw-r-- 1 ubuntu www-data 6696 Apr  3 21:36 fullchain.pem
-rw-rw---- 1 ubuntu www-data 1675 Apr  3 21:36 key.pem

You might need to allow your user to restart NGINX service by adding a permission for it in /etc/sudoers.d/.

Now you can modify the NGINX config like this:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name domain.dev;

    # if you get SSL_ERROR_INTERNAL_ERROR_ALERT or other errors,
    # replace $server_name with actual folder name (domain.dev)
    # -
    # it is also assumed that NGINX is run from www-data user,
    # you can verify that in /etc/nginx/nginx.conf
    ssl_certificate /home/www-data/certs/$server_name/fullchain.pem;
    ssl_certificate_key /home/www-data/certs/$server_name/key.pem;

    root /var/www/$server_name;

    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name domain.dev;
    return 301 https://$server_name$request_uri;
}

Yes, you need to do that manually, as, unlike Certbot, acme.sh does not edit NGINX config files, which is actually nice of it. And as you can see for yourself, the only things required for your website to be served with NGINX via HTTPS are:

Listen on 443 (HTTPS) port
- also listen on 80 (HTTP) port, but redirect everything to 443
  - for .dev domains that might be redundant, as in browsers they always load over HTTPS due to HSTS preload
Have ssl_certificate and ssl_certificate_key variables set to paths of your certificate and key

Coming back to the install-cert command, at first I was confused, because the cron job is only set to renew the certificate when it expires, but there is no job to run install-cert, so who will do that? Then I reckoned that after I ran the initial install-cert, it did something else aside from just copying certificate files to ~/certs/domain.dev, and indeed, the ~/.acme.sh/domain.dev/domain.dev.conf file seems to have relevant changes:

Le_Domain='domain.dev'
...
Le_RealKeyPath='/home/USERNAME/certs/domain.dev/key.pem'
Le_ReloadCmd='__ACME_BASE64__START_c3VkbyBzeXN0ZW1jdGwgcmVzdGFydCBuZ2lueC5zZXJ2aWNl__ACME_BASE64__END_'
Le_RealFullChainPath='/home/USERNAME/certs/domain.dev/fullchain.pem'

So it should be all good then, though I am still confused about the fact that reload command is present both in this config and in the cron job. But okay, I guess, I’ll see how it is in a couple of months or so.

Open 443 port

If your website is still not available via HTTPS, it could be that your host doesn’t accept connections on 443 port. There could be two reasons why.

First, if you are using Oracle Cloud, Microsoft Azure or similar cloud provider, then those have a very limited amount of ports open in their virtual networks by default. In fact, you might need to explicitly open even 80 port. For example, here’s how it looks in my Oracle Cloud panel now:

As you see, I needed to add both 80 and 443.

Another reason of blocked 443 port could be that it is not allowed in firewall rules on the host. Here’s what I had:

$ sudo iptables -L
...
Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW,UNTRACKED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http ctstate NEW,UNTRACKED
...

So I needed to add a rule for HTTPS too:

$ sudo iptables -A IN_public_allow -p tcp --dport 443 -j ACCEPT -m conntrack --ctstate NEW,UNTRACKED

$ sudo netfilter-persistent save

$ sudo iptables -L
...
Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW,UNTRACKED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http ctstate NEW,UNTRACKED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https ctstate NEW,UNTRACKED
...

To diagnose, which one of these two reasons is causing problems, you can use cURL.

If you get this output after some time of cURL trying to send the request:

$ curl -I https://domain.dev
curl: (7) Failed to connect to domain.dev port 443: Operation timed out

…then the port is likely not allowed in your cloud provider network settings.

If you get this output right away:

$ curl -I https://domain.dev
curl: (7) Failed to connect to domain.dev port 443: Connection refused

…then it means that you need to allow connections to this port in firewall rules on the host.

And successful request would look like this:

$ curl -I https://domain.dev
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Mon, 05 Apr 2021 11:48:07 GMT
Content-Type: text/html
Content-Length: 391
Last-Modified: Mon, 05 Apr 2021 09:43:18 GMT
Connection: keep-alive
ETag: "606aowe6-324"
Accept-Ranges: bytes

Updates

21.10.2021 | acme.sh now defaults to ZeroSSL

Starting from 01.08.2021 acme.sh defaults to ZeroSSL.

Can’t say if it’s bad or good, I noticed it by accident, after I issued a certificate for a new domain on a new server. Can’t complain about anything, it seems to just work.

And it is still possible to use Let’s Encrypt (or any other supported CA) with --server letsencrypt parameter.

Declaration of VAR

Let's Encrypt certificate with acme.sh instead of Certbot

Your domain and DNS settings

Looking for the cheapest domain registration

acme.sh

Installation

Getting Let’s Encrypt certificate

How to use certificate with NGINX

Open 443 port

Updates

21.10.2021 | acme.sh now defaults to ZeroSSL

Feeds

Tags

Let's Encrypt certificate with acme.sh instead of Certbot

Your domain and DNS settings

Looking for the cheapest domain registration

acme.sh

Installation

Getting Let’s Encrypt certificate

How to use certificate with NGINX

Open 443 port

Updates

21.10.2021 | acme.sh now defaults to ZeroSSL

Feeds

Social networks

Tags