I needed to set-up a new website with HTTPS and so I took Let’s Encrypt procedure from my past instructions. But to my surprise, Certbot is installed via Snap now, which is just retarded. That discovery triggered me to remember that I read about other ways of getting Let’s Encrypt certificate, such as acme.sh.

acme.sh instead of Certbot

On top of that, last month Electronic Frontier Foundation (creators of Certbot) announced that they have joined the hounding of Richard Stallman (here’s a screenshot, “just in case”), so now they can go fuck themselves for sure.

Your domain and DNS settings

Obviously, you need to have a domain (as an example, let’s take domain.dev):

  1. You register it at some domain registrar
  2. DNS records for that domain need to point to the IP address of your host: at the very least there should be an A record pointing your domain.dev to the public IP of your host
    • either you add DNS records right at your registrar
    • or you point your registrar to nameservers of your web-hoster, and then you manage DNS records (create DNS zone) on web-hoster side

I usually was going with nameservers option and was managing DNS records on web-hoster side, but this time my host was a virtual machine created in Oracle Cloud under Free Tier subscription (absolutely amazing deal, by the way), and Free Tier users cannot have DNS zones, so my only option was to set DNS records at registrar (which actually turned out to be a more convenient option).

Below I’ll describe how I was getting a new domain, but that’s almost completely unrelated to the topic of getting a certificate, so you can just skip to the next section.

Looking for a domain registrar

So, I needed to register yet another domain, but this time for myself. And at first I wanted .io, but it costs 40-50 USD per year. So I started looking at other domains, and .dev seemed like a good alternative: it is also a cool domain (maybe even cooler) and it costs 15-20 USD per year. What I also liked about this domain is that it requires your websites to run via HTTPS. What I didn’t like about this domain is that apparently it belongs to Google.

It also seemed like a good time to get an overview of the current situation on the domains market. First I’ve checked registrars that I’ve been using so far, but neither of them had prices for .dev lower than 15 USD. Then, thanks to Google, I got this list:

Google recommends domain registrars

Having compared prices from most of those, I didn’t find anything significantly better, but then I got to Porkbun, and they offer .dev domains for as low as 12 USD per year.

Out of curiosity, I also checked their prices for my other domains that I already have registered at other registrars, and those turned out to be cheaper as well, some even almost twice as cheap! So I am now considering moving all my domains to Porkbun.

These guys are hilarious: from their website design to domain features like “Dejigamaflipper”. And if you don’t like their name, there is a link in the footer just for you.

Jokes aside, it looks like a great registrar. Not only they seem to have the lowest prices on the market, they also do not charge anything on top for the WHOIS privacy protection (quite often this option costs around 5 USD per year), which is awww. Their website looks alright (though a bit too “Bootstraped”), they provide API, there is 2FA and even WebAuthn.

Sadly, they haven’t payed me anything for this promotion (sorrowful oink).

acme.sh

Basically, acme.sh is an ACME protocol client written in shell script. As the bare minimum, it supports issuing a new certificate and automatically renewing it with a cron job.

Installation

The following command downloads and executes an “installer” script, which in turn will download and “install” the acme.sh itself and its assets:

$ cd ~
$ curl https://get.acme.sh | sh -s email=YOUR@EMAIL.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   937    0   937    0     0  12662      0 --:--:-- --:--:-- --:--:-- 12662
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  204k  100  204k    0     0  1169k      0 --:--:-- --:--:-- --:--:-- 1169k
[Sat Apr  3 20:44:45 UTC 2021] Installing from online archive.
[Sat Apr  3 20:44:45 UTC 2021] Downloading https://github.com/acmesh-official/acme.sh/archive/master.tar.gz
[Sat Apr  3 20:44:45 UTC 2021] Extracting master.tar.gz
[Sat Apr  3 20:44:45 UTC 2021] It is recommended to install socat first.
[Sat Apr  3 20:44:45 UTC 2021] We use socat for standalone server if you use standalone mode.
[Sat Apr  3 20:44:45 UTC 2021] If you don't use standalone mode, just ignore this warning.
[Sat Apr  3 20:44:45 UTC 2021] Installing to /home/USERNAME/.acme.sh
[Sat Apr  3 20:44:45 UTC 2021] Installed to /home/USERNAME/.acme.sh/acme.sh
[Sat Apr  3 20:44:45 UTC 2021] Installing alias to '/home/USERNAME/.bashrc'
[Sat Apr  3 20:44:45 UTC 2021] OK, Close and reopen your terminal to start using acme.sh
[Sat Apr  3 20:44:45 UTC 2021] Installing cron job
no crontab for USERNAME
no crontab for USERNAME
[Sat Apr  3 20:44:45 UTC 2021] Good, bash is found, so change the shebang to use bash as preferred.
[Sat Apr  3 20:44:46 UTC 2021] OK
[Sat Apr  3 20:44:46 UTC 2021] Install success!

E-mail is needed to register at Let’s Encrypt, so they could send you renewal notice.

Let’s see what we got:

$ ls -lah ~/.acme.sh/
total 236K
drwx------ 5 USERNAME USERNAME 4.0K Apr  3 20:44 .
drwxr-xr-x 6 USERNAME USERNAME 4.0K Apr  3 20:44 ..
-rw-rw-r-- 1 USERNAME USERNAME  193 Apr  3 20:44 account.conf
-rwxrwxr-x 1 USERNAME USERNAME 205K Apr  3 20:44 acme.sh
-rw-rw-r-- 1 USERNAME USERNAME   90 Apr  3 20:44 acme.sh.env
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 deploy
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 dnsapi
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 notify

It also says that it did something with crontab. Let’s check what’s new there:

$ crontab -e
45 0 * * * "/home/USERNAME/.acme.sh"/acme.sh --cron --home "/home/USERNAME/.acme.sh" > /dev/null

So it wants to run the script every day at 00:45. Documentation says that this job will be checking if certificate needs renewal, so it will trigger renewal only when certificate is about to expire. For instance, here’s the log I got from running this cron job on schedule:

[Mon Apr  5 00:45:01 UTC 2021] ===Starting cron===
[Mon Apr  5 00:45:01 UTC 2021] Renew: 'domain.dev'
[Mon Apr  5 00:45:01 UTC 2021] Skip, Next renewal time is: Wed Jun  2 20:35:54 UTC 2021
[Mon Apr  5 00:45:01 UTC 2021] Add '--force' to force to renew.
[Mon Apr  5 00:45:01 UTC 2021] Skipped domain.dev
[Mon Apr  5 00:45:01 UTC 2021] ===End cron===

Getting Let’s Encrypt certificate

The acme.sh script supports different certificate authorities, but I’m interested in exactly Let’s Encrypt.

In order for Let’s Encrypt to verify that you do indeed own the domain.dev, your host will need to pass the ACME verification challenge. There are several types of that challenge, but the easiest (I think) is the HTTP-01 (I no longer think so):

  1. It will generate a verification token, put it to .well-known/acme-challenge/ of your website root (suppose it’s /var/www/domain.dev/) and initialize the challenge
  2. Let’s Encrypt will then try to reach http://domain.dev/.well-known/acme-challenge/TOKEN. If it succeeds, then you’ve proven that the domain is yours and you’ll get a certificate. It’s not an issue that .dev enforces HTTPS - this concerns only browsers, so Let’s Encrypt will be able to reach the HTTP 80 port on your host just fine

Since the script can be run from a non-root user without sudo, I would recommend to do exactly that. Perhaps you should even create a new user for this purpose. Make sure that user has write access to the website root folder. As I am using NGINX, it is enough to add user to www-data group:

$ sudo usermod -a -G www-data USERNAME

If user doesn’t get www-data in his groups right away:

$ id USERNAME
uid=107(USERNAME) gid=121(USERNAME) groups=121(USERNAME)

log-off, log-in back and check again:

$ id USERNAME
uid=107(USERNAME) gid=121(USERNAME) groups=121(USERNAME),33(www-data)

Then grant access to the website folder:

$ sudo mkdir -p /var/www/domain.dev/.well-known/acme-challenge
$ sudo chown -R www-data:www-data /var/www/domain.dev
$ sudo chmod -R g+rw /var/www/domain.dev

Finally, NGINX settings:

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    server_name domain.dev;

    root /var/www/$server_name;

    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

And you are ready to request a certificate and to try pass the challenge:

$ acme.sh --issue -d domain.dev -w /var/www/domain.dev
[Sat Apr  3 20:52:13 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat Apr  3 20:52:14 UTC 2021] Create account key ok.
[Sat Apr  3 20:52:14 UTC 2021] Registering account: https://acme-v02.api.letsencrypt.org/directory
[Sat Apr  3 20:52:15 UTC 2021] Registered
[Sat Apr  3 20:52:15 UTC 2021] ACCOUNT_THUMBPRINT='07D34pqZi498dFtJwZs8tO-PoF8mM5MKFXaWgx2TkjY'
[Sat Apr  3 20:52:15 UTC 2021] Creating domain key
[Sat Apr  3 20:52:15 UTC 2021] The domain key is here: /home/USERNAME/.acme.sh/domain.dev/domain.dev.key
[Sat Apr  3 20:52:15 UTC 2021] Single domain='domain.dev'
[Sat Apr  3 20:52:15 UTC 2021] Getting domain auth token for each domain
[Sat Apr  3 20:52:16 UTC 2021] Getting webroot for domain='domain.dev'
[Sat Apr  3 20:52:16 UTC 2021] Verifying: domain.dev
[Sat Apr  3 20:52:20 UTC 2021] Pending
[Sat Apr  3 20:52:22 UTC 2021] Pending
[Sat Apr  3 20:52:25 UTC 2021] Pending
[Sat Apr  3 20:52:28 UTC 2021] domain.dev:Verify error:Fetching http://domain.dev/.well-known/acme-challenge/mfRdXgmXwos0wYgxiplwIB45qJJWxMl_B3ZRqWrgswA: Timeout during connect (likely firewall problem)
[Sat Apr  3 20:52:28 UTC 2021] Please add '--debug' or '--log' to check more details.
[Sat Apr  3 20:52:28 UTC 2021] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

As you can see, for me it failed at first. As I realized, that was because I messed up my DNS records a bit, so they were not pointing my new domain to the right host. After I fixed that, the challenge went well:

$ acme.sh --issue -d domain.dev -w /var/www/domain.dev
[Sat Apr  3 21:05:27 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat Apr  3 21:05:27 UTC 2021] Single domain='domain.dev'
[Sat Apr  3 21:05:27 UTC 2021] Getting domain auth token for each domain
[Sat Apr  3 21:05:29 UTC 2021] Getting webroot for domain='domain.dev'
[Sat Apr  3 21:05:29 UTC 2021] Verifying: domain.dev
[Sat Apr  3 21:05:32 UTC 2021] Pending
[Sat Apr  3 21:05:35 UTC 2021] Pending
[Sat Apr  3 21:05:38 UTC 2021] Pending
[Sat Apr  3 21:05:40 UTC 2021] Success
[Sat Apr  3 21:05:40 UTC 2021] Verify finished, start to sign.
[Sat Apr  3 21:05:40 UTC 2021] Lets finalize the order.
[Sat Apr  3 21:05:40 UTC 2021] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/SOME-ID/ANOTHER-ID'
[Sat Apr  3 21:05:41 UTC 2021] Downloading cert.
[Sat Apr  3 21:05:41 UTC 2021] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/SOME-ID-HERE'
[Sat Apr  3 21:05:42 UTC 2021] Cert success.
-----BEGIN CERTIFICATE-----
HERE-GOES-CERTIFICATE
-----END CERTIFICATE-----
[Sat Apr  3 21:05:42 UTC 2021] Your cert is in  /home/USERNAME/.acme.sh/domain.dev/domain.dev.cer
[Sat Apr  3 21:05:42 UTC 2021] Your cert key is in  /home/USERNAME/.acme.sh/domain.dev/domain.dev.key
[Sat Apr  3 21:05:42 UTC 2021] The intermediate CA cert is in  /home/USERNAME/.acme.sh/domain.dev/ca.cer
[Sat Apr  3 21:05:42 UTC 2021] And the full chain certs is there:  /home/USERNAME/.acme.sh/domain.dev/fullchain.cer

You might also get these errors in the output:

[Sat Apr  3 21:05:27 UTC 2021] writing token:9OMrOenJSJl3nYnNelPcXpl3ugQzZ8hbwokAsAcDD0M to /var/www/domain.dev/.well-known/acme-challenge/9OMrOenJSJl3nYnNelPcXpl3ugQzZ8hbwokAsAcDD0M
[Sat Apr  3 21:05:27 UTC 2021] Changing owner/group of .well-known to www-data:www-data
[Sat Apr  3 21:05:27 UTC 2021] chown: changing ownership of '/var/www/domain.dev/.well-known/acme-challenge/9OMrOenJSJl3nYnNelPcXpl3ugQzZ8hbwokAsAcDD0M': Operation not permitted
chown: changing ownership of '/var/www/domain.dev/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/domain.dev/.well-known': Operation not permitted

But it will likely succeed anyway. If not, then you might need to add www-data user to the group of the user that executes acme.sh (as challenge tokens get his ownership).

Now let’s see what’s new in the folder:

$ ls -lah ~/.acme.sh/
total 248K
drwx------ 7 USERNAME USERNAME 4.0K Apr  3 20:52 .
drwxr-xr-x 7 USERNAME USERNAME 4.0K Apr  3 20:46 ..
-rw-rw-r-- 1 USERNAME USERNAME  304 Apr  3 21:05 account.conf
-rwxrwxr-x 1 USERNAME USERNAME 205K Apr  3 20:44 acme.sh
-rw-rw-r-- 1 USERNAME USERNAME   90 Apr  3 20:44 acme.sh.env
drwxrwxr-x 3 USERNAME USERNAME 4.0K Apr  3 20:52 ca
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 21:05 domain.dev
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 deploy
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 dnsapi
-rw-rw-r-- 1 USERNAME USERNAME  490 Apr  3 21:05 http.header
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 20:44 notify

And here’s what’s inside domain.dev:

$ ls -lah ~/.acme.sh/domain.dev/
total 36K
drwxrwxr-x 2 USERNAME USERNAME 4.0K Apr  3 21:05 .
drwx------ 7 USERNAME USERNAME 4.0K Apr  3 20:52 ..
-rw-rw-r-- 1 USERNAME USERNAME 1.6K Apr  3 21:05 ca.cer
-rw-rw-r-- 1 USERNAME USERNAME 1.8K Apr  3 21:05 domain.dev.cer
-rw-rw-r-- 1 USERNAME USERNAME  624 Apr  3 21:05 domain.dev.conf
-rw-rw-r-- 1 USERNAME USERNAME  968 Apr  3 21:05 domain.dev.csr
-rw-rw-r-- 1 USERNAME USERNAME  206 Apr  3 21:05 domain.dev.csr.conf
-rw-rw-r-- 1 USERNAME USERNAME 1.7K Apr  3 20:52 domain.dev.key
-rw-rw-r-- 1 USERNAME USERNAME 3.4K Apr  3 21:05 fullchain.cer

How to use certificate with NGINX

You could simply use files from ~/.acme.sh/domain.dev/, but documentation explicitly says not to do so. Instead you should “install” the certificate into some other folder (accessible by NGINX’s user), for example:

$ sudo mkdir -p /home/www-data/certs/domain.dev
$ sudo chown -R www-data:www-data /home/www-data
$ sudo chmod -R g+rw /home/www-data/certs

$ acme.sh --install-cert -d domain.dev \
    --key-file /home/www-data/certs/domain.dev/key.pem  \
    --fullchain-file /home/www-data/certs/domain.dev/fullchain.pem \
    --reloadcmd "sudo systemctl restart nginx.service"

$ sudo chown -R www-data:www-data /home/www-data/certs
$ sudo chmod -R g+rw /home/www-data/certs

$ ls -l /home/www-data/certs/domain.dev/
total 12
-rw-rw-r-- 1 www-data www-data 6696 Apr  3 21:36 fullchain.pem
-rw-rw---- 1 www-data www-data 1675 Apr  3 21:36 key.pem

You might need to allow your user to restart NGINX service without prompting for password by adding a permission for it in /etc/sudoers.d/www-data:

%www-data ALL= NOPASSWD: /bin/systemctl restart nginx.service

Now you can modify the NGINX config like this:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name domain.dev;

    # if you get SSL_ERROR_INTERNAL_ERROR_ALERT or other errors,
    # replace $server_name with actual folder name (domain.dev)
    # -
    # it is also assumed that NGINX is run from www-data user,
    # you can verify that in /etc/nginx/nginx.conf
    ssl_certificate /home/www-data/certs/$server_name/fullchain.pem;
    ssl_certificate_key /home/www-data/certs/$server_name/key.pem;

    root /var/www/$server_name;

    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name domain.dev;
    return 301 https://$server_name$request_uri;
}

Yes, you need to do that manually, as, unlike Certbot, acme.sh does not edit NGINX config files, which is actually nice of it. And as you can see for yourself, the only things required for your website to be served with NGINX via HTTPS are:

  1. Listen on 443 (HTTPS) port
    • also listen on 80 (HTTP) port, but redirect everything to 443
      • for .dev domains that might be redundant, as in browsers they always load over HTTPS due to HSTS preload
  2. Have ssl_certificate and ssl_certificate_key variables set to paths of your certificate and key

Coming back to the install-cert command, at first I was confused, because the cron job is only set to renew the certificate when it expires, but there is no job to run install-cert, so who will do that? Then I reckoned that after I ran the initial install-cert, it did something else aside from just copying certificate files to ~/certs/domain.dev, and indeed, the ~/.acme.sh/domain.dev/domain.dev.conf file seems to have relevant changes:

Le_Domain='domain.dev'
...
Le_RealKeyPath='/home/USERNAME/certs/domain.dev/key.pem'
Le_ReloadCmd='__ACME_BASE64__START_c3VkbyBzeXN0ZW1jdGwgcmVzdGFydCBuZ2lueC5zZXJ2aWNl__ACME_BASE64__END_'
Le_RealFullChainPath='/home/USERNAME/certs/domain.dev/fullchain.pem'

So it should be all good then, though I am still confused about the fact that reload command is present both in this config and in the cron job.

Here’s also a useful command for checking the certificate dates, once you have it installed and set-up in NGINX:

$ echo | openssl s_client -connect domain.dev:443 2>&1 | openssl x509 -noout -dates
notBefore=Oct 24 18:30:24 2023 GMT
notAfter=Jan 22 18:30:23 2024 GMT

Open 443 port

If your website is still not available via HTTPS, it could be that your host doesn’t accept connections on 443 port. There could be two reasons why.

First, if you are using Oracle Cloud, Microsoft Azure or similar cloud provider, then those have a very limited amount of ports open in their virtual networks by default. In fact, you might need to explicitly open even 80 port. For example, here’s how it looks in my Oracle Cloud panel now:

Oracle Cloud ports

As you see, I needed to add both 80 and 443.

Another reason of blocked 443 port could be that it is not allowed in firewall rules on the host. Here’s what I had:

$ sudo iptables -L
...
Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW,UNTRACKED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http ctstate NEW,UNTRACKED
...

So I needed to add a rule for HTTPS too:

$ sudo iptables -A IN_public_allow -p tcp --dport 443 -j ACCEPT -m conntrack --ctstate NEW,UNTRACKED

$ sudo netfilter-persistent save

$ sudo iptables -L
...
Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW,UNTRACKED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http ctstate NEW,UNTRACKED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https ctstate NEW,UNTRACKED
...

To diagnose, which one of these two reasons is causing problems, you can use cURL.

If you get this output after some time of cURL trying to send the request:

$ curl -I https://domain.dev
curl: (7) Failed to connect to domain.dev port 443: Operation timed out

…then the port is likely not allowed in your cloud provider network settings.

If you get this output right away:

$ curl -I https://domain.dev
curl: (7) Failed to connect to domain.dev port 443: Connection refused

…then it means that you need to allow connections to this port in firewall rules on the host.

And successful request would look like this:

$ curl -I https://domain.dev
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Mon, 05 Apr 2021 11:48:07 GMT
Content-Type: text/html
Content-Length: 391
Last-Modified: Mon, 05 Apr 2021 09:43:18 GMT
Connection: keep-alive
ETag: "606aowe6-324"
Accept-Ranges: bytes

Updates

2021-10-21 | ZeroSSL is the default server

Starting from 01.08.2021 acme.sh defaults to ZeroSSL.

Can’t say if it’s bad or good, I noticed it by accident, after I issued a certificate for a new domain on a new server. Can’t complain about anything (yet), it seems to just work.

And it is still possible to use Let’s Encrypt (or any other supported CA) with --server letsencrypt parameter.

2023-03-18 | Wildcard certificate using DNS challenge and registrar API

Eventually I got tired of performing a HTTP-challenge every time I needed to add a new subdomain and fighting with various issues related to /.well-known/acme-challenge/ contents discovery/availability/access.

A more convenient way would be to issue a wildcard (*.domain.dev) certificate by using DNS-01 challenge, especially if your domain name registrar provides an API. Porkbun does, so here goes:

$ acme.sh --issue -d domain.dev -d *.domain.dev --dns dns_porkbun

It most likely will fail with the following error:

[Sat Mar 18 14:18:01 CET 2023] You didn't specify a Porkbun api key and secret api key yet.
[Sat Mar 18 14:18:01 CET 2023] You can get yours from here https://porkbun.com/account/api.
[Sat Mar 18 14:18:01 CET 2023] Error add txt for domain:_acme-challenge.domain.dev

…because you need to provide your Porkbun API credentials, for example as environment variables:

$ PORKBUN_API_KEY="HERE-GOES-PORKBUN-API-KEY" PORKBUN_SECRET_API_KEY="HERE-GOES-PORKBUN-API-SECRET" \
    acme.sh --issue -d domain.dev -d *.domain.dev --dns dns_porkbun

But even then it might fail with an error like this:

{
    "status": "ERROR",
    "message": "Domain is not opted in to API access."
}

As it was pointed out in the comments, don’t forget to enable API access in the domain settings in your Porkbun account (obviously, other registrars might have different settings).

If it’s all good, then by default it will add (and then remove?) a TXT record _acme-challenge.domain.dev. If you for some reason would like to have a different record, then you can provide a different alias with --challenge-alias ololo.domain.dev.

But in my case it still failed with the following error:

[Sat Mar 18 14:31:08 CET 2023] Processing, The CA is processing your order, please just wait. (1/30)
[Sat Mar 18 14:31:08 CET 2023] sleep 2 secs to verify again
[Sat Mar 18 14:31:11 CET 2023] checking
[Sat Mar 18 14:31:11 CET 2023] url='https://acme.zerossl.com/v2/DV90/chall/ZwaSIHUoU8O6xHCZ4uFk-A'
[Sat Mar 18 14:31:11 CET 2023] payload
[Sat Mar 18 14:31:11 CET 2023] POST
[Sat Mar 18 14:31:11 CET 2023] _post_url='https://acme.zerossl.com/v2/DV90/chall/ZwaSIHUoU8O6xHCZ4uFk-A'
[Sat Mar 18 14:31:11 CET 2023] _CURL='curl --silent --dump-header /home/USERNAME/.acme.sh/http.header  -L  -g '
[Sat Mar 18 14:31:11 CET 2023] _ret='0'
[Sat Mar 18 14:31:11 CET 2023] code='200'
[Sat Mar 18 14:31:11 CET 2023] domain.dev:Verify error:"error":{

No more details, just some “error”, which looks cut off. Either way, that probably means that ZeroSSL (the default server) does not support that kind of challenge? So I tried with Let’s Encrypt (--server letsencrypt) instead:

$ PORKBUN_API_KEY="HERE-GOES-PORKBUN-API-KEY" PORKBUN_SECRET_API_KEY="HERE-GOES-PORKBUN-API-SECRET" \
    acme.sh --issue -d domain.dev -d *.domain.dev --dns dns_porkbun --server letsencrypt

And that succeeded just fine:

[Sat Mar 18 14:32:11 CET 2023] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat Mar 18 14:32:11 CET 2023] Multi domain='DNS:domain.dev,DNS:*.domain.dev'
[Sat Mar 18 14:32:11 CET 2023] Getting domain auth token for each domain
[Sat Mar 18 14:32:13 CET 2023] Getting webroot for domain='domain.dev'
[Sat Mar 18 14:32:14 CET 2023] Getting webroot for domain='*.domain.dev'
[Sat Mar 18 14:32:14 CET 2023] Adding txt value: K9rLdyD6-q_Hj7FWV4yAUIUiOh9eF4256UVRHoqoitE for domain:  _acme-challenge.domain.dev
[Sat Mar 18 14:32:24 CET 2023] Adding record
[Sat Mar 18 14:32:29 CET 2023] Added, OK
[Sat Mar 18 14:32:29 CET 2023] The txt record is added: Success.
[Sat Mar 18 14:32:29 CET 2023] Let's check each DNS record now. Sleep 20 seconds first.
[Sat Mar 18 14:32:50 CET 2023] You can use '--dnssleep' to disable public dns checks.
[Sat Mar 18 14:32:50 CET 2023] See: https://github.com/acmesh-official/acme.sh/wiki/dnscheck
[Sat Mar 18 14:32:50 CET 2023] Checking domain.dev for _acme-challenge.domain.dev
[Sat Mar 18 14:32:50 CET 2023] Domain domain.dev '_acme-challenge.domain.dev' success.
[Sat Mar 18 14:32:50 CET 2023] All success, let's return
[Sat Mar 18 14:32:50 CET 2023] domain.dev is already verified, skip dns-01.
[Sat Mar 18 14:32:51 CET 2023] Verifying: *.domain.dev
[Sat Mar 18 14:32:51 CET 2023] Pending, The CA is processing your order, please just wait. (1/30)
[Sat Mar 18 14:32:55 CET 2023] Success
[Sat Mar 18 14:32:55 CET 2023] Removing DNS records.
[Sat Mar 18 14:32:55 CET 2023] Removing txt: K9rLdyD6-q_Hj7FWV4yAUIUiOh9eF4256UVRHoqoitE for domain: _acme-challenge.domain.dev
[Sat Mar 18 14:33:10 CET 2023] Removed: Success
[Sat Mar 18 14:33:10 CET 2023] Verify finished, start to sign.
[Sat Mar 18 14:33:10 CET 2023] Lets finalize the order.
[Sat Mar 18 14:33:10 CET 2023] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/SOME-ID/ANOTHER-ID'
[Sat Mar 18 14:33:11 CET 2023] Downloading cert.
[Sat Mar 18 14:33:11 CET 2023] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/SOME-ID-HERE'
[Sat Mar 18 14:33:12 CET 2023] Cert success.
-----BEGIN CERTIFICATE-----
HERE-GOES-CERTIFICATE
-----END CERTIFICATE-----
[Sat Mar 18 14:33:12 CET 2023] Your cert is in: /home/USERNAME/.acme.sh/domain.dev_ecc/domain.dev.cer
[Sat Mar 18 14:33:12 CET 2023] Your cert key is in: /home/USERNAME/.acme.sh/domain.dev_ecc/domain.dev.key
[Sat Mar 18 14:33:12 CET 2023] The intermediate CA cert is in: /home/USERNAME/.acme.sh/domain.dev_ecc/ca.cer
[Sat Mar 18 14:33:12 CET 2023] And the full chain certs is there: /home/USERNAME/.acme.sh/domain.dev_ecc/fullchain.cer

Don’t forget to install the new certificate:

$ acme.sh --install-cert -d domain.dev \
    --key-file /home/www-data/certs/domain.dev/key.pem \
    --fullchain-file /home/www-data/certs/domain.dev/fullchain.pem \
    --reloadcmd "sudo systemctl restart nginx.service"

[Sat Mar 18 15:05:17 CET 2023] The domain 'domain.dev' seems to have a ECC cert already, lets use ecc cert.
[Sat Mar 18 15:05:17 CET 2023] Installing key to: /home/www-data/certs/domain.dev/key.pem
[Sat Mar 18 15:05:17 CET 2023] Installing full chain to: /home/www-data/certs/domain.dev/fullchain.pem
[Sat Mar 18 15:05:17 CET 2023] Run reload cmd: sudo systemctl restart nginx.service
[Sat Mar 18 15:05:17 CET 2023] Reload success

If your website will still reports using the previous certificate, try to clear cache and reload the page or open it in a different web-browser. If it still says that it uses the “old” certificate, then perhaps you (like me) have a bit of a mess inside /home/USERNAME/.acme.sh after all these issuings, and so probably it installed the wrong one. Then you could probably try deleting all the domains folders and re-issuing the wildcard once again. After doing so I now have only one domain folder: /home/USERNAME/.acme.sh/domain.dev_ecc.

Last thing, in case it’s not obvious, since it’s a wildcard certificate, in NGINX websites configs instead of $server_name variable you’ll need to use the exact same path for all your websites that are under this wildcard domain. For instance, instead of:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name something.domain.dev;

    ssl_certificate /home/www-data/certs/$server_name/fullchain.pem;
    ssl_certificate_key /home/www-data/certs/$server_name/key.pem;

    # ...
}

you should have:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name something.domain.dev;

    ssl_certificate /home/www-data/certs/domain.dev/fullchain.pem;
    ssl_certificate_key /home/www-data/certs/domain.dev/key.pem;

    # ...
}

2023-10-27 | Using a DNS alias

We have our own DNS server/service at work, which manages some of our domains, and this server does not have an API for editing DNS records, which quite naturally makes it difficult to pass the DNS-01 challenge for those domains. We could theoretically automate editing our BIND zone file on the DNS server via SSH or some other way, but that is very much error-prone, so we’d prefer not to.

If you are in a similar situation, or/and if your domain registrar does not have an API, do not despair, as DNS-01 challenge can be passed using a simple CNAME record. The way it works is that you add a CNAME record in the original domain, pointing to an “alias domain” that is registered and managed by some registrar that does have an API, and then Let’s Encrypt will read TXT records from this one instead of the original.

For example, let’s say our original domain of interest is our-company.net, and it is managed by a registrar/service without API. And there is also our-company.com domain, which is managed by a registrar that does have an API. So the first step will be to add a CNAME record (in the original domain zone file) for _acme-challenge.our-company.net, which would point to _acme-challenge.our-company.com.

This article says to set it in the following format (pay attention to the terminating dot):

_acme-challenge.our-company.net IN CNAME _acme-challenge.our-company.com.

And that is the record I added in our BIND zone file at first, but as it turned out, this is a wrong format, or at least it was in our case, because Let’s Encrypt wasn’t able to discover this record, as it was still trying to read TXT records from our-company.net instead of our-company.com:

{
  "identifier": {
    "type": "dns",
    "value": "our-company.net"
  },
  "status": "invalid",
  "expires": "2023-10-31T17:10:24Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:dns",
        "detail": "DNS problem: NXDOMAIN looking up TXT for _acme-challenge.our-company.net - check that a DNS record exists for this domain",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/SOME-ID/ANOTHER-ID",
      "token": "SOME-TOKEN",
      "validated": "2023-10-24T17:11:04Z"
    }
  ]
}

and obviously the verification was failing, because those records were set for _acme-challenge.our-company.com.

I don’t know, if it is our DNS server/zone settings or something else to blame, but the right format turned out to be this:

_acme-challenge IN CNAME _acme-challenge.our-company.com.

and that worked out fine.

So the first thing you should do before trying to issue a certificate is to ensure that the CNAME record is added correctly and is successfully discovered by DNS checking tools (and so Let’s Encrypt can is able to read it too). You can use some of the online services for that, such as this one, or a suitable CLI tool, such as dig:

$ dig _acme-challenge.our-company.net CNAME

; <<>> DiG 9.10.6 <<>> _acme-challenge.our-company.net CNAME
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61736
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.our-company.net.    IN    CNAME

;; ANSWER SECTION:
_acme-challenge.our-company.net. 3600    IN    CNAME    _acme-challenge.our-company.com.

;; Query time: 36 msec
;; SERVER: 192.0.2.42#53(192.0.2.42)
;; WHEN: Tue Oct 24 22:13:13 CEST 2023
;; MSG SIZE  rcvd: 93

here’s also a short variant:

$ dig _acme-challenge.our-company.net CNAME +short
_acme-challenge.our-company.com.

Just in case, be aware that you (like us) might have different zones for your domains for internal and external network, so to be sure you should run dig from outside your office network.

Having verified that the record is set, you can now issue a certificate by running acme.sh with --challenge-alias argument pointing to the alias domain (the one that should get TXT records with challenge tokens):

$ domainregistrar_token="TOKEN-HERE" domainregistrar_secret="SECRET-HERE" \
    acme.sh --issue -d our-company.net --challenge-alias our-company.com \
    --dns dns_domainregistrar \
    -d *.our-company.net \
    --server letsencrypt \
    --debug

The order of arguments seems to matter, so just in case don’t shuffle them around.

You can certainly drop the --debug argument, but it is useful for finding out what’s wrong if something goes wrong, and Let’s Encrypt limits issuing a certificate to 5 attempts per hour, so it’s nice to get all the output right away already on the first attempt, without wasting another attempt just to get the debugging output.

For clarity, what will happen (as I understand it) on executing that command above:

  1. Let’s Encrypt will be queried for DNS-01 challenge tokens;
  2. Two TXT records with the tokens will be added for our-company.com domain in the registrar DNS records via API;
  3. Let’s Encrypt will try to read records for _acme-challenge.our-company.net, but since it’s a CNAME, it will read records for _acme-challenge.our-company.com instead;
  4. As those are set (and hopefully are correct/valid), the challenge will succeed and certificate will be issued.

The rest is a standard procedure as before, install the certificate:

$ acme.sh --install-cert -d our-company.net \
    --key-file /var/www/certificates.our-company.net/certs/key.pem \
    --fullchain-file /var/www/certificates.our-company.net/certs/fullchain.pem \
    --reloadcmd "sudo systemctl restart nginx.service"

and use the installation path in your web-server config.

The renewal cron job should’ve been already added during acme.sh installation, but you might want to customize it a bit (not running it every day, restarting NGINX and logging the output):

1 1 1,20 * * "/path/to/.acme.sh"/acme.sh --cron --home "/path/to/.acme.sh" --reloadcmd "sudo systemctl restart nginx.service" >> /path/to/logs/cron-acme.log 2>&1

Also, don’t delete the CNAME record that points to the alias domain, because it will likely be still needed for renewals.

Distributing certificate files to internal servers

In our case we also needed to distribute the certificate files to other servers in our network, so our other internal websites under *.our-company.net domain could enable HTTPS (yes, even though it is an internal network, it is still better or rather required to have authentication requests going through HTTPS and not HTTP). That was actually the main reason why we wanted to get a wildcard certificate, as internal servers that are hosting our internal websites/services are not exposed to the internet, and it is (close to) impossible to issue certificates directly on those servers. Having a wildcard certificate solves this problem, as we just need to distribute its files, which is a trivial task.

We decided that the easiest would be to serve the certificate files from the main server (protected with Basic authentication). Here’s an NGINX config for that (using HTTPS too):

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name certificates.our-company.net;

    # that path is where certificate files were installed to
    # and also where they are served from
    ssl_certificate /var/www/certificates.our-company.net/certs/fullchain.pem;
    ssl_certificate_key /var/www/certificates.our-company.net/certs/key.pem;

    charset utf-8;

    root /var/www/certificates.our-company.net;

    index index.html;

    # and since certificate files shouldn't be available to anyone,
    # that route is protected with Basic authentication
    location /certs {
        try_files $uri $uri/ =404;
        auth_basic "You know the rules";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }

    location / {
        try_files $uri $uri/ =404;
    }

    error_page 404 /404.html;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name certificates.our-company.net;
    return 301 https://$server_name$request_uri;
}

And then other servers get the certificate files via cron job (the certificate gets renewed, so this needs to be done regularly) script like this one:

#!/bin/bash

cd /var/www/certificates

httpStatus=$(/usr/bin/curl -s --netrc -O https://certificates.our-company.net/certs/fullchain.pem -w "%{http_code}")
if [[ $httpStatus != 200 ]]; then
    echo "[ERROR] Failed to download fullchain.pem" 1>&2
    exit 1
fi

httpStatus=$(/usr/bin/curl -s --netrc -O https://certificates.our-company.net/certs/key.pem -w "%{http_code}")
if [[ $httpStatus != 200 ]]; then
    echo "[ERROR] Failed to download key.pem" 1>&2
    exit 2
fi

Basic authentication credentials are stored in .netrc file:

machine certificates.our-company.net
login USERNAME-HERE
password PASSWORD-HERE

otherwise these need to be provided inline via -u USERNAME-HERE:PASSWORD-HERE instead of --netrc.