MZan's learning journal

Guix + Nginx + LetsEncrypt + GoAccess stats

9 min read guix

Scenario

I have a remote-host with a static web site generated by Hugo, with access statistics generated by GoAccess.

I administer the remote-host from a local admin-host, using guix deploy.

Prepare remote-host for Guix deploy

TODO I will describe this in another post. By the way, the 99% of instructions here can be applicable also for an host managed directly.

Preparation of the remote-host

Up to date, I’m creating the working directories manually on the remote-host. So it is not a full declarative setup.

First I need to install the Nginx service, because I need the “nginx” user. This configuration file will contains parts useful also in next phases.

; deploy.scm

(use-modules
  (guix)
  (gnu)
  (gnu packages admin)
  (gnu packages attr)
  (gnu packages ci)
  (gnu packages cups)
  (gnu packages vim)
  (gnu packages version-control)
  (gnu packages file-systems)
  (gnu packages image)
  (gnu packages package-management)
  (gnu packages graphviz)
  (gnu packages rust-apps)
  (gnu packages fonts)
  (gnu packages shellutils)
  (gnu packages code)
  (gnu packages patchutils)
  (gnu packages documentation)
  (gnu packages screen)
  (gnu packages admin)
  (gnu packages password-utils)
  (gnu packages disk)
  (gnu packages networking)
  (gnu packages linux)
  (gnu packages rsync)
  (gnu packages sync)
  (gnu packages compression)
  (gnu packages backup)
  (gnu packages shells)
  (gnu packages toys)
  (gnu packages messaging)
  (gnu packages syndication)
  (gnu packages sqlite)
  (gnu packages virtualization)
  (gnu packages cryptsetup)
  (gnu packages samba)
  (gnu system)
  (ice-9 textual-ports)
  (gnu home)
  (gnu home services)
  (gnu home services shells)
  (gnu packages haskell-apps)
  (gnu packages haskell-xyz)
  (gnu packages commencement)
  (gnu services shepherd)
  (gnu system locale)
  (gnu packages unicode)
  (gnu packages terminals)
  (gnu packages version-control)
  (gnu packages web)
  (gnu packages web-browsers)
  (guix channels)
  (srfi srfi-1)
  (gnu system)
  (ice-9 textual-ports))

(use-service-modules certbot
                     dbus 
                     mcron
                     networking ssh web)

(define %guix-server
  (operating-system
    (locale "en_US.utf8")
    (timezone "Europe/Rome")
    (keyboard-layout (keyboard-layout "it" "winkeys"))
    (host-name "dobbkmelody2")

    (bootloader (bootloader-configuration
                (bootloader grub-efi-bootloader)
                (targets (list "/boot/efi"))
                (keyboard-layout keyboard-layout)))

   (file-systems
      ; ...  
   )
   
   users (cons*
            (user-account
               (name "mzan")
               (comment "Massimo")
               (group "users")
               (home-directory "/home/mzan")
               (supplementary-groups '("wheel" "netdev" "audio" "video")))
               
           ; NOTE: user needed by `guix deploy`
           (user-account
              (name "deploy")
              (comment "Guix deploy user")
              (group "users")
              (home-directory "/home/deploy")
              (supplementary-groups '("wheel" "netdev")))
            %base-user-accounts))

    ; NOTE: needed by `guix deploy`
    (sudoers-file
      (plain-file "sudoers"
        (string-append (plain-file-content %sudoers-specification)
                           "deploy ALL = NOPASSWD: ALL
")))

    (packages
     (cons*
      vim htop git rsync
      ripgrep
      fd direnv
      shellcheck
      screen
      just

      goaccess ; web analytics
      openssl  ; for openssl utilities
      
      util-linux
      %base-packages))

    (services
      (cons*
        (service dhcp-client-service-type)

        (service openssh-service-type
          (openssh-configuration
            (permit-root-login 'prohibit-password)
            (password-authentication? #t)
            (authorized-keys
             `(("deploy" ,(local-file "/home/mzan/.ssh/guix-deploy.pub"))
               ("mzan" ,(local-file "/home/mzan/.ssh/id_rsa.pub"))))))

        (service nginx-service-type)

        ; NOTE: needed by `guix deploy`, otherwise it is sufficient
        ; a `%base-services`, left unmodified. 
        (modify-services %base-services
          (guix-service-type config => (guix-configuration
            (inherit config)
            (authorized-keys
              (cons* (local-file "/home/mzan/lavoro/admin/configs/files/keys/master.signing-key.pub")
                      %default-authorized-guix-keys)))))))))

(list (machine
        (operating-system %guix-server)
        (environment managed-host-environment-type)
        (configuration
          (machine-ssh-configuration
            (host-name "5.252.227.178")
            (system "x86_64-linux")
            (user "deploy")
            (identity "/home/mzan/.ssh/guix-deploy")))))

I update the remote-host, from the admin-host with

guix deploy deploy.scm

I connect to remote-host, and I create these directories

sudo mkdir -p /var/www/mzan.dokmelody.org
sudo mkdir -p /var/www/stats.dokmelody.org
sudo mkdir -p /var/www/acme-challenge
sudo mkdir -p /var/lib/opt/goaccess

sudo chown -R nginx:nginx /var/www
sudo chown nginx:nginx /var/lib/opt/goaccess

LetsEncrypt certificates

LetsEncrypt](https://letsencrypt.org/) requires Nginx, for registering the certificates. Nginx was already activated in the previous pass, but in any case, this is another point in which the declaration is not fully declarative, i.e. there are sequential steps to follow.

I add these configurations to deploy.scm on admin-host

; deploy.scm

(use-modules
  (guix)
  (gnu)
  (gnu packages admin)
  ; ...
)

(define %nginx-deploy-hook
  (program-file
   "certbot-deploy-hook.scm"
   (with-imported-modules
    '((gnu services herd))
    #~(begin
        (use-modules (gnu services herd))
        (with-shepherd-action 'nginx ('reload) result result)))))

;; You may want to use a staging ACME server when testing.
(define %testing-certbot-server
  "https://acme-staging-v02.api.letsencrypt.org/directory")

(define (cert-path host file)
  (format #f "/etc/letsencrypt/live/~a/~a.pem" host (symbol->string file)))

(define %guix-server
  ; ... 
  
       (service certbot-service-type
         (certbot-configuration
          (email "mzan@dokmelody.org")
          (webroot "/var/www/acme-challenge")
          
          ; NOTE: uncomment for generating testing certificates
          ; on LetsEncrypt side. Useful for testing the process, 
          ; without be considered a spammer from LetsEncrypt.
          ; (server %testing-certbot-server)
          
          (certificates
           (list
            (certificate-configuration
             ; NOTE: dokmelody.org is the main certificate, 
             ; and it will contain alternative allowed domains in it.
             ; So there will be only one LetsEncrypt request.
             (domains (list "dokmelody.org"
                            "www.dokmelody.org"
                            "mzan.dokmelody.org"
                            "stats.dokmelody.org"))
             (deploy-hook %nginx-deploy-hook))))))
             
      ; ...
)

I update the remote-host

guix deploy deploy.scm

Up to date (apparently), Guix do not restart automatically some jobs, so I need to connect to remote-host and do

# on remote-host
sudo herd restart nginx
sudo herd start renew-certbot-certificates

I need to repeat these commands every time I add websites and certificates.

If there are problems, these can be discovered doing sudo herd status and/or checking /var/log directories.

Configuring Nginx

I update the configuration file in this way, for serving static websites.

; deploy.scm

(use-modules
  (guix)
  ; ...
)

(define (redirect-root-domain-to-www root-name cert-name)
  (nginx-server-configuration
    (listen (list "443 ssl http2"
                  "[::]:443 ssl http2"))
    (server-name (list root-name))
    (ssl-certificate
      (cert-path cert-name 'fullchain))
    (ssl-certificate-key
      (cert-path cert-name 'privkey))
    (raw-content (list (format #f "
         location / {
           return 301 https://www.~a$request_uri;
         }" root-name)))))

(define (static-website-configuration full-name cert-name)
  (nginx-server-configuration
    (listen (list "443 ssl http2"
                  "[::]:443 ssl http2"))
    (server-name (list full-name))
    (root (format #f "/var/www/~a" full-name))
    (index (list "index.html"))
    (ssl-certificate
      (cert-path cert-name 'fullchain))
    (ssl-certificate-key
      (cert-path cert-name 'privkey))
    (raw-content (list (format #f "
          etag on;
          gzip on;
          expires 10m;
          add_header Cache-Control \"public, no-transform\";

          access_log /var/log/nginx/~a.access.log;
          error_log  /var/log/nginx/~a.error.log; "
        full-name
        full-name)))))

(define %guix-server
  (operating-system
  ; ... 
  
    (services
      (cons*
        ; ... 
        (simple-service
           'static-websites
           nginx-service-type
           (list
            (redirect-root-domain-to-www  "dokmelody.org" "dokmelody.org")
            (static-website-configuration "www.dokmelody.org" "dokmelody.org")
            (static-website-configuration "mzan.dokmelody.org" "dokmelody.org")))))
)

I update remote-host with

guix deploy deploy.scm

then the usual

# on remote-host
sudo herd restart nginx

If it is all right:

The generated Nginx configuration can be inspected in this way

$ ps aux  | rg nginx
root        195  0.0  0.2  16368  8952 ?        Ss   Dec01   0:00 nginx: master process /gnu/store/kn56akgnfkqm0ppyfzhyh63ss06ff42s-nginx-1.27.1/sbin/nginx -c /gnu/store/i98l0jilhbczywv8gshwl49cqfs9g1qi-nginx.conf -p /var/run/nginx

$ less /gnu/store/i98l0jilhbczywv8gshwl49cqfs9g1qi-nginx.conf

Adding GoAccess

GoAccess is a basic but good enough application for analyzing Nginx logs. The stats are like these. It does not require JavaScript. The advantages are that the stats will work also for users with disabled JavaScript, and there is no need for “Cookie consents”. The disadvantage is that stats cannot be complete like in case of JavaScript code.

I will configure the remote-host, for running GoAccess every 30 minutes. The reports will be incrementally updated, so the operation is very efficient. The reports will be accessible from https://stats.dokmelody.org but protected by a web password, for privacy reasons.

GoAccess configuration

On the admin-host I create a configuration file for GoAccess. This file will be sent to the remote-host automatically, from guix deploy.

# files/goaccess.conf

time-format %H:%M:%S
date-format %d/%b/%Y

log-format COMBINED

config-dialog false

hl-header true
json-pretty-print false
no-color false
no-column-names false
no-csv-summary false
no-progress false
no-tab-scroll false
with-mouse false

real-time-html false

agent-list false

with-output-resolver false

exclude-ip 127.0.0.1

http-method yes

http-protocol yes

no-query-string false

no-term-resolver false

444-as-404 false

4xx-to-unique-count false

anonymize-ip false

all-static-files false

double-decode false

enable-panel VISITORS
enable-panel REQUESTS
enable-panel REQUESTS_STATIC
enable-panel NOT_FOUND
enable-panel HOSTS
enable-panel OS
enable-panel BROWSERS
enable-panel VISIT_TIMES
enable-panel VIRTUAL_HOSTS
enable-panel REFERRERS
enable-panel REFERRING_SITES
enable-panel KEYPHRASES
enable-panel STATUS_CODES
enable-panel REMOTE_USER
enable-panel CACHE_STATUS
enable-panel MIME_TYPE

hour-spec min

ignore-crawlers false

crawlers-only false

unknowns-as-crawlers false

keep-last 90

real-os true

# Consider the following extensions as static files
# The actual '.' is required and extensions are case sensitive
# For a full list, uncomment the less common static extensions below.
#
static-file .css
static-file .js
static-file .jpg
static-file .png
static-file .gif
static-file .ico
static-file .jpeg
static-file .pdf
static-file .csv
static-file .mpeg
static-file .mpg
static-file .swf
static-file .woff
static-file .woff2
static-file .xls
static-file .xlsx
static-file .doc
static-file .docx
static-file .ppt
static-file .pptx
static-file .txt
static-file .zip
static-file .ogg
static-file .mp3
static-file .mp4
static-file .exe
static-file .iso
static-file .gz
static-file .rar
static-file .svg
static-file .bmp
static-file .tar
static-file .tgz
static-file .tiff
static-file .tif
static-file .ttf
static-file .flv
static-file .dmg
static-file .xz
static-file .zst
#static-file .less
#static-file .ac3
#static-file .avi
#static-file .bz2
#static-file .class
#static-file .cue
#static-file .dae
#static-file .dat
#static-file .dts
#static-file .ejs
#static-file .eot
#static-file .eps
#static-file .img
#static-file .jar
#static-file .map
#static-file .mid
#static-file .midi
#static-file .ogv
#static-file .webm
#static-file .mkv
#static-file .odp
#static-file .ods
#static-file .odt
#static-file .otf
#static-file .pict
#static-file .pls
#static-file .ps
#static-file .qt
#static-file .rm
#static-file .svgz
#static-file .wav
#static-file .webp

htpasswd settings

I connect to remote-host, and I create the htpasswd file, executing:

# on the remote-host
cd /var/www
sudo echo -n 'stats:' > stats.dokmelody.org.htpasswd
sudo openssl passwd -apr1 >> stats.dokmelody.org.htpasswd
sudo chown nginx:nginx stats.dokmelody.org.htpasswd

NOTE: up to date, if I use htpasswd utility, the password will be not recognized. Using the explicit openssl command was the only way I found, for creating a working password.

GoAccess cron job

I add a cron-job for creating/updating the GoAccess reports

; deploy.scm

(use-modules
  (guix)
  ; ... 
)

(define (goaccess-job domain)
  (let ((log-file (format #f "/var/log/nginx/~a.access.log" domain))
        (report-file (format #f "/var/www/stats.dokmelody.org/~a.html" domain))
        (db-dir (format #f "/var/lib/opt/goaccess/~a" domain)))
  #~(job
      '(next-second-from
        ; run every 15 minutes.
        (next-minute (range 0 60 15)))
      (lambda ()
         (system*
          (string-append #$coreutils "/bin/mkdir")
          "-p" #$db-dir)
          (system*
           (string-append #$goaccess "/bin/goaccess")
           (string-append "--config-file=" #$(local-file "/home/mzan/lavoro/admin/configs/files/goaccess.conf"))
           "--persist"
           "--restore"
           (string-append "--db-path=" #$db-dir)
           (string-append "--log-file=" #$log-file)
           (string-append "--output=" #$report-file)))
     #:user "nginx")))

(define %guix-server
  (operating-system
    ; ...
       (services
      (cons*
        ; ... 
        
        (simple-service
           'generate-website-stats
           mcron-service-type
           (list (goaccess-job "mzan.dokmelody.org")
                 (goaccess-job "www.dokmelody.org")))
))))

NOTE: --config-file=" #$(local-file "files/goaccess.conf") effect are:

  • files/goaccess.conf is copied into the Guix store;
  • the filename is replaced with its position in the store;
  • the store with the file will be sent to the remote-host, during guix deploy;

GoAccess website

I create a website, listing all the GoAccess generated reports.

; deploy.scm

(use-modules
  (guix)
  ; ... 
)

(define (website-stats-dokmelody-org cert-name)
  (nginx-server-configuration
    (listen (list "443 ssl http2"
                  "[::]:443 ssl http2"))
    (server-name (list "stats.dokmelody.org"))
    (root "/var/www/stats.dokmelody.org")
    (index '())
    (ssl-certificate
      (cert-path cert-name 'fullchain))
    (ssl-certificate-key
      (cert-path cert-name 'privkey))
    (raw-content (list (format #f "
          etag on;
          gzip on;
          expires 10m;
          add_header Cache-Control \"public, no-transform\";

          access_log off;
          error_log off;

          auth_basic \"Access stats are protected\";
          auth_basic_user_file /var/www/stats.dokmelody.org.htpasswd;

          location / {
            autoindex on;
          }")))))

(define %guix-server
  (operating-system
    ; ...
       (services
      (cons*
        ; ... 
        
       (simple-service
           'stats-website
           nginx-service-type
           (list (website-stats-dokmelody-org "dokmelody.org")))
        
))))

The usual

guix deploy deploy.scm

and on the remote-host

sudo herd restart nginx
sudo herd restart mcron

Retrospective

This specification is not fully declarative, at least initially. But after these passage, I can add new websites, new certificates, new stats, simply adding few lines like

(simple-service
 'static-websites
 nginx-service-type
   (list (redirect-root-domain-to-www  "dokmelody.org" "dokmelody.org")
         (static-website-configuration "www.dokmelody.org" "dokmelody.org")
         (static-website-configuration "mzan.dokmelody.org" "dokmelody.org")))
         
(simple-service
  'generate-website-stats
  mcron-service-type
    (list (goaccess-job "mzan.dokmelody.org")
          (goaccess-job "www.dokmelody.org")))

This specification can be used also for non-static websites, because the GoAccess part requires only that the Nginx log files are created in a certain (configurable) format.

Guix uses Guile Scheme both as a programming language and as a configuration language. Scheme is a perfect language for these mixed tasks. So a lot of boiler-plate config details can be refactored into code.

During coding, guix deploy was gentle enough, because it generated always a log-file with the errors found on the remote-host side. This log-file was signaled in the middle of other more obscure error-messages, so initially I didn’t noticed it.

Possible improvements

  • generate the /var/lib/opt/goaccess working directory automatically
  • manage the /var/www/stats.dokmelody.org.htpasswd file with some secret mechanism, taking in consideration that: if it is put in the Guix store it is visible to all; the MD5 hash is rather easy to break, if exposed;
  • logs are rotated without advising GoAccess, so in worst case scenario 30 minutes of data can be lost at every log-rotation;
  • improve the code of Guix/Shepherd that restarts services after an upgrade;