Chargen.One

Chargen.One

The latest posts from Chargen.One.

from steve

Now that my OpenBSD.Amsterdam VPS is up and running, and I have working backups, I thought I'd migrate some static sites over to this host and free up another dedicated server I'm using. Adding extra static HTML won't add to the VPS' general load and won't introduce new risks to #Chargen.One.

To do this, I need to implement name-based Virtual Hosting. I'm going to show how this is done for one site, hackingforfoodbanks.org, then build upon it for multiple hosts. Finally, I'll modularize elements of the configuration to make things more manageable, including HTTPS support.

To make Name-based virtual hosting work, it's necessary to update /etc/acme-client.conf, the DNS Records for the domain in question, and the nginx configuration.

Moving DNS

This is the simplest part of the job. It's simply a case of logging into a DNS provider, and pointing the relevant DNS records at the HTTP server. Log into the DNS provider or server, point the relevant 'A' and/or 'CNAME' records to the HTTP server's IP address, and be prepared to wait up to 24 hours.

Now DNS is out of the way, the next thing is to clean up the nginx config from earlier.

Segregating the Nginx config

The config as-is is fine for just hosting Chargen.One but could get a bit unwieldy if I move all of my static sites across. I created a subdirectory in /etc/nginx/ called sites, into which I can add server blocks for each site I want to host. This splits the configuration up into more manageable per-site blocks.

Before adding a new host, I split out the default chargen.one site config into a new file, /etc/nginx/sites/default.conf. This is a copy of the main /etc/nginx/nginx.conf site with everything from the openings server{ to closing } characters included. It looks like this:

server {
	listen       80 default_server;
	listen       [::]:80 default_server;
	server_name  _;
	root         /var/www/htdocs/c1;
	
	include acme.conf;
	
	#access_log  logs/host.access.log  main;
	#error_page  404              /404.html;
	
	# redirect server error pages to the static page /50x.html
	error_page   500 502 503 504  /50x.html;
	location = /50x.html {
	    root  /var/www/htdocs/c1;
	}
	
	# For reading content
	location ~ ^/(css|img|js|fonts)/ {
	        root /var/www/htdocs/c1;
	        # Optionally cache these files in the browser:
	        # expires 12M;
	}

	
	location ~ ^/.well-known/(webfinger|nodeinfo|host-meta) {
	    proxy_set_header Host $host;
	    proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-Forwarded-For $remote_addr;
	    proxy_pass http://127.0.0.1:8080;
	    proxy_redirect off;
	}
	
	location ~ ^/(css|img|js|fonts)/ {
	    root /var/www/htdocs/c1;
	    # Optionally cache these files in the browser:
	    # expires 12M;
	}
		
	location /{
	    proxy_set_header Host $host;
	    proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-Forwarded-For $remote_addr;
	    proxy_pass http://127.0.0.1:8080;
	    proxy_redirect off;
	}
}

# HTTPS server
#
server {
	listen       443 default_server;
	server_name  _;
	root         /var/www/htdocs/c1;
	include /etc/nginx/acme.conf;
	
	ssl                  on;
	ssl_certificate      /etc/ssl/chargen.one.fullchain.pem;
	ssl_certificate_key  /etc/ssl/private/chargen.one.key;
	ssl_session_timeout  5m;
	ssl_session_cache    shared:SSL:1m;
	ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
	ssl_prefer_server_ciphers   on;
	
	location ~ ^/.well-known/(webfinger|nodeinfo|host-meta) {
	    proxy_set_header Host $host;
	    proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-Forwarded-For $remote_addr;
	    proxy_pass http://127.0.0.1:8080;
	    proxy_redirect off;
	}
		
	location ~ ^/(css|img|js|fonts)/ {
	    root /var/www/htdocs/c1;
	    # Optionally cache these files in the browser:
	    # expires 12M;
	}
		
	location / {
	    proxy_set_header Host $host;
	    proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-Forwarded-For $remote_addr;
	    proxy_pass http://127.0.0.1:8080;
	    proxy_redirect off;
	}
}

With that entire block removed from the main config, below the line server_tokens off;, there's just the following remaining in /etc/nginx/nginx.conf:

include /etc/nginx/sites/*.conf;

If I want to disable a site, I change the file extension from .conf to .dis and restart nginx. That way I can easily see which sites are enabled and which sites aren't without having to mess with the ln command or symbolic links.

Adding a new virtual host

The first host is the hardest, but onces up and running provides a template for any future hosts. I keep things fairly minimal, but adding support for PHP-based sites is as simple as copying from the default OpenBSD nginx config. The TLS config still points to the chargen.one certificate as only the certificate's associated hostnames change, not the filename.

    server {
        listen       80;
        server_name  hackingforfoodbanks.org www.hackingforfoodbanks.org;
        root         /var/www/htdocs/hackingforfoodbanks;

        include /etc/nginx/acme.conf;
        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root  /var/www/htdocs/hackingforfoodbanks;
        }

        location / {
            try_files $uri $uri/ =404;
            # Optionally cache these files in the browser:
            # expires 12M;
        }

    }

    # HTTPS server
    #
    server {
        listen 443;
        server_name  hackingforfoodbanks.org www.hackingforfoodbanks.org;
        root         /var/www/htdocs/hackingforfoodbanks;
        include /etc/nginx/acme.conf;

        ssl                  on;
        ssl_certificate      /etc/ssl/chargen.one.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/chargen.one.key;

        ssl_session_timeout  5m;
        ssl_session_cache    shared:SSL:1m;

        ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
        ssl_prefer_server_ciphers   on;

	location / {
	    root /var/www/htdocs/hackingforfoodbanks;
	    # Optionally cache these files in the browser:
	    # expires 12M;
	}

}

The only major differences are the removal of default_server in the listen directives, the changes to server_name and root to point to the correct spot and the removal of all of the dynamic parts associated with Chargen.One. Check whether or not there are problems with the nginx config before restarting by using the following command:

nginx -t -c /etc/nginx/nginx.conf

Providing the syntax is ok, restart nginx with rcctl restart nginx as root, or via doas.

Adding domains to acme-client

The final part of the puzzle is to add LetsEncrypt support for the new domain. The easiest way to add domains to acme-client is through the alternative names feature. Here's what I've added to /etc/acme-client.conf in order to support the hackingforfoodbanks.org URL.

alternative names { hackingforfoodbanks.org www.hackingforfoodbanks.org }

After adding that, and deleting the existing /etc/ssl/chargen.one.crt file, acme-client can be called to add the new domain.

rm /etc/ssl/chargen.one.crt
acme-client -vFAD chargen.one

Note that the alternative names for our new domains are under the chargen.one domain section. The domain section name is passed to acme-client, not the domain itself.

With a fully functioning certificate and nginx setup, run rcctl restart nginx to finish things off, and test the new site in a browser.

Adding HTTPS redirects

You might want to redirect some of your sites to HTTPS rather than serve a HTTP version of your site. While often touted as a panacea, this introduces a mix of advantages and drawbacks.

  • The content being delivered will be wrapped in transport layer encryption, making it harder for someone eavesdropping to identify the content being transferred (confidentiality)
  • As the transfer is encrypted, it becomes hard to interfere with the content.
  • HTTPS relies on a routinely (temporarily) broken permission-based model, often abused by companies and nation states. Thus while it's useful, it shouldn't be relied on for bulletproof 100% security.
  • Currently support TLS versions used in HTTPS aren't supported by most browsers available for legacy Operating Systems such as Windows XP. This means your site may be inaccessible over Windows XP and older versions of Android.

I'm not saying don't use HTTPS for a static site. There is no harm in supporting both, especially for a static web site. Just consider the site's audience and make a reasoned, deliberate decision as to whether or not to support accessing your content over HTTP before proceeding.

This site is accessible over HTTP and HTTPS precisely so users of older systems can still access the content via the reader, but authenticated access only works over HTTPS, and no mixed content is loaded.

As people accessing hackingforfoodbanks.org may not have access to current technology (e.g. foodbank users), I made a conscious decision to leave HTTP access open. For another site, rawhex.com, there's less of a requirement to leave HTTP access open, so I'll redirect that to HTTPS.

It's always annoying when a doc doesn't show the whole config for something complicated, so here's the /etc/nginx/sites/rawhex.conf file in full:

    server {
        listen       80;
        server_name  rawhex.com www.rawhex.com;
        return 301 https://$server_name$request_uri;

    }

    # HTTPS server
    #
    server {
        listen 443;
        server_name  rawhex.com www.rawhex.com;
        root         /var/www/htdocs/www.rawhex.com;
        include /etc/nginx/acme.conf;

        ssl                  on;
        ssl_certificate      /etc/ssl/chargen.one.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/chargen.one.key;

        ssl_session_timeout  5m;
        ssl_session_cache    shared:SSL:1m;

        ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
        ssl_prefer_server_ciphers   on;

        # add HSTS header to ensure we don't hit the redirect again
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;


        location / {
            root /var/www/htdocs/www.rawhex.com;
            # Optionally cache these files in the browser:
            # expires 12M;
        }

}

The HTTP 301 redirect shrinks the rest of the block to almost nothing. The HSTS header is a way to ensure that once redirected, a browser will only make requests over HTTPS, even if the user clicks on a HTTP link. The end result is an A+ score from Qualys' SSL Labs. There are things that can be done to improve the score, but these come at the cost of compatibility with older browsers and Operating Systems such as Windows Vista and 7.

Modularizing further

You might've noticed in the above that I'm repeating a lot of SSL settings. For HTTPS sites, it's best to keep things consistent. As such I've moved my ssl settings (aside from HSTS) into a separate file, /etc/nginx/https.conf. This means I only have to change one file for all HTTPS site configs. The current version of my file looks like this:

        ssl                  on;
        ssl_certificate      /etc/ssl/chargen.one.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/chargen.one.key;

        ssl_session_timeout  30m;
        ssl_session_cache    shared:SSL:2m;

        ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
        ssl_prefer_server_ciphers   on;

I set a higher SSL Session timeout and cache for performance purposes. People should be able to use a single SSL session to cover a full visit to and around the site. People rarely spend longer than 30 minutes there unless they leave a tab open, at which point I'm happy to reinitialize.

Please don't confuse SSL Sessions with HTTP or application sessions. They're different things. If in doubt, the defaults are probably fine.

Now all I have to do is add include /etc/nginx/https.conf; below include /etc/nginx/acme.conf; to my sites, and any changes to ciphers or timeouts will be picked up systemwide with a single change.

Conclusion

Now that I can add static sites to the Chargen.One system, I'll migrate the rest of my content over. With a clean, modular nginx config, content is served speedily and thanks to OpenBSD, to a level of security I'm comfortable with. I still need to find somewhere to move my git repos to, and I'm not sure chargen.one is right for that, but roman has a few ideas that I might borrow from.

 
Read more...

from steve

Sometimes I miss things on the Internet the first time round. I'm not aware they were things until I stumble across them randomly some time later. A week ago I came across the writing of Bronnie Ware, a pallative nurse in Australia who in 2009, documented the 5 most common regrets she encountered from the dying.

The regrets themselves weren't surprising. I've encountered all of them at some point. What surprised me was that the ones I'd encountered were the common ones. In her post, the regrets of the dying, Bonnie lists 5 regrets she commonly encountered from her patients:

  1. I wish I’d had the courage to live a life true to myself, not the life others expected of me.
  2. I wish I hadn’t worked so hard.
  3. I wish I’d had the courage to express my feelings.
  4. I wish I had stayed in touch with my friends.
  5. I wish that I had let myself be happier.

Bonnie went on to write a book about this, The Top Five Regrets of the Dying – A Life Transformed by the Dearly Departing. Her blog post touched a lot of people at the time, including YCombinator founder Paul Graham.

Graham viewed at these regrets as errors, in this case of omission. Not everyone has the opportunities Graham has had in life, and while I can understand his rationale, I'm not sure I agree with it. I believe these regrets have an internal element, but rarely arise in a vacuum.

I looked for signs of these regrets in myself and those around me. I found examples everywhere amongst my neighbours, family and friends:

  1. The regret of the women who married the man who knocked them up because it was expected at the time.
  2. The regret of the men who try so hard to provide for their children they never get to grow close to them.
  3. The gay men and trans women who attempted suicide because they couldn't reconcile their identity with their devout cultural or religious beliefs.
  4. The old man living alone in his house, slowly forgetting everything and everyone he knew.
  5. The women who spend their lives looking after everyone else, barely, if ever making time for herself.

I've experienced all 5 forms of Bronnies' regrets. Thankfully I've always had the ability to do something about it. I don't pretend that others have that capability. In fact I doubt most people are aware of these regrets until long after they've formed.

What I can do, is when I see this in others is be kind, be patient, encourage them to open up, and to listen. But I felt I should find a way to identify the first signs of these regrets in myself.

Graham inverted the regrets to create a list of 5 commands, but I found these to be very negative. Perhaps that's ok for him. It's not really for me. Instead, I chose 5 questions to periodically ask myself. Their purpose is to help me become more mindful of things that make me sad:

  1. Have I been authentic throughout the month, or was there a moment where I became a version of me to meet the expectations of others?
  2. Have I made enough time and space this week to be with those I love?
  3. This week, have I been continuously open and honest with myself and those around me?
  4. When did I last talk about something other than work to those I really care for?
  5. What did I do for me this week?

I've put these 5 questions up here, so I can check in on myself now and again. I also have them in a notes folder so I can go through the list once a week.

If the answer to a question is no, I make a note in Joplin about why the answer is no, and what I'll do to address it. It's ok for there to be a no response to a question, but I should at least make a conscious decision about it when it arises. There are no wrong answers, the thinking alone is often enough to kick me into gear.

My hope is that by asking these questions regularly, I can avoid things before they become regrets, instead of fixing them later on. That way, whenever it's time to die, I can do so with no regrets.

 
Read more...

from steve

*The approach and scripts discussed here use mysqldump to back up a database at one point. Yes I know this isn't in OpenBSD base, but this was added just for a specific system. It's easy not to add it, and everything else is done through tools from the base system.

With any system, it's important backups and restores work properly. With #Chargen.One, I wanted to protect user data, and be able to easily restore. The important things for the backup were:

  • Single timestamped backup file
  • No additional software installed above what's already on the box
  • Backups are pulled from a central server, not pushed to one
  • Portable script
  • Use privilege separation so local accounts can't access backups

Most services like borg backup work best with a backup server visible from the server being backed up. I use a huge NAS to store my backups, and a separate server to store backups of backups. The NAS is behind a Firewall and the other server can see the NAS but not the Internet. As such, I need a backup system that lets me use the NAS to pull from Chargen.One onto the NAS, and then allow my isolated backup server to pull from the NAS. If that sounds a little paranoid, at least you understand why I use OpenBSD.

Backups are taken by root on a nightly basis, and put into a folder belonging to a dedicated backup user. Early in the morning, the backup is pulled from a system which deletes the backup from chargen.one. The remote backup system will store 30 days' worth of backups.

Configuring a backup account

The /home partition is the largest on the VPS, so I created an account to store backups in a temporary folder, create an archive, delete the temporary folder and change permissions so the remote backup system can pull and remove the backup archive.

To set up the backup user account, use the following commands (as root):

# useradd -m backup
# chmod 700 /home/backup
# su - backup
$ cd .ssh

On the backup system, generate an ssh keypair using ssh-keygen -t ed25519. Copy the contends of id_ed25519.pub from the backup system into /home/backup/.ssh/authorized_keys on the server being backed up.

SSH into the backup account on the server from the NAS to make sure everything works.

Each backup archive is created by a script that creates and stores content in /home/backup/backup/. Once backed up, the script will create a timestamped archive file and delete the /home/backup/backup/ directory. The script starts off very simply:

#!/bin/sh

mkdir /home/backup/backup
# Add stuff below here


# Don't add stuff below here
rm -rf /home/backup/backup

Backing up MySQL data

If you want to implement my backup scheme and don't run MariaDB or MySQL, then skip this section and backup using commands from base only.

Because MySQL is configured to use passwords, a /root/.my.cnf file containing credentials for the mysqldump command is needed.

[mysqldump]
user=root
password=your_password_here

The mysqldump command fully backs up all mysql databases, routines, events and triggers.

Add the following to the backup script (all one line):

mysqldump -A -R -E --triggers --single-transaction  > /home/backup/backup/mysql.gz

The --single-transaction option causes the backup to take place without locking tables.

Backing up a package list

OpenBSD uses it's own package management system called pkg. To create a backup of installed packages add the following to the backup script:

pkg_info -mz > /home/backup/backup/packages.txt

This can then be restored from a backup using pkg_add -l packages.txt.

Backing up files

The following files and directories should be backed up:

  • /etc
  • /root
  • /var/www
  • /var/log
  • /var/cron
  • /home, excluding /home/backup
  • /usr/local/bin/writefreely
  • /usr/local/share/writefreely

Use the tar command to create backups. A discussion of the tar command is best left to man tar, but as the backup isn't very large, I'm not using incremental backups, which keeps things simple...up to a point.

OpenBSD's tar implementation doesn't add the --exclude option as it's a GNU extension. Other BSDs such as FreeBSD do add the option, but the OpenBSD team prefer not to have it. I could've added the GNU tar package, but one of the stated goals of the script is to not require additional software to keep things portable. Paths such as /home/backup are excluded using shell expansion instead.

To test this, try the following command:

# tar cvf bk.tar /home/!(backup)

The exclamation mark means exclude anything in the parentheses. For multiple directories, separate the names with a pipe symbol, e.g. !(backup|user) to exclude both backup and user directories.

There are complications and error messages will be shown on each backup if absolute paths are added to the tar command. This means that an email would be generated every night, even if the backup succeeds. Ain't nobody got time for that.

As a workaround, changing to the root directory at the start makes all paths relative, and allows the shell expansion to work. The -C switch can be used instead, but this breaks shell expansion.

The final commands to go in the script look like this:

cd /
tar cf /home/backup/backup/files.tar etc/!(spwd.db) root \
	var/www var/log home/!(backup) /var/cron \
	usr/local/bin/writefreely usr/local/bin/backup.sh \
	usr/local/share/writefreely 

I've used backslashes to break up the lines for readability, but all the paths could be put on a single line if preferred.

I've excluded /etc/spwd.db from the backup because OpenBSD's built-in tar uses a feature called pledge that restricts access to certain files. The file isn't particularly important to this specific backup, but contains the shadow password database, which I'm happy to recreate as part of the restore process.

At this point you might wonder why gzip compression isn't being used in the tar archive. This is because the final archive will be compressed, and there's no point in compressing twice.

Creating the final archive

To distinguish between backups by date, I used a timestamp, generated by the date command. By default there are spaces and colons, neither of which are good for interoperability across Operating Systems and filesystems. Use date +%F_%H%M%S to generate a more reasonable format. Using tar's -C switch changes the tar working directory to /home/backup and stops a leading / error message appearing in the backup.

The final tar command in the backup script should look like this:

tar zcf /home/backup/c1_$(date +%F_%H%M%S).tgz -C /home/backup backup

It's also important to change the file ownership to the backup user so the remote system can delete the backup after it's been created.

chown backup:backup /home/backup/c1_*

The full backup script on chargen.one looks like this:

#!/bin/sh

mkdir /home/backup/backup
# Add stuff below here

# MySQL Backup
mysqldump -u root -A -R -E --triggers --single-transaction | gzip -9 > /home/backup/backup/mysql.gz

# Packages backup
pkg_info -mz > /home/backup/backup/packages.txt

# Files backup
cd /
tar cf /home/backup/backup/files.tar etc/!(spwd.db) root \
	var/www var/log home/!(backup) /var/cron \
	usr/local/bin/writefreely usr/local/bin/backup.sh \
	usr/local/share/writefreely 

# Final archive
tar zcf /home/backup/c1_$(date +%F_%H%M%S).tgz \
	-C /home/backup backup

# Fix permissions
chown backup:backup /home/backup/c1_*

# Don't add stuff below here
rm -rf /home/backup/backup

Automating the backup

As root (via su -, not doas), use crontab -e and add the following entry:

0 1 * * * /usr/local/bin/backup.sh

A new backup is created at 1am, every morning. On the remote server, a cron job calls a script at 3am to pull the backup down via scp using the following:

#!/bin/sh

find /Backups/c1/ -mtime +30 -exec rm {} \;
scp -q backup@chargen.one:./c1_*.tgz /Backups/c1/
ssh backup@chargen.one rm c1_*.tgz

And that's it! The secondary backup server pulls down the contents of /Backups from the NAS, so there's nothing left to do.

I'll write a separate post about restoring, as this post is already getting long, but hopefully it's useful to people who want pull, rather than push backups.

 
Read more...

from steve

Chargen.one is a little different to most writing experiences. It's still, very very much in development, so things are in places where you might not ordinarily expect them to be, and some things aren't there at all.

I've configured everything to support multiple blogs per user. As this is a themed instance, some people may want to have more than one blog. Your first blog uses your username by default, but each author can create up to 5.

For example, I'm interested in personal finance and investing, but by putting any thoughts into a separate blog, I can spare my less interested readers the boredom that comes with such a thing.

You might also notice the lack of image upload function. This is deliberate. If you want to use images, I'd suggest using something your own hosting or something like PixelFed. It's all markdown here, and easy to export too.

When you first hit publish, instead of going to your blog the post goes to drafts. You can then move it to a blog. This prevents you from accidentally publishing something you don't. Your account is also private by default. It can be made public in the settings.

 
Read more...

from steve

I've written this in case anyone else wants to try and build their own instance of #Chargen.One on OpenBSD. If you're looking for the easy route to getting Writefreely working, then docker is the way. This is the hard way, but as a federated blogging site with the *BSD community in mind, I felt it important that it runs on a BSD of some sort.

There are two systems involved in Chargen.One, a build system and a deployment system. There is actually a test environment but that's the same as the deployment environment. The build environment is called c0, the deployment is c1. Both run OpenBSD 6.4 at the time of writing.

This post covers setting up the initial webserver with nginx and letsencrypt. Part two will cover the mysql config, part 3 the build and part 4 my deployment process.

Initial housekeeping

On the build system, start by following the process detailed in man afterboot.

I used an OpenBSD.Amsterdam VM for the deployment system, so there's some tweaks to implement before you start.

Installing Nginx and Lets Encrypt

On both c0 and c1 it's the same. As root, run pkg_add nginx. Update /etc/newsyslog.conf as per the info in /usr/local/share/doc/pkg-readmes/nginx.

Preparing Nginx for LetsEncrypt

Add the line include acme.conf; to c1's port 80 server block below root /var/www/htdocs;

Now create a /etc/nginx/acme.conf file with the following

location ^~ /.well-known/acme-challenge {
    alias /var/www/acme;
    try_files $uri =404;
}

Preparing LetsEncrypt

We'll configure C1 to use LetsEncrypt. All content will be served over HTTPS, with only the reader accessible over HTTP for older systems.

Create a domain entry at the bottom of /etc/acme-client.conf file like the following:

domain chargen.one {
        domain key "/etc/ssl/private/chargen.one.key"
        domain certificate "/etc/ssl/chargen.one.crt"
        domain full chain certificate "/etc/ssl/chargen.one.fullchain.pem"
        sign with letsencrypt
}

Getting certs and a working HTTPS setup

Restart nginx, run acme-client -vAD and you should have working certs. Now it's time to configure HTTPS. The commented out defaults are reasonably sane at the time of writing, just change things to point to your certs. Here's what I had set up. We'll change this later.

    server {
        listen       443;
        server_name  chargen.one;
        root         /var/www/htdocs;

        ssl                  on;
        ssl_certificate      /etc/ssl/chargen.one.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/chargen.one.key;

        ssl_session_timeout  5m;
        ssl_session_cache    shared:SSL:1m;

        ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
        ssl_prefer_server_ciphers   on;
    }
 
Read more...