Gradew

Hosting your own OpenStreetMap tile server with imposm3

Sun, 06 Jan 2019 21:19:00 +0100

I’ve grown quite fond of the OpenStreetMap project, especially the self-hosting solution that comes with it. Of course, I would need an entire server farm to come even close the what Google Maps is capable of, but with the use of the imposm3 tool, well, one can get very interesting results!

I will be using imposm3 for the imports and updates, mapnik for the rendering engine and mod_tile to generate and serve files over the almighty intrawebz.

Why not osm2pgsql ?

If you’ve ever tried importing an entire country with osm2pgsql, you’ll know how much of a pain that is. It took me about 18 hours to import France, and don’t even get me started on diff imports or later country imports.

If I want to have a somewhat sexy self-hosted mapping solution, at least let me have daily diffs and decent performance from the ground up!

That’s where imposm rocked my world! It reduced importing time to a mere 2.5 hours, and I found the tables to be most convenient: some tables are “generalized” to account for various zoom levels, meaning it won’t have to process as much data when only low detail is required. Also, it’s way easier to figure out where all the objects are: roads/paths/tracks/highways are in osm_roads, buildings are in osm_buildings, areas are in osm_areas, etc.

Why can’t I just use that with mapnik then ?

Well, you can, but the generated osm.xml file only works with osm2pgsql imports. That’s when I figured out I needed to understand how this file works, and build my own for imposm3.

Installink mapnik (with libxml2)

apt install make cmake g++ libboost-dev libboost-system-dev   libboost-filesystem-dev libexpat1-dev zlib1g-dev   libbz2-dev libpq-dev libproj-dev lua5.2 liblua5.2-dev
apt install clang libxml2 libxml2-dev libxml2-utils liblwgeom-dev libpq-dev libproj-dev zlib1g-dev libharfbuzz-dev libfreetype6-dev libboost-all-dev python-setuptools unifont postgis

git clone https://github.com/mapnik/mapnik
cd mapnik
git submodule update --init
python scons/scons.py install XMLPARSER=libxml2

Installing mod_tile

apt install libagg-dev

git clone https://github.com/openstreetmap/mod\_tile.git
cd mod\_tile
./autogen.sh
./configure
make -j2
make install
make install-mod\_tile

Next, you’ll need to create /etc/renderd.conf:

[renderd]
;socketname=/var/run/renderd/renderd.sock
num\_threads=4
tile\_dir=/var/lib/mod\_tile
stats\_file=/var/run/renderd/renderd.stats

[mapnik]
plugins\_dir=/usr/local/lib/mapnik/input
font\_dir=/usr/local/lib/mapnik/fonts
font\_dir\_recurse=1

[default]
URI=/tiles/
TILEDIR=/var/lib/mod\_tile
XML=/home/osm/imposm.xml
HOST=localhost
TILESIZE=256

Then, put this in your Apache vhost:

	ServerAdmin webmaster@localhost
	ServerName map-tiles.gradew.net
	DocumentRoot /var/www/tiles

	<IfModule mod_tile.c>
		LoadTileConfigFile /etc/renderd.conf
		#ModTileRenderdSocketName /tmp/osm-renderd
		ModTileRenderdSocketName /var/run/renderd/renderd.sock
		ModTileRequestTimeout 3
		ModTileMissingRequestTimeout 30
		ModTileMaxLoadOld 2

		# In bulk mode, mod_tile does not request any dirty tiles to be rerendered.
		ModTileBulkMode On

		# That's a week
		# ModTileCacheDurationMax 604800
		# That's a year
		ModTileCacheDurationMax 31536000
	</IfModule>

Free SSL/TLS certificates with Let's Encrypt

Wed, 17 Feb 2016 11:13:00 +0100

We’ve all been there: buying self-signed certificates is a bit annoying, especially when it comes to wildcards. A new project has been around for a couple of months now, it’s called Let’s Encrypt.

It’s basically a bunch of scripts that will help you sign certificates for your Apache/Nginx vhosts… and for free, no less! The good news is that these certificates will be recognized by pretty much any browser. Farewell, security alerts! The bad news is: no wildcard support for now. Well… that’s not exactly bad news, since you can create as many certificates as you want, and SNI is rather widespread now.

Installing Let’s Encrypt

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
./letsencrypt-auto --help

Creating/signing a new certificate

cd letsencrypt
./letsencrypt-auto certonly --webroot -w /var/www/vhost_path/ -d FQDN

Configuring Apache

SSLCertificateFile /etc/letsencrypt/live/FQDN/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/FQDN/privkey.pem

Checking your certificates

Certificates signed with Let’s Encrypt will be valid for 3 months. All the more reason to crontab a script I whipped up for the occasion:

#!/bin/sh

rootpath="/etc/letsencrypt/live"
warning=604800 # 1 week

RETVAL=0
ISCRON=0

if [ ! -z "$1" ]; then
    if [ "$1" = "--cron" ]; then
        ISCRON=1
    fi
fi

for d in `ls ${rootpath}`; do
    certpath=${rootpath}/${d}/fullchain.pem
    expirydate=`openssl x509 -enddate -noout -in ${certpath} | sed 's/notAfter=//'`
    if openssl x509 -checkend ${warning} -noout -in ${certpath}; then
        if [ $ISCRON -eq 0 ]; then
            echo "${d} OK"
        fi
    else
        echo "${d} WILL EXPIRE ON ${expirydate}"
        RETVAL=1
    fi
done

exit $RETVAL

The crontab entry will need to look something like this:

MAILTO=admin@yourdomain.com
0 0 * * * /path/check_certs.sh --cron

PostgreSQL replication (part 1/2)

Fri, 08 Jan 2016 15:07:00 +0100

Here’s a quick manual on how to compile, install and set up PostgreSQL 9.5. I will also explain how to set up asynchronous replication between two nodes.

This has been tested on a 64-bit Debian Jessie.

Note: the binaries will be installed to /opt/postgres, and the scripts and data will be installed to /home/postgres. Feel free the change the paths from the script according to your needs.

Compiling and installing PostgreSQL

Install all needed dependencies:

apt-get install wget bzip2 g++ flex bison libreadline-dev zlib1g-dev make

Get the source archive from the official website, un-tar it, compile and install the beast:

tar xjvf postgresql-9.5.0.tar.bz2
cd postgresql-9.5.0
./configure --prefix=/opt/postgres && make install && cd contrib && make install

Then, create a user and its environment:

groupadd postgres ; useradd -g postgres -s /bin/bash -d /home/postgres postgres ; mkdir /home/postgres ; cp /etc/skel/.* /home/postgres ; chown -R postgres:postgres /home/postgres

Change to the postgres user:

su - postgres

Add the following lines to .bashrc (make sure all paths are correct):

export PG_HOME="/opt/postgres"
export PG_DATA="/home/postgres/data"
export PATH=$PG_HOME/bin:$PATH
export LD_LIBRARY_PATH=$PG_HOME/lib:$LD_LIBRARY_PATH

Create a file called ~/vars with the following (make sure all paths are correct):

export PG_DATA="/home/postgres/data"
export PG_WALPATH="/home/postgres/walarchives"
export PG_THISNODE="pg01"
export PG_OTHERNODE="pg02"

Configuring PostgreSQL

Make sure you’re still logged in as postgres, and initialize the database:

initdb -D $PG_DATA

This will create an empty database, as well as default configuration files. Here’s where you start tweaking things on your own, but here’s something to get you started; add the following lines to $PG_DATA/postgresql.conf:

listen_addresses = '*'
log_destination = 'stderr'
logging_collector = on
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d.log'
log_rotation_age = 1d

Starting and stopping PostgreSQL

You are now ready to use your PostgreSQL instance. Here’s how it goes:

pg_ctl -D $PG_DATA [start|stop|reload]

Setting up asynchronous replication

This set-up will assume you have two different servers with no shared filesystem between them. If you do happen to have a shared filesystem, you might want to change the ugly “scp” from archive_command.sh below to a simple “cp” (still ugly, but should be faster).

How replication works in PostgreSQL

Many find PG replication a dirty business to set-up. Granted, things look kind of messy when you start calling shell scripts from your database configuration files. But once you get the hang of it, it actually becomes simple, and even robust.

Bear in mind that a PG cluster is far more suitable in a dedicated infrastructure (like two servers on the same LAN in a datacenter with a SAN attached to them). It can still work in other infrastructure types, but you definitely won’t get the same performance out of it.

Basically, in a nominal state, a cluster will have a master node, which you can use to do SELECTs and INSERTs/UPDATEs/DELETEs, and the like… The slave node(s) will maintain a connection with the master node to replicate all operations made by the master. If you do anything on the master databases, it will be replicated to the slave databases.

This is the purpose of having “recovery.conf” in your $PG_DATA directory. When you boot up PostgreSQL, and it finds recovery.conf, it will start as a slave, and immediately connect to the master node to be kept updated of all changes that are made. Being a slave PG node also means the database will have to be created from a backup of the master node. This can be done on-the-fly by using the init_as_slave.sh script given below. You’ll notice that this step can be achieved with a simple command run from the slave node: pg_basebackup.

In case communication fails between the master and the slave, redo logs are created by the master node, and sent over to the slave node. These are called WAL archives. This is done by the “archive_command.sh” script below. You’ll notice that it uses the “scp” command to copy the WAL archive over to the slave. Make sure that the “postgres” user from the master node can SSH to the “postgres” user of the slave node. And the other way around.

Once the WAL archives have been assimilated by the slave node, it runs the pg_archivecleanup command, as declared in recovery.conf. WAL archives are then deleted from the slave. Make sure you monitor disk space, though: if the slave node fails, the WAL archives will keep accumulating and will eventually (and rather quickly) gobble up all your storage space.

And that’s pretty much all there is to understand about replication in PostgreSQL :)

Configuration

We’ll assume you’re setting up two replicated servers, one of which will be the master, the other one will be the slave. Start by setting up the slave server the same way you did with the master. Then, add the following to $PG_DATA/postgresql.conf on both servers (yes, both - also, make sure the paths are correct):

hot_standby = on
wal_level = hot_standby
archive_mode = on
archive_command = '/home/postgres/archive_command.sh "%p" "%f"'
archive_timeout = 60
max_wal_senders = 5

Execute the following script to create ~/archive_command.sh on both servers:

cat > ~/archive_command.sh<<EOF
#!/bin/sh
. ~/vars

if [ -z "\$PG_OTHERNODE" ]; then
  echo "PG_OTHERNODE undefined!"
  exit 1
fi
fullpath="\$1"
walfile="\$2"
scp "\$fullpath" \$PG_OTHERNODE:\$PG_WALPATH/\$walfile > /dev/null 2>&1
exit 0
EOF
chmod +x ~/archive_command.sh

Then, create ~/recovery.conf on both servers (make sure all paths are correct, and don’t forget to replace ‘other_node’ and ‘this_node_name’ with the appropriate values):

cat > ~/recovery.conf<<EOF
standby_mode = 'on'
primary_conninfo = 'host=other_node application_name=this_node_name'
trigger_file = '/tmp/pgtrigger'
restore_command = 'cp /home/postgres/walarchives/%f %p'
archive_cleanup_command = '/opt/postgres/bin/pg_archivecleanup /home/postgres/walarchives %r'
EOF

Add the following line to $PG_DATA/pg_hba.conf (again, on both servers):

host    replication     all     0.0.0.0/0               trust

Create the final utility script on both servers:

cat > ~/init_as_slave.sh<<EOF
#!/bin/sh

. ~/vars

if [ -z \$PG_DATA ]; then
  echo "PG_DATA undeclared!"
  exit 1
fi
if [ -z \$PG_OTHERNODE ]; then
  echo "PG_OTHERNODE undeclared!"
  exit 1
fi
if [ ! -f \$PG_DATA/postgresql.conf ]; then
  echo "Wrong directory, mate!"
  exit 1
fi
pg_ctl -D \$PG_DATA stop > /dev/null 2>&1
rm -rf \$PG_DATA/*
rm -rf \$HOME/walarchives/*
pg_basebackup -D \$PG_DATA --host=\$PG_OTHERNODE --port=5432
if [ \$? -eq 0 ]; then
  cp ~/*.conf \$PG_DATA/
else
  echo "Could not back up database!"
  exit 1
fi
EOF
chmod +x ~/init_as_slave.sh

Ok, so what are all these scripts for ?

You’ve probably guessed by now… For the moment, PostgreSQL bases all of its replication work on scripts and actions on the filesystem:

copying the redo logs from the master node to the slave node(s) is done by a script, archive_command.sh
automatic purging of the redo logs on the slave nodes is also done by a script: the archive_cleanup_command and restore_command settings defined in recovery.conf
promoting a node from slave to master is also done by creating “/tmp/pgtrigger” on the slave node; this is defined in recovery.conf as well

Usage

You can start the instance on the master node:

pg_ctl -D $PG_DATA start

And replication from the slave side can be initiated as follows:

~/init_as_slave.sh
pg_ctl -D $PG_DATA start

This concludes part 1/2

If everything went according to plan, you should have a working simple PostgreSQL cluster, where:

One node is a master, and can be used to read from and write to
The second node is a slave, and can only be used to read from it
Replication is asynchronous; the performance impact will be kept to a minimum. Also, if the slave node fails, the master won’t mind.

Now, we’ve paved the way for all the replication work, here. But keep in mind that all of this doesn’t make for a full-blown, usable cluster; your application will need to use some kind of PostgreSQL proxy in order to always be sure to connect to a master node. This can be done with PGPool, and will be covered in part 2 :)

NAXSI, a firewall for NGINX

Mon, 04 Jan 2016 15:45:10 +0100

NAXSI is an open-source WAF for NGINX that uses whitelists to determine whether traffic should be authorized. Don’t think of it as some simple, ready-to-use IDS, though. You will need to generate the whitelists yourself, based on your own set-up.

Here is an excerpt of the description:

_NAXSI means Nginx Anti XSS & SQL Injection.

Technically, it is a third party nginx module, available as a package for many UNIX-like platforms. This module, by default, reads a small subset of simple (and readable) rules containing 99% of known patterns involved in websites vulnerabilities. For example, <, | or drop are not supposed to be part of a URI.

Being very simple, those patterns may match legitimate queries, it is Naxsi’s administrator duty to add specific rules that will whitelist those legitimate behaviours. The administrator can either add whitelists manually by analyzing nginx’s error log, or (recommended) start the project by an intensive auto-learning phase that will automatically generate whitelisting rules regarding website’s behaviour.

In short, Naxsi behaves like a DROP-by-default firewall, the only job needed is to add required ACCEPT rules for the target website to work properly._

Here’s how it works:

Install it and set it to Learning Mode (which is the default)
Logs will be appended to /var/log/nginx/error.log, with the “NAXSI_FMT” prefix. Let it run for a while (hours, days, weeks…)
While you do that, set up an Elasticsearch instance and create an index called “nxapi”
Alter nxapi.json to make it point to your Elasticsearch instance
Feed all the generated logs into Elasticsearch:

nxtool.py -c nxapi.json --file=/var/log/nginx/error.log

Generate whitelists and manually review them:

nxtool.py -c nxapi.json -f --slack

Apply your reviewed whitelists and disable Learning Mode; you’re all set!

Now, I know… Using Elasticsearch simply to store the logs might seem a bit much, but using Postgres/MySQL apparently wasn’t so easy. Setting up an ES instance isn’t so hard anyway, I’ll write up an article about that.

The source for NAXSI can be found here: https://github.com/nbs-system/naxsi

In Memoriam - Ian Murdock (1973 - 2015)

Thu, 31 Dec 2015 12:00:00 +0100

Thanks for everything, man.

Google Authenticator and FreeRADIUS

Mon, 05 Jan 2015 12:00:00 +0100

Two-factor authentication is all around us now, and Google has provided one of those soft authenticators, the likes of which Battle.net use for their gaming platform. The only difference is, you can integrate Google’s implementation anywhere you like (at least on *nix platforms). You’ll find plenty of documentation on how to achieve this to implement 2-factor auth on SSH or such services, but I wanted to centralize this and use FreeRADIUS so that more services can use it, namely web services.

The hardest part was actually the PAM syntax; here’s how I achieved it:

aptitude install g++ make autoconf libtool libpam0g-dev
git clone https://github.com/google/google-authenticator && cd google-authenticator && ./bootstrap.sh && ./configure && make install
Set up Freeradius for PAM auth; this is fairly easy, and Google is your friend 😉
/etc/pam.d/radiusd:

# Google Auth + LDAP
auth required /usr/local/lib/security/pam_google_authenticator.so user=0 secret=/radius/${USER}/google_auth forward_pass
auth required pam_ldap.so use_first_pass

And that’s it; FreeRADIUS is now tied in to PAM, which in turns uses Google Auth + your LDAP server :-)

Microsoft and e-mail deliverability

Mon, 07 Jul 2014 13:00:00 +0200

Well, it’s been a while since my last rant at Microsoft, but this time just takes the cake. Here’s what’s happening now: both my mail exchangers have been blacklisted. It might be worth noting that the very same IPs are all in the green on the DNSBLs. Also, these MXs have a throughput of roughly 200 emails / month. YES, I did check EVERY email in and out of my MXs, and none of them were spam (20 of them were personal emails sent to ONE hotmail.com addy). So I’m no spammer. Nevertheless, I got blacklisted. I asked to be delisted, but I got blacklisted again 2 days afterwards. Here’s what the guys at support have to say about this:

Email filtering is based on many factors, but primarily it’s due to mail content and recipient interaction with that mail. Because of the proprietary nature of SmartScreen® and because SmartScreen® Filter technology is always adapting and learning more about what is and isn’t unwanted mail, it is not possible for us to offer specific advice about improving your mail content. However, in general SmartScreen® Filter evaluates specific words or characteristics from each e-mail message and weights them, based on their likelihood to indicate that a message is unwanted or legitimate mail. Unfortunately, after reviewing the information you provided and in compliance with our mail policies, we are unable to offer immediate mitigation for your deliverability issue. However, we have some specific recommendations for you to consider that can help you to improve deliverability over time.

In other words, “We don’t know what the hell we’re doing, so go fuck yourself”. I currently have SPF and DKIM in place. I took some time to check their “guidelines”, and these guys happen to have written a “white paper” on e-mail deliverability; I think someone tricked them into thinking they were some kind of authority on e-mail deliverability. Here’s a funny excerpt from it:

If you believe your e-mail has been blocked in error by Microsoft, please review our best practices and technical guidelines to ensure compliance.

Again, in other words “If we’re mistaken, go fuck yourself too.”

I only have one piece of advice for all Hotmail users out there: switch to another provider! I mean it. If you haven’t, see my previous post on how Microsoft got hold of one of my aliases to sell it to spammers. If that doesn’t frighten you, I don’t know what will.

Xen and IPv6

Mon, 21 Apr 2014 13:00:00 +0200

I whipped up a (seemingly) working IPv6 set-up for both my hypervisors, meaning my services will soon be IPv6-compliant; yay! I did struggle with it, though. There are a couple of gotchas one needs to be aware of:

it is best to use RA, in case your provider changes the default gateway for your server
if your hypervisor is going to be a firewall (which I guess it is), you will need to use RADVD to advertise your dom0 to your domUs
RA and IPv6 forwarding do actually work together, but you need to enable both in sysctl.conf

Let’s get on with it, then. First, let’s set up IPv6 on the dom0:

#!/bin/sh

PATH=/bin:/sbin:/usr/bin:/usr/sbin

sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.xenbr0.accept_ra=2
sysctl -w net.ipv6.conf.xenbr0.autoconf=1

dhclient -cf /etc/dhcp/dhclient6.conf -pf /run/dhclient6.xenbr0.pid -6 -P xenbr0
ip -6 a add 2001:c0de:babe:580::1/56 dev xenbr0
ip -6 ro add 2001:c0de:babe:581::/64 dev xapi0

/etc/init.d/radvd start

This is what dhclient6.conf looks like:

interface "xenbr0" {
          send dhcp6.client-id <your DUID here>;
          request;
}

Then, configure RADVD; it will have to advertise on the internal domU network (xapi0):

interface xapi0
{
        AdvSendAdvert on;
        prefix  2001:c0de:babe:581::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
        };

};

Finally, you may want to firewall this all up a bit:

$IPT6 -P INPUT DROP
$IPT6 -P OUTPUT ACCEPT
$IPT6 -P FORWARD DROP
$IPT6 -F
$IPT6 -X
$IPT6 -Z

$IPT6 -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # Multicast traffic - needed for IPv6 routing
$IPT6 -A INPUT -p ipv6-icmp -d ff00::/8 -j ACCEPT # Link-local traffic as well, I guess it couldn't hurt...
$IPT6 -A INPUT -p ipv6-icmp -d fe80::/10 -j ACCEPT
$IPT6 -A INPUT -p udp --dport 546 -j ACCEPT # DHCPv6

$IPT6 -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT6 -A FORWARD -i xapi+ -o xenbr0 -j ACCEPT
$IPT6 -A FORWARD -i xenbr0 -o xapi+ -p tcp -m multiport --dports 80,443 -d $vm01_ipv6 -j ACCEPT

And there you go! You will need to make sure IPv6 multicast traffic can go from the dom0 to the domUs through xapi+: RADVD needs this to advertise.

IPSec w/ Racoon and TCP MSS Clamping

Tue, 25 Mar 2014 12:00:00 +0100

After a couple of weeks running IPSec in transport mode between two dedicated servers (i.e. on their respective public IP addresses), things started running amok, with the following messages in the syslogs:

Mar 16 16:37:29 host1 kernel: [5985158.107191] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:37:42 host1 kernel: [5985171.146710] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:37:43 host1 kernel: [5985172.458608] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:38:10 host1 kernel: [5985199.337572] pmtu discovery on SA ESP/4f014d9c/ddaec880
Mar 16 16:38:12 host1 kernel: [5985201.193461] pmtu discovery on SA ESP/4f014d9c/ddaec880

This appears to happen when big enough packets are trying to get through, and things go wrong because of the overhead induced by ESP packets. Also, it would appear my iptables rules are breaking PMTU by refusing certain ICMP packets, which are used in PMTU discovery. I will need to dig deeper into that.

In the meantime, here’s a quick (and dirty?) hack to clamp the Maximum Segment Size for TCP sessions and avoid overloading the packet size:

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

If you’re using PF, it may look something like this (though I haven’t tested it myself, but that’s the general idea):

match on em0 scrub (max-mss 1440)

Yubico validation server

Sun, 09 Mar 2014 12:00:00 +0100

In an era where getting your credentials stolen has become child’s play with spyware and keyloggers, it might be best to start using two-factor authentication. For a lot of companies out there, this means adding a second factor on top of an AD/LDAP infrastructure.

It’s actually easier than you think, and it won’t cost you an arm and leg. I will soon write a tutorial on how I achieved this.

The core service you will need is a validation server for YubiKeys. Whether you’re using LDAP or a Samba 4 instance to authenticate your users, it will all still come down to validating the OTP strings your users will be sending. If you search for the Yubico validation server, you’ll find yourself using old/deprecated/broken versions of the tool, depending on your distro (but even on recent ones).

I stumbled upon Raphaël Barrois’ version, hosted on Github, which was almost exactly what I needed (thanks, honestly!). Then again, I had to add some bells and whistles:

Support for PostgreSQL
Auditing capabilities on OTP tokens
A working example of an init.d script for this baby

And bam, there it is: https://github.com/gradew/yubiserve. Feel free to use it! :)